Theme 1: Proving Value, Measuring Outcomes

Measuring What Matters

Kristin Wisnewski, IBM

• IBM is a multi-national company of purchase
• More IBMers than people living in Iceland - equivalent to the number of people living in Ohio
• 51% of IBMers have been employed there under 5 years, 30% don't come in a traditional office everyday
• CIO mission is to manage information with empathy
• IT pyramid of pain - tasks, applications, end-to-end processes, technical underpinning

CIO Design impact:

• Tools & applications
• Tone & culture
• Physical environment

Measurement at a glance - spotlight on continuous improvement

• People - short-term + long term space
• Year-to-date research metrics
• 1H Mailer metrics - general, subject lines,
• Reactive: NPS, satisfaction, ease of use, goal completion, # of help tickets, employee preferences
• Proactive: from years of findings, cross-domain, visionary ideas, huge time and cost savings

VotE - Voice of the Employee program

• Pulse of whats happening now
• Many small forms
• 4000 completed forms per month
• LUX - Library of User Experience
  • Research collection of what user feedback
  • Metrics on a dashboard for all to use - example NPS - "what you measure is what improves"
  • NPS, goal completion, ease & capabilities, scorecard (most useful), comments (qualitative)
  • "always be testing versus always be guessing"

Examples of research-driven improvements

• User research & analytics - BluePages, Help system

Challenges

• NPS alteration controversy - color coding A/B test allowed for better dialog.

DesignOps journey

• Phase 1 - need for DesignOps
• Phase 2 - start of DesignOps
• Phase 3 - Maturing DesignOps
• Phase 4 - TBD Mastering DesignOps

Measurement matters!

Real Talk: Proving Value through a Scrappy Playbook

Dianne Que, Pandora

Being scrappy

• Determined to get shit done without letting shit get in the way
• Challenges:
  • Lack of clarity around process
  • No one place to find answers
  • To prove value, know what your audience values

Creating a playbook

• Hackmans' 5 Factor Model:
  • Being a real team
  • Compelling direction
  • Enabling structure
    • Frameworks, models to "get shit done"
    • Role clarity
    • Workflow - detailed documentation in workflow
    • McKinsey's 'Horizons" framework to facilitate conversations around feature and project prioritization
  • Supportive context
  • Expert Coding
• Provide visibility, visualization and vocabulary.

Proving value

• Initial value - project was a forcing function
• We're not always proving value upwards, but also across and down.
• Playbook rolled out without manager approval (!)

Measuring success

• Intentional
  • Clarity >> 3.1 to 4.6
  • Turn feelings into data
• Unintentional
  • Amplification
  • Collaboration
  • Acceleration
  • Playbook used in On-boarding, cross-functional product kick-offs, other decks
• Outcomes emerge over time.
• Measure baselines.
• No end-state in DesignOps.

What's next?

• Evolve + improve content
• Continue roadshow
• Define DesignOps metrics.

Measuring the Designer Experience

Kim Fellman Cohen, Pinterest

Problems

• Career development, skill building, team events, morale, communications, philanthropy, etc.
• What if your team had a dedicated resource to solve these problems?
• And how do you measure impact of resource?

Stay connected and learn

• Pinterest: 109 people, 8 disciplines
• Team became disconnected: not sure how to grow, not sure how to do best work, not feeling like they belong to a team.
• Challenges: can force a team into a reactive state, can be high risk, anchor or gravity problems, vague or subjective data, high rate of change in organizations
• Enter design-thinking: listen, define, collaborate, ideate, prototype & test, reporting.

Define problems and ideate

• Frame a set of questions to define murky feedback: 1:1s, surveys, focus groups, collaborative boards.
• Use a problem criteria:
  • Is the problem the right scope & size?
  • Does the problem provide enough context for ideas?
  • Is it broad enough to give you room to explore?
• Ideate together:
  • Brainstorm with a diverse group.
  • Synthesize using an idea criteria.
• Can you prototype and test the idea in the next month?
• Do you have the resources?
• Can you measure the outcome clearly?

Formula for measurements

• Define success with a formula.
• We will do a thing | this will happen if the thing is working || we need a thing for support.
• State objective.
• State time to measure.
• Map numbers to the behavior you want to see.

Test ideas like a product

• Build and test your ideas like a product.
• Don't build alone.
• Pilot to learn early and often.
• Open a share unit with stakeholders: share, build, test, iterate

Report and move forward

• Put a report together.
• How might we shape the future?

Group Activity: A Deep Dive Into Value and Outcomes

Kristin Skinner and Kamdyn Moore

How do we measure value?

• How is your organization measuring the value of DesignOps and/or ResearchOps?
• The most common sentiment around answering our question about measuring value: ¯\_(ツ)_/¯
  • 62% of respondents weren't sure...
• Most common barriers reported:
  • Limited time or resources
  • Involving the right people or teams
  • Lack of proven value
• Create the best space to get the best output
• Our collective goal today: identify the tools and tactics for measuring value.
• Common reasons for challenges in measuring values:
  • Every org has unique constraints
  • Bound by where your org is in its DesignOps journey
  • Conflict between what we need to measure vs what we're able to measure
  • Misconceptions about measurement
  • No industry standard within DesignOps

What it means to measure design - best practices:

• Measures are focused on what customers care about
• Measures are wrapped around the product design process
• Measurement techniques mixed methods and domains
• Measurement follows a framework not a fixed set of metrics
• Measurement spans journeys and dimensions of an experience

Google Heart Framework

• Goals
• Signals
• Metrics

menti.com code: 61 77 16

<group activity>

The starting point to measure

the impact of design ops

Call to Action

Join the next RM call

A Design Ops Girl in a Dev Ops World

Indra Klavins, Verizon

Expensive routers create connections

WeWork grammar lesson: You | I | We

A challenge: "program managers are just expensive routers"

Relating to teams (the "we"):

• Start by knowing everyone's origin story, then...

Create a shared understanding around...

• Joy & Delight
• Fear & Shame (of technical debt)
• Mapping the Agile Scrum story
  • Under joy and delight, there is a lot of fear and shame. Indra leveraged agile as her Trojan horse to break through to the teams she worked with.

Schedule quarterly retros

• 3 questions:
  • What went well
  • What didn't go well
  • What started poorly but improved
• Rules:
  • Take turns
  • Resist interrupting
  • Write down what you want to say
  • Practice active listening
  • Respect
  • No multi-tasking
  • Facts not opinions
• What happens in retros, stays in retros

Make connection between teams

• Remember "me" not just "we"
• Find the people with whom you can lock eyes
• Find your tribe, build your tribe
• Teach others

Distributed Design Operations Management

Jilanna Wilson, Zendesk

Your team is remote. Now what?

• Enterprise software used to be really shitty and expensive
• Zendesk principles:
  • Empathy
  • Focus on relationships
  • Humbldent - humble and confident
• SF, 100+ design team, 3400+ employees, 10 global offices
• DesignOps equips and empowers the remote teams
• Number of Americans working remotely has increased by 115%
• Disadvantages for remote workers:
  • Lack of motivation
  • Technical challenges
  • Lack of trust
  • Isolation + loneliness
  • Ideas don't matter
  • Missed opportunities

Co-located teams and distributed teams need the same things

Best practices for distributed teams:

• Communication - reaching out
  • Create regular opportunities to interact with teams.
  • Establish communication etiquette.
  • Set and stick to agendas.
  • Prioritize 1:1s - don't cancel them.
• Community - build connections
  • Send treats and care packages.
  • Look for ways to break the ice and have fun.
  • Give virtual kudos, high fives and shout outs.
  • Find ways to remind people they are part of a bigger organization.
• Collaboration - staying aligned
  • Leverage asynchronous communication and solid digital tools to maintain seamless workflow.
  • Reduce technical constraints.
  • Design systems maintain standards of excellence.
• Continual on-boarding - regular cadence of learning opportunities
  • Create meaningful connections through buddy systems, mentorship programs or virtual meet-and-greets.

Check yourself

• Rotate the pain
• Regular 1:1s
• Invest in proper tools
• Community-strengthening moments
• Robust on-boarding

The Bigger Picture: A Panel Discussion

DesignOps in Wonderland

Carla Casariego + Sarah Spenser, Express Scripts

Welcome to the Wonderland!

Express Scripts, recently purchased by Cigna (2018)

1 Would you like an adventure now, or shall we have tea first?

Need to enter into project "adventure" with a flexible mindset.

Navigate the organization and build relationships.

• Learn where you fit in the broader organization
• Align on complex experience challenges
• Establish credibility and show the value that infusing empathy has on delivering exceptional experiences.
• Prove out the power of multidisciplinary teams.
• Strategically align across critical product priorities.
• Establish a new model for delivering exceptional experiences - "full kit team."

2 Building the tunnel into Wonderland

• Intake iteration 1: chaos
• Intake iteration 2: sub-team alignment - build research content and design backlog
• Intake iteration 3: DX team alignment - prioritize as a full team

Say "no" for a better "yes"

• Protect the end-to-end the experience.
• Analyze value.
• Follow your instincts for exceptions.

3 Late for a very important date!

Evolve Agile, delivery expectations, timelines and roadmaps - "lookahead sprints"

• Introduce teams to new ways of working
• Prove the value of User Centered Design
• Deliver experiences validated with user feedback
• Align to established sprint cycles
• Build in lead time - stay 1-2 (or more) sprints ahead of SW engineering
• Be prepared to re-educate partners
• Provide visibility into initiatives supported by the team
• Serve as roadmaps for strategic priorities
• Maintain and adapt to ever changing needs

4 Protecting the Tea Party

Pivot the org structure: from Producer Network to DesignOps: centralize business foundations

• Identify value streams

5 Most everyone's mad here.

Find commonalities with other organizations to solve challenges and apply leading practices - create a strategy for experience operations

• Look the the strengths of the team
• Research emerging trends, leverage trusted advisors and peers in agencies or other companies
• Learn from others' journeys and experiences
• Garner support from leadership and partners in the organization
• Determine whether to align on value streams or specific experience disciplines
• Navigate the ever-changing organizational landscape

Reflections

• Strategically network and partner
• Create formalized & collaborative intake
• Establish agile cadences & roadmaps
• Centralize appropriate "run the business" functions
• Learn from others' experiences

Lessons From the DesignOps Journey of the World’s Largest Travel Site

Eniola Oluwole, Velir, previously at TripAdvisor

Design systems process was the catalyst on adopting DesignOps

TripAdvisor stats:

• $1.6B revenue annually
• 20 years old
• 16.3M average daily visitors

Design culture

• Myth: centralized design org | really: stand-alone groups
• Myth: user-centered | really: business-centered
• Myth: ensure usability & user insight | really: visualizer of product spec
• Each team had their own process
• No single source of truth
• No centralized governance or communication
• Make it green

Core problems:

• Teams were not collaborative
• Lack of efficiency and scalability
• Change resistant

The plan: Assess > Define > Organize > Make them care :-)

• We talked with our users - everyone want to know "what's official"

Assess - create design principles

• One place for all teams
• Anyone can access
• Easy to update
• Easy to search & navigate
• There is someone accountable

Define - create the pattern library and sync with Story Book

• Partner with business & product
• 5-month process

Organize - short-and long-term

• Used Confluence
• "The great clean-up" - went through every pattern existed (!)
  • If it's off-brand, not best practice or tested down, it was removed.
• Devise incentives

It was a fail.

Overall lack of adoption

• Lack of communication
• What is in the library didn't match the website
• No sense of ownership, never updated
• Inconsistent levels of craft & fidelity

Try again. Fail again. Fail better.

The NEW plan: Re-Assess > Re-Define > Re-Organize > Make them care :-D

Reassess: how are people using the design system?

• 72% design system novices
• Rarely, then daily
• 85% want to find patterns
• Hard finding what you need
• 0% wanted to understand
• 63% reuse old patterns

We must:

• Provide better on-boarding & training
• Use card sorting exercises to create a better taxonomy
• Focus on pattern examples, not usage, or research
• Increase new pattern awareness
• Keep patterns approved and up to date

Product Design Steering Group (PDSG) created

• Decision-making at multiple levels

Changed our reviews:

• Escalate debates to PDSG
• Questioned and stopped non-pattern work going forward
• Invited product managers to participate
• Had research team play a more prominent role
• Streamlined approval process
  • "I want X pattern"
  • Is it new? Yes...
  • PDSG
  • Decision
  • Share results
• Used Slack for design system
• Better partnership with development
• Enable others to care

What's next?

• Looking for a dedicated design-ops lead
• Better growth management
• Full design to code parity

Lessons learned

• You won't know until you Do
• Create a steering group
• Use real-time chat
• Really understand how it's working
• Enable others to broadcast wins
• Understand things will not go as planned

Transforming Strategic Research Capacity through Democratization

Marjorie Stainback and Kelsey Kingman, Fidelity

Education + Enablement allowed the team to cut out 11 days off projects - 11 days faster

Leverage and get the most out of your existing team through democratization

• Challenge: 1:8 researchers per designer - needed to scale research and quick(er) turnaround
  • Supports geographically dispersed teams
  • Evaluative

Program management

• Introduced remote unmoderated usability studies
  • Measured usage - know numbers and who was using, collect constructive criticism
  • Define desired outcomes - decrease time to user feedback (< 14 days)

Key ingredients

• Shared vision
• Education - 3-day class
• Validity - concern about poorly written test questions.
  • Created guard-rails: assign research buddy, presentation feedback, repository checkpoint

Pitfalls

• Don't schedule during the holidays
• Do not allow unlimited sign-ups - 6-7 students
• Do not forget to schedule time to implement the feedback
• Don't forget to have a kick off meeting

Other discoveries

• Articulate your purpose in 10 points or less
• (Over) Communicate
• Plan to scale

Key takeaways:

• Create a plan
• Decide how you want to measure
• Build buy0in with shared vision
• Test a pilot
• Iterate and move forward

When you empower others, you open up possibilities.

“Prototype” vs “Prototype”—Breaking Down and Rebuilding Our Understanding of What We Do

Saara Kamppari-Miller, Intel (Rapid Prototyping Advocate)

Breaking down and rebuilding our understanding of what we do.

"That's not a prototype"

• We spent time defending words rather than discussing the content of the work.
• Re-org with UX + industrial + product designers
  • Forget the labels (ignore the pain), let's hit the ground running.
  • Words matter
    • Complain about it
    • Do something about it

Team exercise - affinity diagram of what we do

The affinity diagram was an object of discourse.

Digitized the affinity diagram (!) - the team framework was in our sticky notes:

Team framework:

• We frame opportunities
• We research
• We group stronger
• We build, we design

Takeaways:

• Words matter. Don't ignore the pain.
• Make the intangible tangible.
• It's not about the diagram, it's about the dialog.

Theme 3: Change Management

Setting the Table for Dynamic Change

Jacqui Frey, Mailchimp and Alison Rand, InVision

"I decided I can't pay a person to rewind time, so I may as well get over it." - Serena Williams

History of change management (as a thing)

DesignOps as an Agent of Change

• Change requires more than a single vision, or a few projects. It takes careful orchestration throughout the organization to become a part of the working fabric of day-to-day operations.

Thinking in Systems: process design is also design

Change management core principles

• Start with a small group - don't come in hot
• Identify a foundational change

A GAME! https://www.menti.com/fu8f3qfibr

Even more importantly is to understand why we are making the changes we are making, and what are the tradeoffs.

Set the table, host the party.

Listen with tactical empathy.

Move thoughtfully and fix things.

Design is Not the Frosting on the Scaled Agile Layer Cake

Erin Hauber, USAA

How I went from angry to optimistic about a change

"Change is a constant in Design." - Meredith Davis

Design was moved again to the 'team' level - not as a core strategic asset. But this started to look like an opportunity.

Truths of engaging with change

• Empathy applies to us, too
  • Use a shared definition of value: desirable + compliant + viable + feasible
• Be the antidote - demystify design
  • HCD + SAFe + DE + AP | Design Thinking and SAFe

This is a movement...

Leading Through Change

Purvi Shah, Visa

Value of Design is known to increase and improve:

• Revenue
• Shareholder Value
• Speed to Market

DesignOps = Change = Hard

Levels of Design Maturity

• No design
• Design as style
• Design as process
• Design as strategic

"Great companies continue to aspire and drive thoughtful and informed change when they are strong. One way of doing that is by offering world-class user experiences." - Alfred F Kelly

• Focus on the user need delivers results.

Visa's design tenets:

• Unified culture , community of practice and visibility
• Invest in talent growth and capacity
• New ways of working metrics and measurements

For focus:

1. Understand the eco-system - understand the people, systems and processes
2. Set your vision + intentions, then work from within - find shared priorities, embed design, contact awareness
3. Get clear on motivation - get clear on the why. Know the human, understand the context, make it personal.
4. Pace yourself. Make time and space, be bold / have patience, check in often (them and you).

Learning Over Outcomes

Brenna Fallon, Google

It matters what you build BUT it matters more if you learn.

All I ever needed to know I learned in Montessori. Larry and Sergey were both brought up in a Montessori school. Montessori principles:

Independence

• Trainings grounded in self-discovery.
• 20% philosophy - 20% of time to do as you'd like
• OKRs

Order

• How do we tune the product design process for learning?
• Double-diamond method can be a flexible structure needed around the process - what's interesting is the first diamond:
  • What's the root problem we're trying to solve
• Be wary of having too many decision points for all research - will get too waterfall-y
• Loosely coupled, tightly aligned

• Research drives the product train

Planes

• Planes of development - where do you sit?
• Tuckman's stages of team development + re-org

Google's 5 dynamics of an effective team

• Psychological safety
• Dependability
• Structure & clarity
• Meaning
• Impact

The greatest sign of success for a teacher...is to be able to say, "the children are working as if I didn't exist." - Montessori

Closing Keynote: Amplify. Not Optimize.

Dave Malouf

Why are you a design operator? What is your origin story?

Dave influenced by Pratt & the spirit of Rowena Reed

• Interactions impacting the outcome of design.
• Models of contextual inquiry turned into discussion pieces.
• Design amplification...but what am I amplifying?

Design is an amplifier across...

• Production / development
• Engineering
• Business
• And design itself

What makes good design happen?

Great design happens because of great designing.

The value is in the verb, the action of designing.

Amplifying what?

• Form giving - we communicate through the forms we create
• Clarity - structure and navigation
• Behavioral fit
• Exploration - first diamond!

How might we make design happen better?

Design happens better when:

• Different activities happen at their most appropriate cadence.
• Intentional spaces & processes encourage serendipity, emergence, exploration.
• A balance of both quantitative and qualitative data are synthesized into actionable insights.