A Product Executive’s Guide to Human Insight

The problem: How do you move faster and make your products better at the same time?

The biggest limit on the productivity of a product development organization is the tradeoff between speed of development and customer delight. Delighting customers requires consistent, in-depth insights into their thinking and needs, but the traditional sources of that insight, such as focus groups and customer interviews, move far slower than an agile development process. 

Because of the speed mismatch, many product teams compromise on human insight. It’s common for product owners to base decisions on their gut instincts, or to substitute indirect sources of insight like analytics and social listening, which are limited in scope and can produce misleading findings. The problem is very widespread: About two-thirds of product decisions today are never validated with customer feedback before they’re finalized.¹

This guesswork creates big risks for your organization:

• You waste sprints in pursuit of the wrong solution
• The morale of your engineers declines when they have to do rework
• You may lose sales to competitors because another company hit the sweet spot first

The faster you try to move, the less time you have for customer insight, and the worse the risks become.

The solution: Fast human insight lets you optimize both quality and speed

New methods of collecting human insight enable product teams to apply fresh customer insight in real time to every decision they make, without slowing down. A product team no longer needs to trade off between customer delight and speed; it’s possible to optimize for both. This has very substantial benefits to a product organization:

• Problem discovery is continuous, so your teams can start faster with a better understanding of their critical success factors
• Your prototypes are perfected with high confidence before they’re handed off to development, resulting in less rework
• You save money and time because you’re no longer wasting sprints
• Your developers are happier and more loyal because they’re spending time on new problems rather than rework
• When there are problems in a shipping product, your teams quickly form much higher quality hypotheses on what needs to be fixed, increasing their success rate on the first try
• A constant flow of customer insight permeates the organization, so you and your managers have to spend less time aligning people. This reduces waste and lets you move faster.

The impact on product organizations can be very substantial:

• Chartio increased the rate of customer task completion from 10% to 80%.
• Quip completed up to a dozen iterations in a single week.
• Citrix Convoi enabled everyone in the product team to gather human insight, increasing productivity.

This document tells you how to add fast human insight to your product development process. We’ll describe the problems you can solve, where in the process you should apply insights, best practices for deploying them into your organization, and how to resolve the most common speed bumps you may encounter along the way. 

We’ll start with the specifics on the problems you can solve…

Where to apply human insights: everywhere

The speed and convenience of modern human insight lets you use it differently. People schooled in old-style market research, which is too slow to integrate in most decisions, assume they can use it only in only one or two stages of the product development process, for example problem discovery or prototype validation. Fast human insight lets you apply customer insights continuously across the development process. You should think of it as the grease that you can apply to any decision point: It enables everything to move faster, with higher confidence, less friction and lower likelihood of breakdowns. 

Here are the most common ways that product teams are improving their productivity through fast human insight:

In the discovery phase

When you’re understanding customer needs and problems, fast human insight lets you do much more accurate and detailed discovery, faster than traditional interviews. Teams often shortchange the discovery process because it’s very painful:

• It’s often very slow and difficult to recruit the right interview subjects
• Scheduling is tedious and plagued by no-shows
• The interviews themselves are very time consuming
• Because of those barriers, teams often compromise by interviewing people who aren’t exactly the target customer 
• Teams sometimes interview only current customers rather than the new users you’re most anxious to reach
• Because of logistics problems, a team may interview only a small number of repeat participants, or may recruit in only a single city rather than nationwide or globally 
• Teams may try to extract feedback from from social media rather than interviews, in which case you’re at risk of optimizing for fanatics instead of average customers

The result can be products that have far too many features, are overpriced, or that simply fail to engage the average customer.

A good human insight solution will let your team recruit and schedule discovery interviews automatically in less than 24 hours. You should be able to target your specific customer in any geographic location, and reach users and non-users alike.

The outcome: Discovery can be completed within a single sprint. It happens in days rather than weeks or months, and you’re very confident that you heard from your real customers.

In the design phase

Human insight should be used continuously from the very start of ideation. The sooner you detect a mistake you’ve made, the easier it is to fix, and the less money and time you will have wasted going in the wrong direction. 

Engineering time is the single biggest expense item in most development organizations, so you can save enormous amounts of money by ensuring that you don’t send the engineers off to implement something customers will reject.

If you do prototypes before development, you should insist that your teams validate their earliest sketches and ideas by running them past real users. As they iterate, the tests should be repeated. Almost anything can be tested: A written description of a product, a verbal description, even a napkin sketch (literally; we’ve seen companies test photographs of napkin sketches, and it works). This process takes a couple of hours per test if you use a modern human insight solution, and if you get the right solution your designers and PMs don’t need the help of a researcher. 

Ironically, most companies wait until the end of definition before they test things on customers. Generally that translates to running an acceptance test on a high-fidelity prototype just before it gets handed off to development. That is the absolutely worst time to get feedback, because it’s usually too late to change anything.

If you don’t prototype, fast human insight enables your team to learn even faster than they can by deploying code. Rather than waiting for analytics, your team can put a build in front of customers and get reactions to it within a couple of hours, and the feedback will include both what they feel and why they think that way, making it much easier for the team to determine what to change next.

In the build phase

Your teams should use fast human insight whenever they have to make a change to any customer-facing element of the product. For example, it may turn out that an interface widget created by the designer is too difficult to implement and needs to be changed. This sort of change can severely affect the overall customer experience, but you won’t know it unless the change is validated.

Align teams and settle disputes. One of the most difficult tasks for a product manager is aligning all members of the team around a unified vision. The challenge is not that people resist the shared vision, it’s that people naturally focus on slightly different interpretations and so they will think they’re signed when actually they are not. Often you won’t know this is happening until their work doesn’t align or there’s a dispute over something that you thought was settled. The customer videos produced by a human insight solution are very effective at building a shared understanding, because everyone can see the same customers talking about their needs and desires. This produces a shared vocabulary and mindset that helps the team move faster as development proceeds.

Human insight can also be very useful for settling disputes in organizations that have a diversity of opinions on the right direction. Best practice in Agile is always to have a single decider on a project who is empowered to end any disagreements. But let’s face it, some company cultures require more discussion, especially if the disagreement is with a stakeholder who has a senior role in the company. These situations can delay a project and damage employee morale. Fast human insight can usually settle the disagreements within hours, without extensive argument, by providing objective video evidence of how actual customers react to the issue. We all aspire to operate in a company culture that makes decisions crisply, but when that doesn’t happen, fast human insight is a powerful way to herd the cats.

Prior to launch

If the project has been validated along the way, you won’t have to do many end-of-development tests. But there are a couple of exceptions:

• If you’re integrating with another module or experience developed elsewhere, it’s important to validate the overall first-time user experience. In one real-world case we’ve seen, one team developed the product, but another part of the organization developed the onboarding experience. They needed to validate the overall onboarding + product experience prior to launch. Preferably that should have been done earlier in development, but sometimes that doesn’t happen, so now is your last chance to do it.²
• If you’re distributing a mobile app through an app store, we strongly recommend using human insight to validate the elements you’ll be using in the app store listing: screen shots, icon designs, text, etc. Launching a mobile app is like launching a movie – you don’t get a second chance to make a first impression. So you want to get this right, and you want to work the details. It’s surprising what big changes in adoption can happen due to subtle changes in the listings.

Post-launch iteration

There are two areas in iteration where you can increase productivity:

• Diagnosing issues identified by analytics. After launch, the analytics may tell you that something is going wrong. Typically you’ll see that customers are behaving as expected until they get to a particular point in the experience, and then they’re either going someplace unexpected, or they’re dropping out altogether. For example, picture a checkout process where many customers are abandoning their carts when it’s time to enter their shipping address. Although analytics packages are great for identifying these problems, they don’t do much to diagnose them. Your team may have to run several experiments to figure out what’s wrong. It’s much faster to run a quick human insight test, take people to the problem step, and then ask them to describe what they’re thinking. This will enable your product team to make a high-quality hypothesis on what’s wrong, so they have a good chance of fixing the problem on the first try.
• Improving the efficiency of A/B tests. Depending on who you ask, between 50% and 80% of A/B tests fail to give a statistically significant winner.³ That can waste a lot of time, especially if you’re testing a part of the app or website that has relatively low traffic, and therefore needs a lot of time for a single test. Using fast human insight, you can pre-qualify the alternates for an A/B test in hours, identifying the ones that are most likely to succeed, and flagging any problems in wording or design that might have caused an otherwise good alternate to fail. This can significantly increase the success rate of your A/B tests.

How to deploy fast human insight in your organization

Now that you understand the power of fast human insight, it’s time to put it to work in your organization. Like any other powerful transformation, this is a journey in which you’ll need to lead your team and overcome obstacles. Here are the best practices you should follow, and the speed bumps you should expect along the way:

1. Recognize that it’s a transformation, and set your expectations accordingly

The biggest mistake companies make when deploying fast human insight is to assume magic will happen automatically if they deploy a solution. Fast human insight isn’t just a product, it’s a change in the way people work. It requires tweaks to work flows and job definitions. Your people will need to develop new habits, which takes time to implement. You are a key element in this transition – you’ll need to be firm, you’ll need to enforce the right behaviors, and you’ll need to be patient with the process. 

2. Shift your researchers’ focus from answering questions to empowering others

Research teams come in a variety of titles: UX Research, CX Research, Customer Research, Insights Team, Market Research, etc. In a traditional organization they’re all expected to provide a service  in which employees ask them questions and they provide answers. If you have one of those teams in your organization, the first step in the insights transformation is to alter their service mentality. A transformed insights team works as a leader and facilitator: They split their time between doing the most strategic research on their own, and helping other people in your team get their own answers self-serve.⁴

The reason you need to change their role is simple math. Most product decisions today are not vetted with customer insight at all, so there’s an enormous gap between the latent need for insight and current research capacity. You may not even be aware of this gap because in many cases teams have assumed that they can’t get help, and have stopped asking. Gathering insights for every important decision would overwhelm the typical customer research team, which is usually overbooked already. 

Even if you’re willing to fund a massive increase in customer research, the process of going to a researcher, asking for a project, and then waiting for results is usually too slow for most day-to-day agile decisions. By the time you get the results, the team has already moved on. The current system simply will not scale.

The only sustainable solution is to empower product managers, designers, and other squad members to gather tactical human insights on their own: reviewing prototypes, discovering needs, and validating decisions in the moment when they are being made. A good human insight solution will enable this sort of self-serve testing. But you still need the research team. They will continue to drive the most complex, strategic studies, and they will also supervise the tests run by others to ensure that they’re being done properly. 

This can be an uncomfortable change for a research team that sees itself as the source of all customer insight. They may be rightfully worried that non-researchers will run bad studies or may misinterpret their results, causing the company to make mistakes. What they need to understand is that the company is already at risk because so many decisions today are based on guesswork or misleading sources of information. The business risk of an occasional mediocre study is far smaller than the business risk created when people routinely guess with no human insight at all. 

When a research team switches to facilitation mode, it follows three steps:

• The research team identifies the most complicated and strategic studies that drive the business, and continue doing those on their own. You’re not looking to remove the research team, just to focus them on the most impactful issues.
• The researchers create and champion a self-serve approach for the remaining tactical tests. Those usually include prototype tests, usability tests, and basic discovery interviews. The researchers’ role is to create best practices and train and coach non-researchers as they learn to collect and interpret those insights. Over time they can hand off more and more responsibility to them as they come up to speed.
• Your researchers will also continue to play an important quality control role in the company’s human insight testing, by determining who has the right to conduct tests, and what level of supervision each of them need. A good human insight solution will have workflow controls built in to enable this supervision.

We’ve written an extensive guide to help research teams make the transition from info controllers to facilitators. You can find it here.

3. Transform one process at a time, and keep it simple

Start by changing a single process. Because human insight is so powerful, it’s tempting to deploy it for many usages across your team all at once. We don’t recommend that. The hardest part of deploying fast human insight isn’t standing up the software, it’s changing the work habits of the people who will be using it. That needs to be planned and executed in stages. 

We recommend that you start by transforming a single part of your org’s workflow. For example, you could start with improving discovery interviews, since they are a problem area for almost every organization. Or if your dev process includes a design stage, prototype validation is another great starting point. Whichever starting point you use, your goal is to:

• Define how the current workflow must be changed to include human insight (for example, exactly what do you test, when do you do it, and what do you do with the results?)
• Train everyone in the new process
• Make the switchover
• Enforce use of the process, and
• Celebrate wins so the process becomes second nature to everyone

Once you’ve accomplished that for one process, you can move on to the next process you want to transform.

Starting with a single process has several benefits:

• It reduces the workload on the people who will be training and supervising the change
• You can test out your monitoring and approvals process to fix the inevitable kinks and challenges  
• You can monitor adoption very closely to make sure it succeeds
• It reduces the effort you must put into motivating the team. Once your team members have seen the benefits of fast insight in one area, it will be much easier to get them to adopt human insight in other processes

Keep the training simple. Another very common mistake companies make when rolling out a human insight program is to attempt to turn non-researchers into full-fledged researchers. This usually fails for several reasons:

• There’s a huge amount of training needed to become a full-fledged researcher. People often get master’s degrees in this subject; it is unrealistic to expect that a few in-house training sessions, no matter how compelling, can produce the same level of expertise
• Not everyone is qualified to become a full-fledged researcher. It requires a particular mindset to do the job well.
• Most of your team doesn’t want to become full-fledged researchers. They’re already very busy with their day jobs.

If you try to teach full research skills to non-researchers, you’ll see high dropout rates from the training sessions, and low adoption rates across most of the organization (the only adopters will be the small subset of your employees who were already research groupies).

Instead of trying to turn people into researchers, you need to configure the product so it meets them halfway. Your research team can create templated tests and other support material to make it easy for non-researchers to collect insights on their own. You then teach people to use the template, rather than teaching them generalized research skills. This is a much shorter training process and results in higher rates of adoption.

The most time consuming part of running a human insight test isn’t launching the test, it’s interpreting the results. So you should deploy a solution that makes it extremely easy to identify key findings without slogging through hours of user video. Features to look for include:

• AI-driven automatic analysis of user sentiment and reactions
• Clickable visualizations of test findings
• Automated creation of video clips and highlight reels (collections of clips that your team can share with others)

For example, the training for a designer could focus on how to use a templated test to gather feedback on a prototype, and use of the visualizations from the test results to analyze the findings. Designers can easily see the need for this (so they will be highly motivated to make it work), and when the system is properly configured, it will be fairly easy for them to do the work themselves. This will put you on the path to high adoption.

4. You need an insight ops function

You need someone inside your team supervising the operational details of your human insight program. This person (or persons) will do the following:

• Set standards and workflows. This is the single most important ops role, but it’s also the one that most often gets neglected. Every organization does agile slightly differently; you need someone to map out exactly when and how employees are expected to run tests within your workflow. These decisions can be complicated and tense because you’re changing the daily behaviors of employees, so your ops person needs to be a diplomat as well as a planner. For example, you might require that a prototype can’t be transferred to the dev team until it has been validated with a human insight test. You need someone to work out exactly how that will be enforced, and to answer the inevitable questions about edge cases that always come up.
• Schedule and conduct training, and assess skills of your employees
• Based on that evaluation, set the access privileges for each user. Who is free to create a test on their own, who needs help, and who should be allowed to use only templates?
• Configure the overall system. For example, creating workgroups within the human insight system so employees focus on the tests relevant to them. In a good human insight system, you can also hide confidential tests so they can’t be seen by the full organization.
• Create test templates and participant groups. A good human insight solution will let you create your own test templates, and also to predefine demographic groups for use in a test. So, for example, you could create predefined groups for each of your target customer profiles. You employees could then launch a test on a particular target with a single button press.
• Monitor adoption and usage.

A good human insights solution will have built-in support for all of these functions. Your vendor should also give you the option to outsource part or all of the insights ops role if you don’t have anyone in house who can do it for you.

5. Plan for backsliding

It’s inevitable. We all want to believe that a compelling new approach to decision-making will be enthusiastically embraced by everyone in the org, but the reality of human beings is that they’re very wedded to their old ways. Once the excitement from your initial deployment of human insight wears off, they will tend to revert to their familiar old business practices. 

Also, you should recognize that when you ask employees to validate their work frequently through human insight, you’re asking them to do a little bit of extra work in the moment to avoid the risk of much bigger mistakes later. It makes sense in the long term, but most employees don’t live in the long term. Even if it takes only an hour to validate a decision, that’s still an hour they could have spent doing something else. Making blind guesses is always going to be the fastest methodology of all. It’s also the riskiest – but the more self-confident your employees are, the more they’ll believe they can guess reliably without any need for validation.

To overcome this natural resistance, you need to deploy a series of carrots and sticks:

• As the leader of the organization, you should publicly celebrate the wins from the new approach. Have people show their highlight reels in meetings, share examples of decisions that were improved through human insight, and make a point of praising this work.
• Have your teams create watch parties in which they get together to watch videos from human insight tests and discuss what they mean. The watch parties are a great activity when everyone is in the office, but you can also do them remotely (use the chat function to get people commenting on the videos as they’re playing).
• Have your research team create an “empathy feed” featuring the most compelling customer comments from all the tests done by the organization, and share that around the company
• Require employees to show their work. The more often you ask, “did you do a human insight test on this?” and the more frequently you ask to see clips from the tests, the higher adoption will be.
• It’s essential that testing be required as a part of your team’s workflows. That’s why the insight ops role is so important.

6. Get help

You should expect your human insight vendor to stand with you every step of the way as you transform your processes. Even if you have robust research and ops teams to manage the journey, your vendor should offer documentation, deployment guides, onboarding, and ongoing support services to help them guide you through the transition.

If you don’t have a robust team to drive the process, you should expect your vendor to be able to step in with consulting resources that fill the gaps. Those can include:

• Supplementing the capabilities of your research team as they transition into their new facilitation role
• Helping you craft your new insights-driven processes and the best practices for enforcing them
• Assessing the skills and training needs of your employees, and providing that training
• Creating templates and premade participant groups for use by your team
• Configuring your human insight system: creating workspaces, and managing the usage privileges of your employees

Next steps

When you’re ready to start your insights transformation, you should start talking with vendors to understand what your options are. If you already have a vendor doing UX research for you, they may offer scaling services. Another good place to look is G2, which aggregates customer reviews of business software. Their section on user research includes companies that range from very simple and cheap to full-fledged enterprise solutions. 

As your team speaks with vendors, you can use this document to give you ideas on what questions to ask. (link to doc on “how to select a human insights company.”)

You’re also welcome to post in the comments here if you have any questions. We also welcome suggestions on how to improve this guide.

References

1.  For example, only 38% of high-fidelity prototypes are validated with customer tests before they’re moved to production. Only about a third of product feature decisions are validated with customer feedback before they move to production. Source: 2022 Customer Experience Survey, UserTesting.
2.  First-time experience testing is especially important for mobile apps, because of the dynamics of app stores. Launching an app is like launching a movie: If the first weekend’s reviews are bad, you’ll have a very difficult time recovering. So you need to be certain the first-time experience is perfect. Fast human insight can answer that in hours.
3. https://www.insiderintelligence.com/content/most-a-b-tests-don-t-produce-significant-results
4. If you don’t have any customer research function at all in your org today, you’ll need to hire or identify people who can supervise your rollout of fast human insight. These might be internal employees or outside consultants. A good human insight vendor will also offer consulting services to help you with the transition.

Photo by Alvaro Reyes on Unsplash

The opinions expressed in this publication are those of the authors. They do not necessarily reflect the opinions or views of UserTesting or its affiliates.