How to Use Human Insights in the Idea Stage of Product Development

One of the most challenging and rewarding tasks for a product manager is creating and verifying high-impact product ideas. Most people don’t think of using human insight at this point in the process, preferring to flesh out and iterate on concepts before sharing them with customers. But bringing fast feedback in at the very start of the process can give you insights that strengthen your hypotheses and reveal strengths, threats and opportunities that numbers and guesswork alone simply cannot. It’s never too early to benefit from fast human insight. Here’s how to use it in the idea stage.

For this article, we looked at Airbnb Categories. With the launch of Categories, Airbnb is trying to turn the travel planning process on its head, by having people start with the types of lodging or recreational experiences they want, and determining the destination from there. It’s a big idea that undoubtedly required a big investment on the part of Airbnb. 

Here’s how Airbnb’s home page looked before the change (late 2021). The emphasis was on having users enter a location and date range. There was also an intriguing image inviting them to explore (the images were swapped on a regular basis).

Below is the new Airbnb home page in late 2022. Users still have the option to start with a location and dates, but categories are now prominently displayed, and sample listings for a category appear on the home page. Rather than trying to entice you into exploring, Airbnb now drops you into the exploration process by default. It might seem like a small change at first glance, but it’s actually a dramatic change to the core functionality of a major website.

That got us thinking: How might a Product Manager have explored the viability of such a game changing idea early on? Instead of continuing to wonder, we decided to test some of the assumptions underpinning the Airbnb Categories concept ourselves. As a result, we gained insight into the promise and potential pitfalls of Airbnb’s big move, and how you can apply those learnings to your own planning process. 

What we learned about the concept and customer behavior

The Airbnb Categories model assumes that a meaningful portion of travel behavior is driven foremost by the desire to stay in a certain form of lodging and/or the desire to engage in a certain activity (as opposed to being in specific locations). Our first step was to test those assumptions. We encountered support for both of them within hours of launching a few UserTesting tests. 

However, if we concluded our exploration there, we would have missed out on some pivotal insights. Next we explored both if and how location-agnostic travel planning is done. Consumers’ descriptions of the latter were very revealing. For example, starting points for this form of trip planning included: 

• Recommendations from friends 
• Travel blogs 
• Tourism websites 
• Google Maps 
• Social media
• News and magazine articles 
• Blogs
• YouTube and Google

In addition, VRBO, Kayak, TripAdvisor, and, notably, Airbnb were also mentioned as starting points. 

As next steps, the location-flexible travelers we heard from considered: drive time, nearby amenities, the availability of activities in addition to the hobby they were planning the trip around, safety (especially for solo female travelers), weather, reputation and flight availability, before ultimately choosing a location. 

Overall, these findings pointed to the fact that, prior to the existence of Airbnb Categories, the location-flexible trip planning process often looked like this:

1. Identify locations that allowed for a given experience (which typically requires the use of numerous sources)
2. Narrow the list based on factors such as nearby amenities, flight availability, etc.
3. Choose a location
4. Choose accommodations in the selected location

While under a Airbnb Categories paradigm, it could look like this:

1. Choose an experience
2. Source accommodations in locations that allow for a given experience 
3. Narrow the list based on factors such as nearby amenities, flight availability, etc.
4. Book selected accommodations on Airbnb

In other words, Airbnb Categories positions Airbnb to control the entire  location-flexible trip planning process, consolidating a search process that otherwise would run off of Airbnb-owned channels, creating numerous opportunities for customers to buy from someone else.

The fact that many travelers reported filtering down from a set of location options based on their existing knowledge or relying on their friends and family’s knowledge, was also noteworthy. Put differently, if they or their friends and family weren’t already aware of the availability of cabins in Red River, NM, those cabins were extremely unlikely to be in contention for their cabin vacation. Under an Airbnb Categories paradigm, previously unknown supply would be more likely to factor into a traveler’s consideration set, a potentially major win for suppliers in those lesser known locales.

Additionally, our tests revealed that hobby-oriented travel planning is extremely personal. For instance, one of the travelers we heard from was a beginner surfer who planned trips to beginner friendly surfing destinations. This signaled to us that Airbnb’s hobby-oriented categories may require a degree of nuance to adequately serve travelers with varied skill levels tied to a given hobby. Otherwise, Airbnb may risk losing trust in its hobby-oriented recommendations.

Finally, our tests highlighted seasonal dependencies as a potential vulnerability for the Airbnb Categories model. As an example, one traveler in the video below described visiting Miami primarily to spend time swimming in the ocean, but found that conditions were not welcoming. Another described a last minute realization that they were planning a hiking-oriented trip to the Bahamas during hurricane season. Again, we were alerted to a scenario that may need to be addressed to avoid loss of trust.

At this point, you might be wondering how we unlocked so many insights with a simple UserTesting test? Better yet, you might be wondering how you can leverage this type of testing at the idea stage. In the next section, we will walk you through our process.

How we set up the study

Step one is to outline the assumptions underlying the idea. In the case of Airbnb Categories, assumptions may have included:

• A meaningful percentage of leisure travel is driven more by the availability of specific activities in a location than the location itself
• A meaningful percentage of leisure travel is driven more by the availability of specific lodging types (e.g. yurts) than the specific location of the lodging 
• Exposing travelers to supply in lesser known destinations will make Airbnb a more attractive platform for hosts in those areas
• Location agnostic travel planning is not unique to Airbnb customers

Next you want to consider which assumptions present significant risk to the viability of the idea. For example, the assumption that ‘location agnostic travel planning is not unique to Airbnb customers’ is risky because, if proven untrue, investing in Airbnb Categories would fail to grow Airbnb’s customer base. 

After identifying your riskiest assumptions, it’s time to test them. For illustrative purposes, let’s focus on the second assumption above (a meaningful percentage of leisure travel is driven more by the availability of specific lodging types (e.g. yurts) than the specific location of the lodging).

Building a Test Plan

Audiences

Travelers are a fairly broad population so we do not need a lot of filters for this test. In fact, we only set an age minimum of 22 (see screenshot from the test below).

Screener questions are much more relevant. Our challenge is that we’re trying to screen for people who plan travel unconventionally, but you can’t just ask that question directly so we have to back our way into it. 

Let’s start with a yes/no question, which we typically advise against, but we are going to make the ‘right answer’ counterintuitive (see screenshot below).

The first screener question is designed to screen out people who say yes to everything. While vacation home ownership is not necessarily problematic for our test, it’s fairly uncommon. Therefore, we lose few people by asking this question, but we’re likely to eliminate anyone who’s answering yes to all of the questions. 

In the second question, we explore the behavior underpinning our hypothesis but set the questions up in a multiple choice format that accepts more than one answer per respondent. In keeping with best practices, we won’t allow people who selected every option to be screened in. While we risk losing some people who are good fits for our test as a result, it keeps the overall quality of responses high.

In the third question we inquire about their travel frequency and screen out travelers who have paid for vacation lodging less thanthree times in the past 12 months. 

Test Questions 

Now let’s dig into the travelers’ experiences to ensure the reliability of their self-attestation and to understand how and where they performed the tasks associated with the location-flexible travel planning process. To avoid generalities, we’ll ask the respondents to focus on their most recent experience with destination-agnostic travel planning. Our goal is to get them to tell the story of their location-agnostic trip planning process (see screenshot from our test plan below).

We don’t want to assume that their location agnostic trips were successful and/or miss out on other valuable lessons from them so it’s important to ask about how the trips went as well.

Given that their most recent destination-agnostic travel planning experience may have been an outlier, we also include a question about other destination-agnostic travel planning experiences that they may have had.

Finally, Live Conversations (moderated interviews) give us the opportunity to dig into anything of interest further, so we end each test by asking if the traveler is open to participating in one.

Conclusion

Leveraging a human insight platform like UserTesting at the idea stage enables product managers to build on quantitative insights with a richness that only personal narratives can provide. Not only did our Airbnb Categories test signal support for key underlying hypotheses, but it revealed strengths, threats, and opportunities that quantitative surveys and metrics alone simply cannot. Armed with this more nuanced data, an Airbnb PM might consider exploring integrations with weather, drive time and flight availability data providers, and/or set up traveler review monitoring for issues related to seasonality or skill level misunderstandings. 

In summary, while the idea stage is early on in the product development process, it’s not too early to benefit from human insight. 

Photo by Mesut Kaya on Unsplash

The opinions expressed in this publication are those of the authors. They do not necessarily reflect the opinions or views of UserTesting or its affiliates.