Research Operations Done Right

MM: How did you get into this role? And how did your skills evolve along the way?

AL: I found myself in UX research and research operations by accident. Before jumping into the UX field, I was a research lab manager at The University of Texas at Austin. Because of that job, I had some relevant research experience on my resume which helped me get my next job. I was recruited as a contractor at Google where I was a UX research participant recruiter. I worked with researchers from several Google business units, recruiting participants for research projects – that was solely my job. There were like 60 of us (recruiters) because Google has a very large UX research function. I was managing timelines, scheduling participants, and handling logistics for eight to ten research projects at any given time. It was a lot and probably more than one person should have been doing at one time. But because I did it so much, I started to optimize the way I was recruiting participants and started to socialize my methods, management started to take notice and assigned me more ops related tasks, such as improving the recruiting process, updating our guide on conducting phone screens, and streamlining kickoff meetings with researchers, etc.

That’s how I got more into ops, by taking on more responsibilities. I was there for two and a half years, and decided it was time to go because the burnout was real and I was ready for more responsibility. I jumped over to Indeed as a UX research coordinator. I did recruiting, but at a much smaller scale. There were only four of us and we were really trying to revamp the ReOps team to be more functional. We updated templates and best practices, established better working guidelines between ReOps and researchers, and began to build better relationships with our cross-functional partners. I worked a lot with legal and privacy during my time at Indeed to ensure that research was being conducted ethically. While doing that I helped streamline how ReOps worked with legal and privacy to make sure that duplicative work wasn’t happening, by creating one-pagers and documenting these conversations better.

So that’s when I really started to home in on ops. I realized that I liked building infrastructure for teams, and improving processes that would benefit cross functional teams. It can be a pain at times, but I do enjoy it.

Then, an opportunity arose at Indeed because they were starting to build out the social impact research team. I went on a three month rotation with this new research team to help build out their recruiting processes and governance. That was a difficult task because they wanted to speak to job seekers facing barriers, such as no college degree or bias due to age, ethnicity, disability, race, veteran status, or criminal record. I worked with multiple cross-functional teams and built their participant management playbook for the team and helped kick off a couple of research projects.

After the rotation, I negotiated my way onto the team full-time as a UX research project manager. This gave my career the push it needed and boosted my confidence.

Then I saw H-E-B was hiring a research ops program manager, the first one ever at H-E-B. I applied, interviewed, and got the job! I was a bit nervous because it was a big leap for me to go from a project manager to a program manager. At the start of my role at H-E-B, I established the ReOps frameworks and programs. I now set the quarterly plans, execute on the work, and at times oversee others who contribute to ReOps work, and report out metrics. It is challenging and stressful at times but in a good way and I’m learning as I go!

MM: I need to ask a pronunciation question. For some reason I had assumed that I would shorten research operations to ResOps. But I think you’re saying ReOps. Is that the proper term?

AL: It used to go by “ResOps” and I think some may still refer to it as “ResOps”. If I recall correctly, research operations was called ResOps at Google and Indeed. But then there was a transition at some point and the community started to refer to research operations as ReOps. I think it might’ve been when the ReOps community website went live – which I highly recommend visiting. Lots of research practitioners have established the pillars and frameworks of research operations. It serves as a great resource for anyone building out a research operations practice.

Research ops is more than participant management

MM: When we were doing the prep for this interview, you talked about how the ReOps role is perceived. You said people tend to have a very narrow definition of it. What are you hearing from other ReOps practitioners? What’s the stereotype that you’re dealing with?

AL: The stereotype, for the most part, is that ReOps is participant management only. While it’s a crucial part of the job, ReOps is so much more.

I want to be clear that to some extent, I do understand the perception. Managing the participant experience is important to conduct good research but it’s tedious and comes with its challenges.  The responsibilities consist of managing expectations with both the researcher and the participants, sending initial outreach emails, conducting phone screens, scheduling participants, sending email confirmations and reminders, and incentivizing participants. It’s a big job that often gets overlooked or doesn’t get the recognition it deserves.

I think this stereotype is widely promoted due to job postings and job descriptions. You’ll find that the majority of ReOps jobs have recruiting and managing participants as a core part of the role, but most ReOps professionals do much more.

Elements of a full research ops function

MM: So that’s the stereotype. Actually, stereotype is probably not the right word, because you’re saying there’s a reality there. But what should they see the role as? What’s the broader story?

AL: To successfully recruit participants, someone has to build these processes from scratch while working with cross-functional teams. All the templates and best practices, including kickoff meetings, screeners, surveys, email communications, one pagers – the list can go on and on. This is what I mean when I say ReOps is much more than managing participants. I think that ReOps should be more a part of the research strategy at companies. 

Aside from this, ReOps also has a hand in tool management, research compliance and governance, budget, socializing research, and research repository maintenance. I’m sure I’m forgetting much that I do and other ReOps professionals do, too. 

Another part is managing the tools that research teams or anyone conducting research may use. For example, H-E-B uses UserTesting for quick recruitment and on-demand testing. I’m an admin and help manage our seats, oversee our usage, and take part in the contract renewals and negotiations. Aside from this, I’m also on the lookout for other UX tools that the research team can leverage and I take care of onboarding and implementation of those tools.

ReOps also helps create templates for the researchers, or anyone that conducts research, such as research plans, screener templates, screener question banks, reports, etc. This is done for consistency’s sake and so no one has to start from scratch when they want to conduct research.

I want to also circle back to research repositories because this is important. After research is done, research readouts or reports need to live somewhere. ReOps may help build out a repository, the process for adding to it, and oversee the maintenance of it. This is important so teams can have access to insights that can drive new work and help make data-driven decisions.

Lastly, process improvement is a big thing I do. I’m constantly looking for feedback on process inefficiencies. A process could work for one year, and then if your company, org, or team grows or scales, it’s important to revamp a process and streamline it to fit the needs of the present.

How to start a research operations team

MM: If somebody reading this is just now setting up a ReOps function, do you have any advice for them?

AL: I would say take the time to understand your specific org and the research it does. Every research function is different across companies. So, take the time to go on a listening tour of any current pain points, create a baseline survey, send it out, and synthesize all of the thoughts and opinions. From there, you can prioritize the work as immediate needs, nice-to-haves, and stretch goals. Plus, this will immensely help to build out a quarterly and yearly roadmap.

When I joined H-E-B, I did go on a listening tour with the researchers but I also wish I would’ve talked to more of our designers as well. If I could go back, I would have spent more time to fully understand the pain points or the immediate needs of the entire org. Just taking the time to listen is key, because at the end of the day, your researchers and designers are your stakeholders. Getting them to trust you is important.

How to grow the role

MM: Imagine that this is being read by somebody who’s new to a ReOps role and who’s in that pigeonhole where the role is just recruiting management. Any advice for them on how they grow the role beyond that? How do you help the company see this?

AL: I think it’s a two-parter. First part: I think asking for more responsibility is sometimes key to growing in any role. When I look back, that’s how I got my start. I just knew I wanted to do more and I knew I was capable. I noticed things I could improve that would benefit not only me, but others on my team. Participant recruiting is an important part of research, but it can get very mundane and can become routine quickly.  So during my spare time, I sought out feedback from those doing research, on what was working or not working. I would simply ask, “Can I try something different? Just let me take a stab at it and see if you like it.”

Second part: I think a huge part about ops in general is building trust. Show that you are paying attention and listening by producing work that reflects past pain points. From there, competency will come through on its own and people will most likely come back for your help or include you in new initiatives or projects.

The role of templates in research: a starting point

MM: You’ve mentioned templates a couple of times. A lot of researchers will say they like the idea of templates, but when it comes to actually doing research they always end up creating from scratch anyway. What have you seen? What is the role of templates? What should somebody expect the level of adoption to be? Who are they for?

AL: I don’t think anyone can make a template that’s going to fit the needs of all. A template should be more of a strong recommendation, so someone doesn’t have to start from scratch. I think all templates should serve as a baseline that others can tweak to serve their project’s needs.

I love templates. I’m a template girlie. I would like to think they make people’s lives easier.

I also think templates could help with brand consistency and help keep things clean. Say you’re starting a new team, and you want to be seen as polished and organized. If you create templates, other teams may notice that and sometimes say “Hey, that’s the research team’s work” because the documentation is recognizable. 

MM: So ReOps plays a role in that branding of the research team. Is that what you’re saying? 

AL: I think so. I wouldn’t say it’s a mandatory part of the role but definitely a project someone in ReOps can take ownership of. I work very closely with a research manager who is a fantastic designer, so he mostly rebranded the research team with logos, icons, and Confluence page banners on all our documentation. Because of this I’ve heard comments from across the company that they really like our documentation space and I think the research team comes off very polished.

Reporting structure: The advantage of teaming with design ops

MM: Speaking of the people you work with, talk to me about your organizational structure. How do you fit in the organization?

AL: I’m part of the DesignOps team but I am matrixed out to the research team as their very own ops person. For the most part, I’m the only one in my team who works on anything research operations-related. I work very closely with our two research managers and researchers on all ReOps efforts, so I’m very intertwined with both teams.

MM: Do the UX researchers report into the design org? Or is that a separate organization? 

AL: They report into the design org at H-E-B. 

MM: And then is design part of the product organization? 

AL: We have a product org, an engineering org, and a design org and each of them have directors/VPs.

Do a listening tour

MM: Think about all the stuff you’ve done as a ReOps person. Is there anything that you’re especially proud of, like a project that really went great?

AL: When I started at H-E-B, there were no research managers in place. I was expecting to work alongside a research manager and have a partner of sorts, and when that was not the reality it was stressful at times. I definitely had a lot of “Oh, no! Can I do this!?” moments early on. At the time, it felt Iike I was putting out fire after fire, so that wasn’t fun. But my manager, the design ops team, and the researchers really came through and heavily supported me during this time. So I was very proud of myself for establishing the research operations practice mostly by myself with little to no guidance from research leadership at the time.

I was just going off what I thought was best, taking stuff from my past companies that worked well, and leaning on the ReOps community website for a lot of what I was building. Now, I work with two research managers and they’ve really helped shape the ReOps programs as well.

If you would like something very specific that I’m proud of, I created a recruiting playbook at Indeed that was like 90 pages and covered four different recruiting processes. It included step by step guides with pictures, templates, tips and tricks, team POCs, etc.

MM: Do you also get involved in the training of people, or is that somebody else? For instance, I’ve talked to some financial institutions that have all sorts of rules about what you can’t ask somebody, because it violates the federal privacy rules. So training in that sort of process, is that a ReOps thing usually, does somebody else do that?

AL: Yes and no. The design ops team has created a fantastic onboarding guide for our new partners that join the design org, and it goes over everything they need to know when joining.

When a new researcher joins, I am in charge of completing their onboarding guide, so I’ll take the time to include any other specific research information such as our confluence space, storage space, and info on how the team functions. I do try to make myself readily available to them during their first few weeks.

How to set boundaries

MM: So it’s not so much formal training, but it’s onboarding. We talked about big wins. Now talk to me about the pain a little bit. Were there struggles? And how did you deal with them?

AL: This is a very specific struggle: Joining a company with no research managers in place was hard. I had to remind the research team that I wasn’t a research manager and had no experience as a researcher. Sometimes, I just felt like I was failing them. So, that was a bit of a personal struggle. 

Another struggle I see within the role itself is creating boundaries on what the role is and isn’t. Throughout my ReOps career, there have been times where I’ve felt like I was viewed as an assistant. There’s a lot of little requests that come my way that I can do but I shouldn’t do because it’s technically not my job. I’ll usually guide someone to the information they’re looking for. But the struggle is creating that boundary, and not being seen as, “Oh, they didn’t want to do that.” That can be hard to get past sometimes. But in this situation, I’ll usually gently remind people what research ops is and isn’t.

MM: If somebody’s stuck in that situation, any pointers on how to deal with it?

AL: I think just be polite and respectful. Say no to a request that might not fall under your role, but direct them to the resources they’re looking for that will help them complete the task.

How to measure success: Do they use your process?

MM: How do you measure the success of a ReOps team?

AL: Considering all the time and energy we take in planning and trying to create things that are easy to follow, the thing I measure the most is “Are the resources being used?” That’s how I measure my success. I ask myself a lot: Did we make something digestible and accessible? And did I make it easy enough that someone thinks, “wow, this was so helpful”?

There are easy metrics to pull from. We use Confluence, and so I’m able to view how many times a page was viewed. 

Also, as a program manager, I plan the work and report it out quarterly. Because of this, I can see the planned work against actual work delivered per quarter. This makes it easier to track, document, and visualize all the work that was produced, so I see this as a way to measure our success as well.

Market research vs. UX research

MM: Let’s talk about market research versus UX research. Are they totally separate worlds that never talk to each other? Is there overlap? Are they growing closer over time, in your perspective?

AL: I like to think that we’re growing closer to each other. I view market research as more business-oriented research. Their focus or goal is more sales -driven, so more of the “How do we make the business more revenue?” approach. UX research is a little more focused on the user, optimizing the product design, and creating accessible experiences. Both are important but the research objectives and goals are just different between the two practices.

At H-E-B, our market research team takes the needs from across the entire company, whereas for UX  research, the work is usually driven from product. So they definitely have more requests, but we’re slowly leveraging each other more and more.

Our market research team has our database of participants to recruit from, so I wanted to build that relationship, gain their trust, and be viewed as a partner. At the end of the day, we can both help each other by sharing knowledge or insights that the other may not have. Prioritizing building relationships between research teams at companies is so important.

MM: Do you think ReOps in particular should be playing a role in that? Should you be some of that glue?

AL: I think so, I played a big part in this. When I joined, I met with several members from H-E-B’s other research teams to start to form these partnerships. I set up a monthly meeting with some of their members as a way to keep each other updated on the work we’re each doing and to keep tabs if there was ever an opportunity for us to collaborate.

MM: So did you end up both sharing the same participant database? Do you have one for the whole company now? 

AL: They own it and when the design research team would like to use it, we send a request to see if their team has the bandwidth to help support the research project.

Lessons in democratizing

MM: Let’s talk about a subject that comes up with a lot of companies, which is democratizing – by that, specifically, I mean empowering non researchers to be able to do some of their own tests. Have you been involved in any of that?

AL: Yeah, definitely. As I’ve built out my ReOps programs, one of them is specifically called ‘Design Research Democratization’. That program’s focus is creating playbooks, templates, and resources for anyone conducting research in our org. We’re a small research team compared to the rest of the org, so being able to hand off usability testing is the goal. 

We’ve created research resources, guides, and best practices to enable designers to do their own concept or usability testing when they can. This allows for the researchers to focus on more foundational, generative, or exploratory research.

Overall, I think documentation is important to democratize research because this also sets up some sort of guardrails for those that will be conducting research.I think it’s important to make documentation easy to follow, to try to leave out any jargon that would be hard for a non-researcher to understand.

In my opinion, creating documentation is easy. Maintaining documentation is the hard part. Staying on top of auditing and updating everything you create should be planned into ReOps work as well. So, constantly ask yourself “Do the resources that we created still make sense? Are they still efficient? And are we staying up to date with the latest research methods, techniques, and strategies?”

Another thing we’ve done to democratize research was hosting a lunch and learn for the design org on UserTesting and how to use the platform. That was really helpful.

MM: So how’s democratization worked out for you? Are you satisfied with the progress that you’ve made? 

AL: I think it’s going well. Before we hosted the UserTesting lunch and learn, we already had some research savvy designers that were using the tool, but usage of the tool definitely increased after the lunch and learn. That was nice to see. 

Building out and updating resources is a constant right now, but what may be even more important for democratization is communicating that these resources exist. This requires sharing this information multiple times and reminding others that these resources exist over and over again.

Being open to feedback after the release of research resources is also key to democratization, for me at least. Everything created can be iterated on for the better so I try really hard to create an environment where any and all feedback is welcome and appreciated.

MM: Has that helped? Has that gotten them to open up a little more?

AL: I like to think so. For example, earlier this morning, I had a conversation with a researcher and they had some constructive criticism about our research Confluence space. It’s good to know when something isn’t performing the way it should and can use improvements. If not, it may just sit and won’t be usable and that hurts my ops heart.

MM: When I’ve talked with people who’ve done some democratizing, sometimes they’ll get a surge of activity from the non-researchers when they first roll it out to them, and then they kind of go back to business as usual. It’s extra work to do the research, and so they kind of skimp on it. Have you seen any of that?

AL: I haven’t seen a major drop off, I’ve actually seen an increase in activity and it’s steady right now.

Because I’m one of the admins of our UserTesting account, this helps me keep track of who is using the platform so that’s a nice metric to have.

I do have insight into some of our resources that are being used. For example, back in the summer, we released a Store Research Recruitment Guide for those that would like to conduct research in H-E-B stores. There’s a couple of hyperlinks in the guide that send me a notification when they are clicked so that’s one way I can monitor that others are at least visiting one of our guides.

MM: The other thing that I sometimes hear from companies that have democratized is fear of bad research: These are not professional researchers, they’re gonna do things like self-confirming research. Did you have any worries about that?

AL: I don’t think I’m too worried right now. It might be because we are a smaller design org and we’ve created lots of resources for anyone conducting research to follow so we definitely built in some guardrails. Also, most of our designers use UserTesting if they’re doing their own testing and I get to actively monitor that.

Also, because our researchers are matrixed out to design teams, we get a nice pulse check if any designers are doing their own research because they’ll usually ask for a consult or feedback on their research plan. This also provides us insight if any research outside of the team is going beyond usability or evaluative testing.

Aside from that, we do offer research office hours weekly and highly encourage anyone who is conducting their own research to set up some time with the research team before diving in.

We’ve built an environment that makes it easy for anyone to seek out help from the research team if they need it.

MM: Current times being what they are, I’m obligated to ask you about generative AI. Have you looked at it at all? Any thoughts on how it affects research? 

AL: Right now, one of my projects is actually working with legal and privacy to better understand how design research can use generative AI, specifically trying to see if we’re allowed to feed our data into it. Can it help the team synthesize? We have not actively put anything into generative AI, because we need to make sure that anything we do respects privacy laws.

I have used it for some ReOps work though. When we built out our new Confluence space, we used ChatGPT to help us create the information architecture of the space. Also, it helped us by providing the outline of multiple templates to build off of instead of starting from scratch.

So it’s been helpful from an ops perspective. From a research perspective, we are still trying to figure out what we can do and can’t do with it.

Photo by Pixabay on Pexels

The opinions expressed in this publication are those of the authors. They do not necessarily reflect the opinions or views of UserTesting or its affiliates.