Improving the ROI for Your User Research

So, you decided to do some user research in order to find out where you can make improvements. After a few hours of user interviews, you ended up with a notebook full of scribbled information that all seemed really critical. How in the world do you figure out what to do with all that information? If your answer is “talk about it all abstractly with everybody in the company or write a huge paper that nobody will read and then go on with business as usual,” you're in good (bad?) company. But you have to DO something with all that data. You have to analyze it and turn it into actionable items that your engineering department can use to fix your product. It's not always easy, but I'm going to give you an approach that should make it a little easier. This isn't the only way to do your test analysis, but it's one of the quickest and easiest that I've found when you are concerned with key metrics.

When to use this method:

• You have an existing product with a way to measure key metrics, and you’re interested in improving in particular areas related to your bottom line
• You have a limited research and development budget and want to target your changes specifically to move key metrics
• You are looking for the “low hanging fruit” that is getting in the way of your users performing important tasks with your product
• You are working in an agile development environment that is constantly tweaking and improving your product  and then testing the changes

When not to use this method:

• You have an existing product that you are planning to completely overhaul, and you want to understand all of the major problems before you do your redesign
• You are trying to create an overall awesome, irresistible user experience that is not related to a specific metric
• You are designing a new product or feature and are observing people using other products to identify opportunities for innovation

If you fall into the first bucket, read on… The Five Basic Steps:

• Identify key metrics you'd like to improve
• Identify the tasks on your site that correlate with improvement in those metrics
• Observe people performing the appropriate tasks
• Identify the barriers preventing people from completing or repeating the tasks
• Develop recommendations that address each specific barrier to task completion

Step 1: Identify Key Metrics

First, identify the metrics that you care about, and look at the distinct steps that your users are most likely to take to change those metrics. This way, you can plan your qualitative testing to show you exactly what is getting in the way of your users reaching each subsequent step. As an example, let's imagine that you have a social networking site that earns revenue when people give virtual gifts to each other to display on their profile pages. Because your revenue depends on lots of users purchasing lots of things for each others' pages, you might care about the following metrics: how many people register, how many people customize their profile pages, how many people make friends, how many friends they make, and how many people purchase one or more gifts. Obviously, that last is the most important, since that's where your revenue comes from, but the others are also important, since improving those numbers should increase everything downstream, if done in the correct way. Your goal is to get people from point A to point D as quickly and pleasantly as possible. To do that, you want to look at all of the barriers you have erected in the way of your users accomplishing those tasks. Here's are the pieces of the user flow that change your key metrics: Registration > Profile Customization > Finding a Friend > Purchasing a Gift

Step 2: Identify Individual Task Flows

Now that you know what your key metrics are, you need to make sure that you observe your test participants trying to accomplish the tasks that will eventually lead to changing those metrics. To plan your tasks and analyze your data, it helps to break down your individual tasks into smaller user flow steps based on your user interface. For example, while the number of successful registrations may be a single metric, there may be several screens or discrete interactions involved in your entire registration process. Make a list of what they are now. As an example, I ran a test recently for a product that had a 4 step registration process. The user flow for registration on the site looked like this:

Step 3: Observe People Performing Tasks

So let’s say that, based on the metrics you want to change in your hypothetical social networking site, you've determined that you need to gather data about registration, profile customization, finding friends, and purchasing a gift. You’ve scheduled 5 people in your target demographic to come in for user test sessions. I’m not going to teach you everything you need to know about running a test, but here are some basics. First, let the user begin using your product however they want and note all of the things that they do that aren’t the tasks you’ve identified. Remember, any time spent not doing your target tasks is time that is not being spent improving your metrics, so you want to know what these other things are! If you give your participant a targeted task right away, you will never learn what other things they are doing instead. Finding these distractions allows you to eliminate them or change them so that they also improve your key metrics. Watch everything your user does with the perspective of the ideal flow for your metric. Do your participants wander off track or getconfused during registration? Do they fail to find their own profile? Do they not have the information they need to find a friend? Do they fail to understand what gifts are or how to purchase them? Do they get distracted by a totally different part of your site that doesn't contribute to improving your key metrics? Once you’ve allowed the participant some free exploration time, you may need to move them to a particular task. For example, in our current example, it might be that you need to ask the participant to try to purchase a gift, even if that’s something they wouldn’t do in the context of a study environment. You can move them along by saying something casual like, “Did you notice that there is a way to purchase gifts? What do you think about that?” Meanwhile, record what the participant is doing and encourage them to talk about their impressions of the product.

Step 4: Identify Barriers

Once you've gathered data by watching 4 or 5 people repeat the same tasks, you need to analyze your data to figure out where the barriers are and communicate them effectively to the team. Determining the barriers should be pretty easy. Just ask yourself the following questions:

• Where did participants seem confused or distracted and stop performing a task?
• Which things took longer than they should have?
• Why did participants fail to accomplish a task?
• What tasks did the participants perform that would not improve the key metrics?

In the product I tested with the four step registration process, the metrics showed significant drop off one each screen. On screen number 2, the user was asked to choose a unique user name and enter some information. It had a surprising amount of drop off considering how simple it was. After watching a few user tests, I noticed that many people were trying to select user names that were not available, but they were failing to notice the error message telling them to choose a different one. Even if they noticed the error, several people had a tough time finding a user name they liked and would spend minutes typing in word after word, getting frustrated, and finally giving up. By watching people try to register for the product, we identified several significant barriers to successful registration in just one step of the process. In fact, we found similar problems – some large, some small – in each of the steps. Once we’d identified the barriers, organizing and explaining them to the team was easy with a simple, annotated flow of the steps leading up to the goal along with the problems associated with each step. For example, the flow for the registration test might have looked like this:

Landing Page

• Didn't understand the product
• Felt the product was aimed at teens
• Thought the network was very small

User name

• Didn't know when a user name was taken
• Couldn't come up with a decent name

Personal Info

• Were concerned about privacy
• When failed Captcha, all info had to be re-entered

Download

• Were uncomfortable downloading b/c of viruses
• Didn't know what they were downloading

Overall

• Were bored during registration process or complained that it was taking a long time

Step 5: Develop Recommendations

Now that you know what barriers are keeping your customers from accomplishing the goals you've set for them, you need to generate recommendations for ways to remove those barriers. You can do this by looking at exactly what the barrier is and what the users' reactions were to it, and then brainstorming ideas to help the user overcome that barrier. Yes, this is the tough, somewhat creative part. You don't want to just take any recommendation that the user gives you. You need to understand what problem they're having and come up with a way to fix it that doesn't cause any other problems. Going back to my example, once we found the problems on the user name page, we looked at several solutions. We definitely needed to make it more obvious to the user when they selected a name that was already taken. We came up with a few simple ways to solve this particular problem: we could suggest other user names when people tried one that was already taken; we could make the error more noticeable; we could make the Next button obviously disabled so that they couldn't even try to move on; we could free up some names that had been taken but weren't currently in use so that the selection of user names was better. We then selected a couple of these solutions based on expected ROI calculations. Even the lowest effort of these suggestions, improving the visual design of the error, caused an immediate and statistically significant decrease in drop offs for that step in the registration flow, and eventually improved revenue by increasing the number of successful sign ups. The barrier was lowered, if not entirely removed. You may have noticed that the last section is labeled "Overall." This is where you look at the process as a whole, rather than a sum of its parts. For example, one comprehensive solution for drop off at each step might be to reduce the number of steps it takes to register (almost never a bad idea!). Doing this wouldn't necessarily mean that you wouldn't have to address the other problems individually, but you might get fewer drop offs simply because it would take less time for users to finish the process.

How Is This Different?

You might be wondering how this is different from more traditional ways of running user tests and analyzing data. In any user test, you'll need to come up with tasks, observe users, figure out where they're having problems, and come up with solutions for the problems. There is a difference though. My experience has been that companies do not fix every single UX problem that they find in user research. I don't love this, but I do understand the need to prioritize important changes. So, if you're only going to fix a few problems, you should make an effort to identify the most important ones. The counter intuitive part is that the most important problems might not be the worst problems from a user experience stand point, but they will definitely be the ones that are keeping your users from reaching the goals that improve your most critical metrics. Using this framework will help you identify your key user flows, find and communicate the major barriers to success, and propose targeted solutions that will improve both your user experience and your business.