Be Experiment Smart

Experimentation has hit the mainstream. It’s become de rigueur to sprinkle references to “A/B testing” into job descriptions, business plans, and company websites. But let’s look past the buzz for a moment. How you experiment is as important as whether you’re experimenting. Productive experimentation means picking the right experiment for the right phase in your project. I’m teaching a class at the Stanford d.school this quarter that’s all about experimentation: we’re calling it Prototyping and Rapid Experimentation Lab. Along with my co-teacher Pam Hinds and our course assistant Nik Martelaro, we’re helping advanced design thinking students learn their way around how and why to experiment. There are two core principles at work here. First, test any idea that you’re considering with as little commitment as possible. You want to learn on the cheap, before you’ve invested a lot of time and money going down the wrong path. Second, experiment along multiple axes. You’ve heard of “A/B testing” – but there are many, many kinds of A/B tests. What exactly are you hoping to learn? You need to know how your idea stacks up according to these dimensions:

• Interest – would people want this?
• Use – does this solve a real need in a way that people want it solved?
• Usability – is this easy, sensible, and efficient to use?
• Implementation – can this be done?

 I’m borrowing from some ideas proposed by Houde and Hill in their seminal paper What do Prototypes Prototype? (1997), and giving these ideas a spin to be more relevant to what and how we design today. Here, I’m focusing on the first three axes (interest, use, and usability). Keep in mind, though, that you’ll eventually need to launch experiments about feasibility as well. Let's take a look at each axis independently to understand how to dive in.

Interest: would people want this?

Asking people “would you buy this?” is not going to give you an accurate sense of what they think of your idea. You need to be sneakier about assessing their interest. Create a situation where somebody happens upon your idea passively, and let them tell you whether they’re interested in this first taste. This is a great place for quantitative data – you want to prove objectively that there is enough potential to push forward.

Advertising

Run an ad (or several) and see if people click. Try Google ads, Facebook ads, or ads on community sites that your target user might visit (a foodie message board if your product is for restaurant diners, say). If you’re feeling especially sly, start chatting about the prospective product in forums where your users hang out and see what people think!

Landing pages and painted doors

To get more detailed feedback on your concept, link those ads to an actual landing page that describes your concept and link the ad described above to it. Create a button or other affordance on your site that is a doorway to the functionality you’re considering, but don't build out the functionality yet. Gauge visitor interest by tracking page views, clicks, and/or sign-ups to learn more. These quantitative approaches will show you what’s not working, but not why. For more qualitative feedback, go to a cafe (or wherever your potential users hang out) and show your design around for some quick guerilla feedback.

Create content

If your idea has to do with a certain kind of content, try starting a bare-bones blog, twitter feed, or pinboard. Driving and observing your traffic will teach you a lot about what readers are interested in. For example, if you’re hoping to make an app related to vegan eating, you might experiment with recipes, photos, restaurant reviews, travel guides, and nutrition advice, just to see what seems to catch on. Alternatively, try Hidden Radio’s approach – submit a description of your product idea to a well-trafficked blog and see if people seem interested. The team learned, in a week, that they’d hit on an idea worth pursuing.

Us or them?

Create a landing page for your product. Show your product to users, then show a competitor's product (switch up which product you show first). Ask people to compare them. Which do they prefer? Which do they think looks most practical, most fun, most valuable?

What about surveys?

Surveys are notoriously poor at gauging interest. They can work well for getting factual, quantitative, objective data about real events (as long as respondents don’t feel social pressure to pick a particular answer). But you can’t learn much from a survey if you’re asking open-ended or hypothetical questions – people will say yes to anything. If you must ask a survey question about a nonexistent offering, create a quick prototype and ask about people’s reaction to the prototype.

Use: Are we solving the right problem?

So people are interested in your idea, and you’ve built a prototype. Now you need to figure out if you’re really solving their problem – and if you’re doing it in a way that makes sense for them.

Testing in context

Your product doesn’t exist in a vacuum. To get a sense of how context impacts how people use your product, build a prototype using one of a million tools out there and test it in context – or fake the context as closely as you can for testing purposes. For example, say you’re making an app that helps people troubleshoot their internet at home. To test it in context, you might start by instructing your user to do something engaging on the internet. Next, you, the tester, could artificially cause the network to fail, and see if your user can successfully use your product to fix it.

Wizard of Oz (WOZ) Tests

WOZ tests use low-tech workarounds to make it look like a computer is doing work. Create a working front end, but then set up a human-powered kludge for the backend. If you’re building an experience for online takeout, build the interface, complete with “place your order” button – but have people behind the scenes call-in orders to restaurants. You’ll learn a ton about your product without investing time or money in an order automation system. In her book UX for Lean Startups, Laura Klein describes how Airbnb used WOZ testing: they had a sense that better photos increased a rental property’s desirability, so they hired a few pros to take photos of a few selected properties. The properties with the professional photos did much better – so much better that Airbnb now offers professional photography as a free service to people listing properties and has an automated backend to support it.

source: airbnb.com

Usability: Can people use it?

After interest and use has been assessed to determine if you have a good idea to being with, you can start to “debug” your prototype. What’s unclear, what’s hard to find, what needs tweaking?

Moderated usability testing

I’m sure you’ve seen this technique before: make a prototype, ask people to complete a task using your prototype, and see if they can do it. If they can do it, congratulations – your prototype is easy to use. But don’t forget to test its usefulness as well as its usability. I’ve seen plenty of products that are beautifully simple, exceptionally clear, and completely pointless. At Sliced Bread, we conduct a variation on the classic usability test that we call a Fast Insight Test. We usually run 3 to 5 tests of around 30 minutes each, and we test early sketches, not just finished prototypes. These quick tests give us a treasure trove of information, without all the expense and overhead of a usability testing lab (we often test in context or through screensharing) or a long study. Often we squeeze even more value out of tests by iterating between sessions, or even on the fly. For more info on how to do it yourself, see our previous post on Fast Insight Testing.

Unmoderated usability testing

A whole industry has sprung up around automating user testing. You’ll have no trouble finding a service that will recruit users and run them through a task for you, then give you results: try Usabilla, usertesting.com, Mouseflow, Five Second Test, or Loop11. These tools will help you gather quantitative feedback on the usability of very specific aspects of your site. They’ll show you where the fire is, but not why it’s burning.

Now go out there and do it

There’s a lot of subtlety to experimentation, but at its core, it’s about doing. Whether you’re looking for information about interest, use, or usability, you can frame an experiment to help you out – you can get started this very afternoon. In my next post, I’ll talk more about how to develop the questions and hypotheses that drive a good experiment.