User testing: an input to innovation, not a source of it


Benz “Velo” model (1894)

It’s hard to imagine a life without cars. Before the automobile was invented getting around was a costly and particularly time-consuming business. That quick drive to the hardware store that takes 15 minutes in the car might have taken several hours on horseback, or an entire day in a horse-drawn carriage. Looking back, it’s hard to imagine how anyone was motivated to move anywhere at all. (In fact, most people didn’t… before the automobile, and particularly before rail, it wasn’t uncommon for people to spend their whole lives in the town they were born in.)

So if you could go back roughly 130 years and show someone the automobile, they would love it, right? If you asked someone the question, “is this product something you would buy and use?” the answer would be a resounding “Yes!”. Right?

Well… not really. When the first automobiles rolled onto the streets in the late 1800’s, they were met with skepticism and fear. People (and horses) were terrified by the noise, and people just couldn’t understand why anyone would need to go so far or why they would be in such a hurry. In other words, the automobile was an invention for a problem no-one had. Or, to be potentially more precise, a problem they didn’t yet know they had.

If you had shown concept drawings of the automobile to a focus group in 1885, or a working prototype to a user testing group, you might have walked away thinking that you’d be better off working on putting a clock radio* in your range of horse-drawn carriages.

The point is, you can’t expect users to know what they want. Innovation doesn’t come from asking a customer focus group “what products do you want that haven’t been invented yet?”

The iPad was a solution to a problem that no-one really had. Companies and products that innovate are successful because they can predict user behaviour before the users go anywhere near it. They are also good at convincing (selling) users that they have problems that their products can solve. No-one had a standing-motorised-transport-problem before the Segway was invented, but the company behind the gyroscopically controlled contraptions still managed to ship over 50,000 units by 2009.

We recently ran some early user testing on a product concept that we are working on. Based on the results, some members of our team were hugely disheartened: most of our test users, when asked if they could imagine them getting major value out of one of our concept’s major use cases, said “no”. Some thought we should go back to the drawing board. I think they missed the point…

User testing is one input to product design; one of many. Getting the input and responses of potential users early in the design process is crucial; however to make the results really meaningful you need to interpret them in relation to the test user’s context… and sometimes I think you just need to take the responses with a grain of salt. You also, I think, need to understand that innovation often comes from having the courage to challenge users on what they think they need and what problems they have.

* I’m of course aware that there were no clock radios in 1885. The first transistor radio wasn’t invented until 1954 by Sony in Japan. Call it poetic licence.

Release it: now

Release your software to real users as early as possible.

Building software for years without ever releasing any of it is like working in a kitchen and letting the food go cold and mouldy before serving it up.

Building software is a collaborative process – not just amongst you and your colleagues – or between you and your partners – but between you and your users. Feedback from real users helps you guide the future of your product and your immediate investments. It helps you prioritise and optimise. Making product decisions without real user feedback is quite often, I think, guesswork. Yes, it’s (hopefully) educated guesswork – but it’s guesswork all the same.

You can learn a lot from user testing and user research prior to release – and there’s no way you would want to skip these steps – but nothing can replace the feedback you get when real people use your product, with real use cases, under real load.

If you see that you’ve been sitting on the latest version of your product for months without any of it hitting a user, or if you find yourself wanting to squeeze “just one more feature” in before you drop the release, then stop and think: can I give my users meaningful, measurable value with this release? If so, then release it. Now!