Doing user research can be eye-opening and helpful for any decision making. More and more companies acknowledge the value of user research and usability testing is becoming part of the product development process. With online tools like usertesting.com or loop11.com it is easy for everyone nowadays to set up a test. So everyone can become a user researcher. Is it that easy though?

The tricky part is not so much in doing the research. Both planning and observing or interviewing (the “before” and the “during” phase) require a lot of expertise (see also how to define a research problem) and putting the right questions together is definitely a skill by itself. But this is not the most troublesome part in user research.

The “after” phase is the real challenging part in user research, when it comes to analyzing and interpreting data and making sense of everything. One of the main qualities of a good researcher is the ability to stay fact-oriented and to treat every outcome objectively. But as human beings we are very prone to cognitive biases in this phase – and the only way to overcome these biases is becoming aware of them.

Cognitive biases to be aware of for interpreting research results

The first and most important is probably the bias blind spot: the assumption that we as researches are not or less affected by biases than other people – which is not true because that is simply how our brain works. This effect was researched very well by Daniel Simons and Christopher Chabris in their study of “Gorillas in Our Midst“.  Even though we will never be free from biases – we can become more aware of the influences of biases and not jump to conclusions too quickly.

The other important bias is the confirmation bias: we tend to look for information that confirms our existing belief-system and hypothesis. As with all biases this happens unconsciously, our attention is simply guided towards information that “fits” in our existing mental picture we have about a problem. A way to overcome this bias is by defining the research hypothesis and becoming aware of existing assumptions – to stay more open for different outcomes.

If we do observational studies we start with out interpretations during the sessions already as we are trying to look for patterns. This can lead to an anchoring bias: we tend to focus more on the first pieces of information we get – the first observations we make during research, the first problems we see and the first conclusions we come up with. One way that can help to overcome this bias is using note takings of observed frequencies, even in qualitative studies with small sample sizes to stay more honest with ourselves.

Lots of research has been done around cognitive biases and there are exhaustive lists out there with examples for them. I don’t want to go into to many details in this article – as the conclusion is the same: the more we, as researchers, learn about biases, the more aware we become of the possibility of fallacies the better we can become in making good interpretations that will lead to better decisions.

Interpretations of usability tests

Another aspect in data interpretation is coming to the right conclusions and the decisions we make based on user research. The other day I could caught myself from being too narrow in my recommendations. I observed several people in a usability tests who all had problems to finish a task we had given them in our test scenario. This is obviously a problem and we need to change the tool to enhance the usability. The main reason why people couldn’t finish the task was that they missed seeing a certain section on a screen. When I asked more questions about it afterwards I learned that it wasn’t so much about not “seeing” that specific element – but it was more about the process that we used in that case. People weren’t expecting to have different options so they simply ignored additional information about it.

The conclusion is on the one hand to make information more prominent – but we did that in our case already. Perception is based on knowledge. If we see someone struggle with an issue it not only tells us that we need to make adjustments to visual cues. It also tells us about the customer’s current knowledge state – and the gap that we should try to fill. Instead of only focusing on the interface itself we should also focus on customer’s existing knowledge and providing them with the right information upfront to form the right expectations.

Conclusion

To become better with data interpretation stay open and creative in your conclusions, ask a lot of questions and ask for critique of your data results to overcome biases.

Helpful Sources

Nielsen Norman Group on selecting online research tools

Book: What makes your brain happy and why you should do the opposite