Creating unbiased environments for user testing

Thoughts and methods to reduce the tension and biased responses in user research and user testing sessions.

Creating unbiased environments for user testing

Throughout my career, I have conducted countless user testing sessions and observed many more.  There are key patterns I have noticed which either enhance or break down the integrity of the session.  If you are familiar with hosting user testing, you have probably had times where the feedback was a dead end, with no tangible value to it, which wastes both time and money. I hope this article will help minimize these issues and guide both veterans and juniors alike.

Know your audience

If you are in any product role, it should go without saying that knowing your audience is a key factor to success. There are many approaches and books discussing how to conduct user testing, however, we should start at a fundamental level to minimize biased responses.

When customers try a product at home or in their office, they are in their safe zone. They have full control and understand the decisions they make without consequence or judgment.

People may try new features, or if something isn't working they will have various workarounds.

When you take someone out of their comfort zone and add observers on top of that? Well, they may inadvertently be on the defensive and take actions carefully; being more cautious and less true to themselves and their real interaction style.

Your goal as a host is to remove these notions and bring them towards a neutral state of mind.

Another issue is that many of us will focus on the questions to ask and what we want to achieve. We almost always forget that the people we bring in is in a new and unfamiliar setting. We need to make them as comfortable as possible.

Language barriers

Hola! ¿Como Estas? No, I'm not talking about foreign language barriers. What we want to eliminate is the terminology we use that causes mental barriers. Using the word test, either in the recruitment title or through conversation, we may unintentionally create a subconscious anxiety response. In many sessions I've hosted,  customers would jokingly ask if they are going to be graded on their responses. Both the customer and I know it's a false statement and possibly a way to break the ice, but we cannot dismiss the subconscious change in mental states when the word test is brought up in conversation.

When running sessions, try to ban the word test. When interacting with the customer try to utilize words like opinion or feedback, this may help them relax knowing it's a casual session. Any positive mental changes, even by the slightest degree will help usher in better feedback.

Environmental factors

With the design first movement, some companies have invested a lot into their facilities. I’ve been to some facilities where the user testing room is designed with a one-way mirror and obvious cameras from each angle. It seems great at first for the host and something very snazzy, but quickly you realize guests may feel like they are in an interrogation room.

Let's just say, after a few sessions in that room, we started hosting our sessions in a normal office - many people didn't feel comfortable. Some of our visitors were vocal, and others that didn't voice it were obviously uncomfortable by their demeanour.

Keep the space where feedback is gathered casual and comfortable. Try conducting your session in an open area rather than a closed room. Finding a space where it's quiet but still have people moving about can make for an easier session.

Let's not forget, every person is different -  if you want to go above and beyond, you may ask them which they prefer, in a closed office or out in the open. Catering for the customer and giving them some sense of control can help ease the tension of a new environment. I would go as far as being able to meet at a cafe if the option is available and preferred.

Observers

Naturally, people may become nervous if there is a large group observing their actions in the same vicinity. Their responses may be the same with the knowledge of a remote group observing too. Try to have a maximum of three to four people observing at a time and ask permission to record the session for other colleagues to review.

Sometimes it may help to introduce the person to the group that will be observing, and give yourself five minutes to do small talk in the group. After the meeting, allow them to once again be greeted by the group and allow the group to thank them for coming in. This will create a stronger bond and relationship between the staff and customers. If they weren't totally at ease during this session, they may grow more comfortable at future sessions hosted by the same group.

Setting the stage

Your prototype is the meat of this entire session. Creating a proper start and endpoint is crucial and there are two primary rules for success.

Set expectations from the beginning. Tell your guest they should be trying out the product as if nobody is around. Ask the user to talk out loud their thinking and what actions they plan to do next. It's not easy for people that are used to voicing their thoughts, but tell them to continually answer out loud by stating the following: "I'm going to do insert action because..." or "I'm confused because..."

A quick tip: If you have consent to record the session and the prototype is robust, ask them if they want a drink and let them know you will leave the room for a couple of minutes. This will allow them to continue on their own and note down areas they get stuck.

One issue with having a moderator is many people immediately ask for help without trying to complete the task on their own, of course, is not what happens in the real world.

1: Don’t show static screenshots

As much as static screenshots may allow for quick feedback — it doesn't work to show the user static view. When a static screen is shown to a customer, it becomes purely about aesthetics and becomes objective to the persons' opinion.

2: Personalize the experience

Prototypes need to give the impression of completing a given task without guidance. Dropping someone in the middle of an interaction flow isn’t realistic and doesn’t imitate the real world. The goal of these sessions is to emulate as closely as possible the experience someone would have if they were at home or working with your product.

Create the prototype to make the user feel as if they are logging into or registering an account on the application if that is part of the normal workflow. Try to match the display name and details to the customer in the interface to make it feel familiar and personalized to them. Who knows, your prototype might feel so real they will try to do other tasks and spawn a conversation of their intentions.

This whole notion goes back to knowing your audience. Don’t gather feedback from people that will not have any context. If a feature is being developed or redesigned that is only understood by longtime users, bringing new or prospective customers is probably not the best idea.

Responding to feedback

After the requested tasks have been completed, allow your guest to give you any feedback they may have. Hold this as a discussion and do not counter any feedback they have, always agree and converse accordingly — not all feedback has to be implemented.

By agreeing to everything, the customer will feel more at ease and heard, this is a session for them to give back. If they end up talking about the usability or feature requests, try to respond in the following human way:

  1. We would love to implement this, may you explain more about what problem this will solve for you?
  2. Is this a problem that plagues you often?

The responses to these questions can open up further discussion and may even lead to new avenues for the product upon further research.

Reconcile

After the session, it's time to analyze on a deeper level with colleagues while the session is fresh in your mind. I always suggest leaving at least thirty minutes between sessions to regroup. Remember it’s always okay to reach out to the user to clarify any feedback — this sort of feedback loop will also build relationships with customers which of course brings a stronger customer loyalty.