The digital hot-pot : Preparing for an A/B tests 101!

So here is the thing with A/B tests:

First of all, they are a great way to understand whether you are doing something right or wrong.

You find a problem, do your research, then come up with a (possible) solution, and then you test it. Hypothetically (no pun intended here), this should be this easy right?

We all think that we are open-minded to all different kinds of ways and ideas, but if we are honest with ourselves; we are all biased. 'My way or the highway'. Some are obvious about that, some not, and some pretend to be Switzerland of mindsets when it comes to that.

So what should you do? There are multiple solutions to increase the chances of improving the success rate of an A/B test the first time around.

First understand that you aren't the only person who uses a website. So the way you think a page should be structured, might not be 'it'. Here are some simple steps to get started:

You can start by doing a holistic analysis:

1. Scan the page from your personal POV (Yes, your point of view is still important) e.g. discover usability issues, look for missed opportunities, find obstacles/ roadblocks/ friction that would prevent a user from performing a task (clicking/ finding a CTA)

2. Quantitative data: Analyse the page using tools like Adobe Analytics, Google Analytics, Heat-mapping etc.

3. You can use Jakob Nielsen's 10 usability principles as a guide

Pro's of a heuristic analysis: 1. Low to no budget needed 2. Saves time, 3. Good for low-traffic sites (below 300 visitors)

Con's of a heuristic analysis: 1. Prone to biases. (as mentioned earlier), 2. Need an expert 3. It is an isolated process 4. Usually not enough for a final evaluation (cannot be used in silo)

Now go a bit further by using qualitative methods:

1. Customer surveys (SurveyMonkey is a good tool)

2. Team input

3. Interview transcripts

4. Customer reviews

So what is the difference between quantitative data and qualitative data? :

Qualitative data focuses on words, descriptions, concepts and ideas

Quantitative data focuses on numbers and statistics (hard numbers)

You can get a lot of insights with the qualitative method that you cannot with the quantitative, so it is important to use both before running an A/B test.

Once you have gathered all of your data, start comparing and connecting.

First by yourself, then present it to your team and discuss it. Brainstorm possible solutions and then come up with your first A/B test.

1. Don't test too many components at a time (ideally one)

2. If you need to test multiple components, you might want to A/B test two completely different versions of a page

3. You still might not get the result that you expected, so redo the process and test again.

LIVA

Lisa Va.