Two versions of the same page will be pitted against one another, measurements will be taken and a hopefully a winner will be declared. Typically version A will be the existing design and version B will be a modified version of A.
How do you perform A/B testing?
Traffic will usually be split so that 50% of users see page A and 50% see page B. We then measure the performance of both pages using metrics like engagement and goals (such as adding to basket or signing up to a newsletter).
Once we have enough data, we can use the measurements and a statistical confidence level to declare one variation a winner.
Why is A/B testing important?
A/B testing is essential when it comes to digital marketing because it turns hunches into numbers.
It’s pretty easy to look at a webpage and say it could be improved with better colour contrast or a bigger call to action, but how can you tell if those changes will actually have a positive effect on the page? You could just plough on and make the changes without testing (while paying for expensive dev work) and end up with a worse performing page.
Or we can use our testing tools to make quick changes to a page, test them thoroughly and then make decisions based on actual facts, instead of intuition.
The results of A/B testing can be surprising or unpredictable. Sometimes the winner at the end of a test will not be the version you’d have expected at the beginning. This highlights just how important it is to test before making changes to a website.
What do we test?
Deciding what to test on a page depends on what the page is designed to do.
If we are testing a product page, we might test the amount of copy in the description, the size and placement of the ‘add to basket’ button, the size of product images or the colour of text for reviews.
We focus a lot on A/B testing for the homepage and for the pages which have a clear call to action, i.e newsletter sign up.
There are lots of options when testing, but some standard tests include:
• Layout of a page
• The size, colour and position of a call to action
• The copy of a call to action
• Product titles or descriptions
• Using different images
• A long page V short page
• Longer forms V shorter forms
• Colour themes of a page
By looking at the amount of traffic and the conversion rate for a page we can estimate how long a test will need to be run for to collect enough data. This can be anything from a couple of days or two weeks.
Classic Examples of A/B testing
1. In this straightforward test the colour of a sign up button was changed, but the text was exactly the same:
They found that 21% more people clicked on the red button than the green button. Source: http://blog.hubspot.com/blog/tabid/6307/bid/20566/The-Button-Color-A-B-Test-Red-Beats-Green.aspx
2. In this test a secure trust symbol was tested. You’d generally expect a trust symbol to improve conversion but the results were surprising:
The version without the secure symbol actually saw a 400% increase in conversion over 3 days. This just goes to show that our intuition and expectations aren’t always right.
3. This one is a prime example for showing how something seemingly small can make a big difference. The amount of information you ask a user for can change how willing they are to give you, if any at all:
The only difference is that variant B has one less field to complete. This small change resulted in many more people filling out the form; which made Expedia lots of money.
So there we have it – a quick overview of what A/B testing is and how it can massively help your website’s goals.
Have you started A/B testing yet?