In part 2 of this series we discussed the first part of multivariate test design — determining your test array — how many elements and how many variations of each you you should test. In parts 3 & 4 we will cover the rest of MVT test design, element selection and element variation.
Successful testing starts with clear goals. It is these goals (metrics) that will inform your strategies for element selection and variation. Being that we are interactive marketers our metrics tend to be an action of some sort or another. This action (sometimes referred to as conversion) can be a click, it can mean a sale, it can refer to any outcome. For example, if I am trying to increase Average Order Value (AOV) or Revenue per Visitor (RPV) I may select different elements (and most certainly create different variations) than if I’m optimizing Order Conversion Rate. Making a decision what success means before you start test design is important but it is essential before you start this part of multivariate test design.
Once we begin testing a large part of the value in MVT is the element contribution reporting. The best tests will answer questions for you around your goal but the element contribution reporting will raise many additional questions that you’ll want to find the answers to. Therefore the next consideration we want to make with our test design strategy is to select elements where we desire to achieve greater understanding of their influence on the outcome.
The nature of MVT and the learning created lend itself to ongoing and iterative testing cycles. In Part 1 I discussed that the nature of fractional factorial testing requires us as marketers to select the elements that we believe are the most important factors to the metric we are trying to optimize. What influences an action? It can vary quite a bit. There are no defined best practices for element selection for each type of test array. Generally however we look to the following elements as those that have the most influence on people’s experience.
Navigation – There are some strong opinions in optimization around having or not having navigation on landing pages. The best thing of course is to test. As you can see with this case study.
Headlines – Headlines are often the most influential element on the page. Many elements may not be noticed on your page but more often than not the headline is noticed as people have been been trained to look for and scan headlines for clues to relevance.
Offers– Offer testing can lead to huge learning. How does your audience best respond to offers and how does this effect your bottom line results? Does 10% or 25% Off make a difference for AOV or Conversion? Do more act with a dollar offer or a percentage. Do gated offers ($10 off orders of $50 or more) get better metrics than straight percentage off offers?
Messaging – How do you present your benefits and unique selling propositions. Are bullet points enough or do you need more copy? In what voice are you speaking to visitors (more on this in element variation). What is your mix of content?
Price – Sometimes nothing makes more of an impact than price. If you have the ability to synch price testing with your back end reporting and analytics this is essential to understanding the market value of your products and services.
Calls to Action – Those little buttons and links really matter a lot. I love doing MVT on the buttons themselves however as an element in an array you can often get a quick win testing a button or call to action link. Highly recommended.
Next up is the most creative and my favorite part of MVT, element variation.
jonathanmendezblog.com 
Leave a Reply