Multivariate Testing Overview

Part 1: Introduction to Factorial Designs
I’m not sure how many parts this overview will be but since it seems MVT has finally “arrived” I’m starting at the beginning and hope to pound out a complete overview in the next month or two of every important factor to understanding and being successful with multivariate testing.

Multivariate testing (also called MVT, multivariable testing or experimental design) is one of the three common techniques for landing page and onsite optimization (A/B testing and onsite targeting being the other two) though it can be used effectively for ad testing as well. While multivariate testing does not usually provide the performance lifts of A/B testing or the ability of onsite targeting to deliver relevance, when used properly it is an essential part of ongoing and iterative online marketing testing and optimization.

MVT provides incredible intelligence and learning. The greatest benefit to MVT is what is referred to as element contribution. Element contribution informs the marketer of the percentage contribution each tested element has on conversion rate. This learning is highly valuable as it provides understanding of the triggers that influence behavior and can lead to numerous iterative test ideas that can provide further learning and improvements in conversion.

There are two main design of experiments that marketers use and multivariate testing vendors base their technology around. These are fractional factorial design and full factorial design. Let’s examine the pros and cons of each.

Fractional Factorial Testing
Fractional factorial testing (also commonly referred to as Taguchi Methodology) requires the marketer to choose two or more elements to test using one or more variations of the selected elements. This method is called fractional because not every page element is tested. The marketer selects what they believe are the most important elements based on what they are testing.

The combination of elements and alternatives is called an array and is used to create the pages used for testing. Certain arrays mitigate the effects of aliased interactions better than others. In my experience the L8 and L9 arrays are very good at this. L8 arrays are used for 7×2 (7 elements and two variations) and L9 for 4×3 (4 elements and 3 variations) MVT.

Example of a 7×2 L8 Array

Pros
The benefit of fractional factorial testing is that less data is required in order to reach statistical confidence in the results. This means that with a solid test design results can usually be achieved in shorter time periods. Usually weeks. This is highly beneficial for marketers that like to “fail faster” and take more iterative approaches to testing.

Cons
Since not every element is being tested certain interactions are aliased. This means that while the results may achieve statistical confidence full interactions between all elements are not measured. If the marketer selected the most important elements these interaction effects can be mitigated. Of course it’s not always easy to determine the most important test elements.

Full Factorial Testing
Full factorial testing uses every element of the page to create a test array measuring all interactions. Because so many interactions are measured this method requires a great amount of data (thus time) to achieve results.

Pros
As mentioned each interaction is accounted for so when confidence in the results are achieved there are no questions about possible interactions skewing the results.

Cons
Since every element must be considered and tested a large amount of data is necessary to achieve results. Incorporating variations of each element can also be a taxing creative effort. Because of these reasons full factorial tests need a much longer period of time to gain results, usually months. SInce online behavior can change (for many reasons) over such long periods results may tend to benefit certain temporal swings.

Summary:
As a marketer I have always used fractional methodology and with tremendous success. I like the idea of velocity in marketing — test, learn, test, learn, test. Instead of one large test I prefer focusing attention on certain areas or elements to achieve deeper understanding. I also believe this iterative approach lends itself the dynamic nature of online marketing.

Part 2 will focus on designing MVT test arrays based on traffic analysis, page design and conversions.