Multivariate Testing Overview

Part 1: Introduction to Factorial Designs
I’m not sure how many parts this overview will be but since it seems MVT has finally “arrived” I’m starting at the beginning and hope to pound out a complete overview in the next month or two of every important factor to understanding and being successful with multivariate testing.


Multivariate testing (also called MVT, multivariable testing or experimental design) is one of the three common techniques for landing page and onsite optimization (A/B testing and onsite targeting being the other two) though it can be used effectively for ad testing as well. While multivariate testing does not usually provide the performance lifts of A/B testing or the ability of onsite targeting to deliver relevance, when used properly it is an essential part of ongoing and iterative online marketing testing and optimization.

MVT provides incredible intelligence and learning. The greatest benefit to MVT is what is referred to as element contribution. Element contribution informs the marketer of the percentage contribution each tested element has on conversion rate. This learning is highly valuable as it provides understanding of the triggers that influence behavior and can lead to numerous iterative test ideas that can provide further learning and improvements in conversion.

There are two main design of experiments that marketers use and multivariate testing vendors base their technology around. These are fractional factorial design and full factorial design. Let’s examine the pros and cons of each.

Fractional Factorial Testing
Fractional factorial testing (also commonly referred to as Taguchi Methodology) requires the marketer to choose two or more elements to test using one or more variations of the selected elements. This method is called fractional because not every page element is tested. The marketer selects what they believe are the most important elements based on what they are testing.

The combination of elements and alternatives is called an array and is used to create the pages used for testing. Certain arrays mitigate the effects of aliased interactions better than others. In my experience the L8 and L9 arrays are very good at this. L8 arrays are used for 7×2 (7 elements and two variations) and L9 for 4×3 (4 elements and 3 variations) MVT.

Example of a 7×2 L8 Array

The benefit of fractional factorial testing is that less data is required in order to reach statistical confidence in the results. This means that with a solid test design results can usually be achieved in shorter time periods. Usually weeks. This is highly beneficial for marketers that like to “fail faster” and take more iterative approaches to testing.

Since not every element is being tested certain interactions are aliased. This means that while the results may achieve statistical confidence full interactions between all elements are not measured. If the marketer selected the most important elements these interaction effects can be mitigated. Of course it’s not always easy to determine the most important test elements.

Full Factorial Testing
Full factorial testing uses every element of the page to create a test array measuring all interactions. Because so many interactions are measured this method requires a great amount of data (thus time) to achieve results.

As mentioned each interaction is accounted for so when confidence in the results are achieved there are no questions about possible interactions skewing the results.

Since every element must be considered and tested a large amount of data is necessary to achieve results. Incorporating variations of each element can also be a taxing creative effort. Because of these reasons full factorial tests need a much longer period of time to gain results, usually months. SInce online behavior can change (for many reasons) over such long periods results may tend to benefit certain temporal swings.

As a marketer I have always used fractional methodology and with tremendous success. I like the idea of velocity in marketing — test, learn, test, learn, test. Instead of one large test I prefer focusing attention on certain areas or elements to achieve deeper understanding. I also believe this iterative approach lends itself the dynamic nature of online marketing.

Part 2 will focus on designing MVT test arrays based on traffic analysis, page design and conversions.



5 responses to “Multivariate Testing Overview”

  1. Jason Geater Avatar

    Very informative post.
    I’ve read Tim Ash’s book a few times now, and I was never able to “get my head around” the difference between full factorial and fractional. Your explanation made it simple to understand – fractional uses some page elements, full uses all page elements.
    Thanks for the great information and I look forward to the next post.


  2. Billy Shih Avatar

    Well written! This is a concept that more marketers need to be familiar with.
    I also wrote about fractional and full factorial test design, specifically in terms of Google Web Optimizer, and the Taguchi method on my blog. I agree with you and find that fractional factorial makes a lot more sense, especially with how quickly seasonal changes impact people’s preferences and therefore your optimal page.
    However, I think you need to clarify this statement: “multivariate testing does not usually provide the performance lifts of A/B testing”. Multivariate testing often is a substitute for a/b testing and, in fact, if you tried to emulate a MV test using A/B testing, it essentially is the same as doing a full factorial test, which we both agree is not as useful.
    I believe when you say a/b tests you are referring to testing a new and drastically different page against the original. This new page has either large changes in design/aesthetic, all the way to new functionality. This is something that an MV test typically could not compare accurately.


  3. Jonathan Mendez Avatar

    Hi Billy,
    I don’t subscribe to your theory that MVT can or should often be a substitute for A/B testing. There are many factors that drive the test design decision but most often it gets back to:
    1) The question you are trying to answer
    2) The amount of interactions
    3) Traffic and conversion volumes.
    Usually one methodology is simply better than another based on those factors.
    As far performance lifts, the nature of MVT requires a certain cohesion of the various elements even among the alternatives. This reduces the overall differentiation to the design/aesthetic and as you accurately pointed out it is that creative differentiation that gets the big lifts.


  4. John Hunter Avatar

    I have collected some resources on multivariate testing including a collection of design of experiment articles that I think are very helpful. I am biased as George Box, Stu Hunter and my father wrote one of the classic books (in my option – but also in the opinion of others) on the subject Statistics for Experimenters.


  5. Lena Lindstrom Avatar

    I have used Google’s website optimizer tool for a year plus and have found it a pleasure to work with. Have spoken to Maxymiser and Accenture regarding their products and services… but due to charges have continued using Google. Anyone here who went with Maxymiser or Accenture – how did you find working with them?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a website or blog at

%d bloggers like this: