Top 10 Multivariate & A/B Testing Results from 2007


This list compiles the most counterintuitive and therefore most interesting test results from multivariate and A/B (or split) tests I was involved in over the past year. I thought that this would be more valuable, a better read and more modest than just a rank list of tests that had the highest lift in conversion metrics.

These results were surprising either to me or to clients. I realize that it’s hard to place these results out of context with what they were tested against. Many of these tests or the ideas behind them have been blogged about over the past year and I’ve provided those links where I can. So, in no particular order, the ten most interesting 2007 results with notation on what type of test and where the test took place.

1. The highest factor of influence on conversion rate was having global navigation present (MVT – landing page). More info on this test

2. Different call-to-action button colors, shapes and locations performed better on different site pages (MVT – product pages, category pages, checkout flow). More info on this test

3. Three bullets and a call to action button increased conversion rate vs. a content focused page describing the service in detail (A/B – homepage).

4. A single product increased conversion rate vs. a choice of three products (A/B – landing page) More info on this test

5. A long scroll text heavy page (ala direct mail) performed better than shorter more concise pages (A/B – landing page) More info on this test

6. Having each radio button for additional search results be opt-out instead of opt-in increased overall query volume (MVT – search engine interface). More info on this test

7. Replacing benefits content with a larger image of the product helped increase conversion rates (A/B – landing page/product page)

8. Replacing marketing content with a search query field — aka “Googleizing” increased almost all performance metrics (A/B – homepage)

9. Taking the small grey print in the footer that the company didn’t want people to see and putting it front and center helped improve conversion rates (A/B – landing page)

10. Client elected not to implement winning homepage because the optimized user experience reduced page views.

Making this list was more difficult than I anticipated. So many results from testing over the past year were counterintuitive. Even as expert marketers we cannot predict how people will respond to creative and messaging. Therefore, don’t take these results as recommendations. However, they are all great test ideas for 2008.

For me, the main takeaway from this list (though there is a result on the list that refutes this) is that visitors know much more about your product or service than marketers give them credit for. Focus more on making it easier for someone that is ready to convert than on persuading people that are still in consideration stage. Chances are those consideration folks are making their decisions based on content gathered elsewhere rather than on your landing page or site. That thought is probably worthy of its own post.

It’s been a great year for testing. Many happy returns in 2008!


4 responses to “Top 10 Multivariate & A/B Testing Results from 2007”

  1. Richard Hearne Avatar

    Another great post. All I could ask for now is a nice .pdf white paper outlining all of the above in one easily digested document 🙂
    Looking forward to becoming an Offermatica client in the new year.
    Best rgds


  2. Billy Shih Avatar

    Thanks Jonathan! You have a great blog too. I was considering adding you to my blogroll but thought your blog was going in a different direction. This post shows that you aren’t necessarily doing that though. I’ll add you today.
    Your last paragraph, is exactly what I was saying in my post. People try too hard to please everyone, when you just have to grab those people who are ready convert.
    Testing definitely surprises me almost every time. Best practices have their place, but testing is the only way to know what’s best for that page’s visitors.


  3. Look What G-Man Found Avatar

    Multi-Variate and Split Testing Results

    When youre testing your landing pages for affiliate campaigns, youll want to make sure that you continually test to see what improves your conversion rate.
    There are basically two ways to do this. The first is through standard A/B testin…


  4. Online Marketing Mavens Blog Avatar

    Best Online Marketing Lists of2007

    Everyone loves a good list, so I wanted to share my favorite online marketing lists for 2007. These helpful lists will give you some great ideas for your online marketing in 2008.
    1. Big List of Blog Search Engines
    A good blog is hard to find and…


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a website or blog at

%d bloggers like this: