Testing and Optimization for the Holidays

It’s over 100 degrees today in New York City however my mind is thinking about freezing cold mall parking lots that don’t have a parking space. What does that mean? It means that now is the time to start optimizing for what will likely be the largest online shopping season ever. So, where and how do we begin our efforts?

I’m often confronted with companies that want to get the answers to many questions in a single test of their landing page or ad. Of course, there are companies out there pitching that methodology and it may seem like a great idea. It’s not. The best testing should be simple, iterative and focused on answering a single question. This will naturally lead to more questions. That’s great because testing should be an ongoing and fundamental part of online business.

Testing should also be isolated as much as possible. Everyone from scientists to programmers has long understood the benefits of isolated environments for testing. This helps separate the signal from the noise…a phenomenon of outside and often uncontrollable factors influencing the test. The more outside factors you can remove from the test equation the more confident you can be that your findings are accurate. And if you can get a large enough data set in an isolated setting you can be confident with the answers to even the most difficult questions.

Let’s say an online retailer wanted to optimize their customer funnel to increase conversion and revenue. They probably have an extensive site hierarchy that includes top-level pages, category pages, sub-category pages and product pages…and then of course the checkout process. This is a long funnel and there are many factors (noise) that influence a conversion.

The best strategy for them would be to divide each step in the funnel-flow process (and forget checkout at the moment) and create test plans for each page with a single question to be answered. What works best getting the person one more step closer to conversion? So instead of one big multi-page test we would run four separate tests.

Additionally, if we take each step of the funnel and optimize it in isolation we will make a greater impact by not only improving it an overall manner but also optimizing the funnel for where and how users are entering it. This is very important.

For example, will users coming to these pages on deep linked PPC ads behave differently than users coming from the homepage? Which pages from what AdGroups and keywords does the search traffic like best? These are great questions. By segmenting out our test groups we’ll have answers to these questions and more in only a few weeks. Best of all, should we find that not every segment likes the same messaging, creative and content, we already have our JavaScript in place from the tests to deliver targeted content.

It’s important to mention again, and this really the takeaway I hope you get…testing is an iterative process. Start out with a simple A/B..N test and then move into a multivariate test on the winners. Test results down to the keyword level against different traffic sources (search, direct, media) and user stage segments (first time visit, return, customer, etc.). Once satisfied with a great page dynamically deliver the winning content for each source and segment. Sound simple? It is.

The results of this testing and optimization process will have a huge impact on conversion. For large retailers we would be looking at millions and millions of dollars this Holiday season in additional online revenue. Now that’s a simple result everyone can understand.

Here’s another great post on Holiday Shopping optimization courtesy of Matthew Roche:

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s