Data is Easy. Optimization is Hard.

For the past 15 years web analytics has been namely about
one thing, counting. Counting is very important and as an industry we have become
infatuated with it. Never before could a medium really count what was happening with it. Attributable media was and
continues to be truly revolutionary.

Attribution quickly formed the business models for the new medium
— traffic buying, affiliate marketing (have we forgotten how Amazon got to be
Amazon), SEO and Email are all channels that emerged based on one principle.
You could count what you got in return.

In measured media, performance becomes its driver and thus
optimization became the obligatory next progression. However for much of
digital media, namely site content/user experience & display advertising, it
has been a long slog towards optimization. A medium capable of being optimized
to deliver persistent relevance may only be (by my best estimate) about 5%
there.

Of course if we want to look for optimization inspiration we look at Google. Paid Search serves as our shining example of what
optimization is capable of. Many are trying to emulate their models for
success but search is only a small piece of our digital experience.

Still, there is one important lesson we can take away from search. We can
optimize the medium up to the point of diminishing returns. This is simply
amazing. Yet, most “experts” and certainly C-level execs don’t even realize this
is possible. Most clients will not believe it. But it is an economic law (and one that data and ad exchanges will put to the test).

Not to be lost in all the dashboards and spreadsheets is what
we are counting – the experience of people. Performance is based on the actions
that control the medium – content, experience and people’s response. There is
of course the ability to optimize the buy-side however if we have learned anything as performance marketers it is that the sell side
optimization factors most highly into optimization. In this regard we can simply define our goals as optimizing the presentation and delivery of content.

This feedback loop is one reason the web experience has the
amazing ability to be optimized and should improve for years to come…if we can
get there. Optimization is hard. It requires some measure of testing and
analysis — from bidding systems to creative performance. And data is a funny
animal. There is no such thing as a perfect data set and often there are outliers.
As we’ve seen this past year, even the world’s best systems are unable to
prevent market forces from skewing predictive models to the point of rendering
them useless.

There is also the “Predictive Paradox.” As we witness
everyday in real-life people’s behavior is simply unpredictable over short periods
of time. Yet incredibly and more often than not, over longer periods of time
the differences in behavior normalize without significant variation. What this
means to behavioral marketers is two things: 1) it is really hard to accurately
assess wins that take place over short periods of time & 2) it is really
hard to improve results by a large margin over a long period of time.

The other difficulty with optimization besides statistical
significance is quality of data — how much noise is in the data set. Optimization
of ads or any content has so much to do with the surroundings. Content, layout,
visitor source, visitor familiarity with site, and probably about another
hundred or so factors. With so much noise the more complex the testing the more
false positives in the numbers. Yet for some reason we’ve been led to believe
complexity is sexy when in fact it is accuracy that is sexy.

There is one last thing we should never forget as we spend
more time optimizing digital media and it may be the most important aspect. Counting
tells only part of the story – it tells you what people did. It does not tell
you why.  It does not provide
answers or shed light on the actions people never realized they could take. To
a large degree these answers can only be achieved through testing methodologies
and observational research. Be wary of any predictive or targeting system not
continually validating its assumptions with these methods…and good luck optimizing.

Previous Relevant Posts:

Expert Guide to Multivariate Testing Success

Capturing the Gorilla: New Ways to
Quantify the Value of Search

On-Site Targeting: Crawl, Walk, Run Strategy

The Power of Brand to Influence Outcomes or Why Brands Will
Always Rule Digital