Algorithmic Search and Discovery

When Rich Skrenta who has been a pioneer in Search with Topix (and just launched Blekko) writes the following, I take notice:

“Algorithmic search is sinking.

The only way to combat this and return trust and quality to search is by taking an editorial stand and having humans identify the best sites for every category. The algorithm can't find its way through the web's growing hall of mirrors anymore. And it's only going to get worse.”

However, Rich is only referring to one set of algos that are sinking. On the paid side algos are certainly rising, not sinking. A HUGE number of transactional queries end up clicking the ads – that are often more relevant (due to a/b testing of headlines and the like) and more useful than the underlying organic results.

Paid search aside, I take issue with Rich's solution of human editors/curation. I'd like to put the context of the problem in a different light. That of the multitude of wonderful mid/small retailers out there that people should not be afraid to buy from.

This has been the case since day 1 of the web. In fact, a few pretty nice sized web businesses have been built on the backs of these types of retailers. If we look at the current/latest major Google algo changes (May Day et al) the SERPs (once again) have skewed towards the Amazon's and Wal-marts of the world. This is the gentrification of the SERPs and I think it's a bad thing for Search.

At its core Search is about discovery. We should not be solving Search's problems by placing limits on our ability to discover especially if the solution – human curation – can be just as harmful and limiting in this regard as any algorithm. In fact, I'd argue human curation will be more limiting to discovery than Algorithmic Search.

Don’t think this type of human curation and editorial controls does not extend to content discovery the same way that it does commerce discovery. We’ve seen a great rise in curated content streams and it has an important place. However, any of us that have used Search to find out about information have surely stumbled onto helpful, useful and important blogs or other articles, essays, papers that we NEVER would have been exposed to without the algorithm.

The answer is not either or. The answer is both. The more tools we have on this web for information discovery the more we can discover and the better that experience can be. Yes, we’re navigating rough waters but the algorithm should not and will not end up as a wreck on the ocean floor, but rather a vessel that can and will help us discover new lands.