The Long Tail vs. The Social Graph

Read Write Web has an interesting transcript of a panel session with Jimmy Wales (Wikia), and Jason Calacanis (Mahalo) at the DLD conference in Germany. Both introduced their service and spoke of the strengths of their brand of user-augmented search and how that competes with Google.

The industry all has their eyes on the prize, the holy grail of search. Everyone all wants to build a search service that can take a query and return the most highly relevant result. Jimmy Wales believes users will contribute to build the most relevant result (not unlike Eurekster), whereas Jason sees a need to have an editorial component shape that result set.

Google, on the other hand, still focuses on their algorithm, and the long tail. Yes, the long tail (deep and wide crawl of available documents) presents an infinite set of data in which to search. But in that size lies the problem, content relevancy. Pagerank, in all its brilliance and wonder doesn’t understand about content or even context relevancy. What it does do though, is validate the source.

Jason introduced the problem quite succinctly:

One person can pollute the internet with hundreds of thousands of pages in a matter of minutes.

So if the long tail is polluted, what is the answer? Context or content relevancy.

Mahalo’s answer for context relevancy is editorial:

If you look at the long tail of search we are looking at filling the top spots with journalistic search results.

Which is a very valid response. Look, here is a search for cherry pie, instead of links to pages containing lyrics to the Poison song (yes, I have just dated myself), here are links to actual pages of value for the query. But there are two obvious questions here: can this scale (or is it meant to), and what if we get into more complex queries where the question of bias is introduced?

Melissa Meyer of Google agrees:

To take an algorithm and enhance it with editorial without introducing bias is the solution

The problem with this statement is pretty simple, it is impossible to remove bias programmatically and even editorially.

We, as humans, are all biased. Why? Because we have different experiences, and different points of view. That point of view is not always visible to the naked eye, so we have to trust the source even if we don’t know who that source is. And how do we know if that point of view is not driven by an army of SEO’s that are looking to game the system beneath a cloak of hundreds of users as their disguise?

This is where the power of the social graph steps in. We know what we trust, and we also to a degree trust our friends. By harnessing that trust, we can tailor search to emphasize these levels of trust. This model works the same in the physical world as well. Would you try a restaurant based purely on an ad? Maybe. Would you go based on a good review? Quite possibly. Would you go based on the recommendation of a friend? Definitely.

Our definition of context, or relevancy, is inherently based on trust. We inherently trust our friends, or people we feel have the same concept of relevancy as ourselves. You may not trust tomshardware.com, but my friends do. So my concept of trust trumps yours when I search. This is not a bad thing. Your sphere of trust, and that of your friends, could prefer sits like overclockers.com, or mini-itx.com, etc etc. We need to stop thinking of content and context relevancy as a universal truth. Its different for everyone.

This, has been our core ethos since day one. Building a service that allows people to tap into their own understanding of trust, and allow that trust to expand and augment the algorithm. For them.

Of course, I am biased. But you knew that.

Share This

Tuesday, January 22nd, 2008 at 12:09 pm and is filed under Wikia, Maholo, long tail, social graph. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.