Indexing the hidden web

There has been some good commentary on Sergey Brin’s interview with the Guardian. It’s probably best summarized in John Gruber’s comment:

The assumption here is that the only way to search is through Google, and that the “open Internet” is only what Google can index and sell ads against.

This resonates with me because I think Google has put enough ads on the internet. It’s impossible to take anything Google says at face value if they talk of “open” but their intentions say “ads”. But here’s the thing: Brin is actually right. There is great data hidden behind apps that should be indexable.

In the race to win the App Store, we forgot about the web. Think about Instagram. Millions of photos are being shared that are inaccessible via a search engine. These photos can’t be found again and aren’t discoverable. When you search the web, how does it make sense that public photos on Flickr show up, but public photos on Instagram do not?

We shouldn’t have to choose between Apple’s closed systems and Google’s ad-driven business. I want to talk about improving the web without automatically being pro-Google. This tweet from John Marstall made me realize it:

I’d prefer to separate “indexable” from “indexable by Google.” Yes, they’re mostly the same for now. Might not always be.

We need competition in web search. More than Bing. A new search engine is the number 1 item from Paul Graham’s frighteningly ambitious ideas:

The best ideas are just on the right side of impossible. I don’t know if this one is possible, but there are signs it might be. Making a new search engine means competing with Google, and recently I’ve noticed some cracks in their fortress.

Of course it’s crazy, but so was the 1st search breakthrough, lightning-fast AltaVista, and so was the 2nd major innovation, Google itself. It’s time for a search engine that isn’t all about ads. It’s time for search that understands apps and embraces data from web services as much as it does from web sites. It’s time for the 3rd act in search.