Question

Create a distributed search engine where the nodes are all participating browsers as follows: every browser stores any public pages that the user is browsing, after checking that they are not in the system yet (using a distributed hash table such as Bamboo). A new page is indexed and the information contributed to a distributed search engine such as pier, which is also used for searches. This solves in one move the problem of quasi-monopoly in search and the problem of keeping up with the growth of the internet (if a page has been browsed, it has been indexed) and the problem of rating pages (a popularity rating is immediately available). There are several technical hurdles to make this a reality, but search is too important to be in the hands of one company, evil or not.

Answer 1

Hi A P,

Sounds like an interesting concept - but why would you want to store the pages decentralized on the client? Problems like storage and access costs apart - wouldn't it make more sense to have this on a central server?

You could use the browser as the crawler (which makes a lot of sense) and also use the browser to do some of the heavy lifting in terms of compressing the information into manageable chunks of data.

-Pascal

Answer 2

Pascal, thanks for your comment. The arguments for fully distributed search are many: cost, reliability, freedom. You would not let a single company run DNS, would you? But we are letting google run search for almost everybody. When they are down the internet is pretty much toast. If they turn evil, we are in serious trouble. The technical hurdles are also real, I am not deluding myself. From an open source point of view, it's the only way to run an open source search engine, there is no money for a centralized facility. I found this related project yacy.net. It requires dedicated servers, so it might not amenable to high churn peers such as browsers, nor it can leverage the implicit manual curation that a user can provide, yet it is very relevant. I hope this addresses some of your objections.

Answer 3

Beeseek?

It's a P2P search engine under development, maybe a Firefox plugin could fix integration!

http://www.beeseek.org/