Comments on: Creating a Search Engine

By: jim cary

jim cary — Fri, 27 May 2011 05:58:57 +0000

its a good guideline for new peoples and i think its great

By: edward smith

edward smith — Mon, 23 May 2011 06:18:49 +0000

Hey this is good guideline for me.Thanks a lot.

By: Hot Water Systems

Hot Water Systems — Fri, 08 Apr 2011 08:17:58 +0000

find this article very informative this post really increase my knowledge

By: amit

amit — Fri, 15 Jun 2007 17:17:53 +0000

This is a wonderful idea. I have had experience in building vertical search engines… can we chatch up some time to discuss? Looking forward to hear from you.

Cheers

By: e.p.c.

e.p.c. — Wed, 30 May 2007 15:47:16 +0000

I have been using Yahoo!’s web services (http://developer.yahoo.com/search/) to provide the raw data for a vertical search engine I was working on. Assuming you are doing a basic crawl and index of much of the web there is little value in building your own search engine: you get stuck with a lot of capital costs and bandwidth costs for slurping all the data down. Now, if you are focussing on a small set of sites (small being a relative term, let’s say a specific industry vertical resulting in maybe 10,000 sites) then there might be value in truly building your own engine. But if you’re just screwing around, go with something like Yahoo’s service or the Amazon a9 search service (which is fed by Microsoft’s MSN/Live index). Google had a search service service as well (using SOAP) but have killed it in favour of an AJAX-y javascript based service. Technically easier to implement but you lose control over the UI and all interaction is with Google.

By: Adam Phillips

Adam Phillips — Wed, 30 May 2007 09:11:02 +0000

Darren,

I wrote my own search engine a couple of years ago, being entirely unimpressed with any offering which claimed to be a search engine but also runs on a windows platform – in short, there were none. You can see it in action at The Web’s Too Big : PR. I wouldn’t recommend attempting to write your own. I mean, it’s entirely possible, as I have proved, but don’t try it if you want to sleep for the next year or so.

The engine itself was written specifically to solve a problem for a vertical search engine. In its current usage it crawls and indexes the content of all PR companies in the UK, although this could easily be changed for whatever sites a user wishes.

I also tried Nutch, and at the time it wasn’t very good, although to be fair that was two years ago. I have a colleague who is using it now and apparently the documentation has got much better. Nutch is Java based and sits on Unix/Linux.

Anyway, drop me an email if you are really thinking about writing your own search engine. I have some handy tips I could share with you to spare you some pain.

Adam

By: Andrew

Andrew — Tue, 29 May 2007 14:49:21 +0000

Toss Lucene or Heritrix on a Rackspace server ($250/mo a pop) and you’re good to go. Almost every vertical search engine I’ve met has bootstrapped off Lucene or Heritrix. Some roll their own eventually (and a very small minority roll their own from the start), but both open source projects are robust, fast, and free, so I’d hire a consultant that has experience with either of their of these two crawlers.