Is there a good open source search engine including indexing bot which can be used to make up speci

Posted by Skuta on Stack Overflow See other posts from Stack Overflow or by Skuta
Published on 2008-10-18T23:52:01Z Indexed on 2010/04/22 10:13 UTC
Read the original article Hit count: 166

Filed under:

free

|

search

|

engine

|

crawl

|

bot

Hello,

Our application (C#/.NET) needs a lot of queries to search. Google's 50,000 policy per day is not enough. We need something that would crawl Internet websites by specific rules we set (for ex. country domains) and gather URLs, Texts, keywords, name of websites and create our own internal catalogue so we wouldn't be limited to any massive external search engine like Google or Yahoo.

Is there any free open source solution we could use to install it on our server?

No point in re-inventing the wheel.

© Stack Overflow or respective owner

Related posts about free

Tools and Utilities for the .NET Developer

as seen on Geeks with Blogs - Search for 'Geeks with Blogs'
Tweet this list! Add a link to my site to your bookmarks to quickly find this page again! Add me to twitter! This is a list of the tools/utilities that I use to do my job/hobby. I wanted this page to load fast and… >>> More
Don’t miss this very popular presentation on Punchout in iProcurement on June 26th 2012

as seen on Oracle Blogs - Search for 'Oracle Blogs'
Don’t miss this very popular presentation on Punchout in iProcurement on June 26th. See Doc ID 1448447.1 for the Webcast details. … >>> More
Help with malloc and free: Glibc detected: free(): invalid pointer

as seen on Stack Overflow - Search for 'Stack Overflow'
I need help with debugging this piece of code. I know the problem is in malloc and free but can't find exactly where, why and how to fix it. Please don't answer: "Use gdb" and that's it. I would use gdb to debug it, but I still don't know much about it and am still learning it, and would like to have… >>> More
FREE Windows Azure evening in London on April 15th including FREE access to Windows Azure

as seen on Geeks with Blogs - Search for 'Geeks with Blogs'
[Did I overdo the use of FREE in the title? :-)] April 12th to 16th is Microsoft Tech Days – 5 days of sessions on Visual Studio 2010 through to Windows 7 Phone Series. Many of these days are now full (Tip - Thursday still has room if rich client applications is your thing) but the good news is the… >>> More
Do you know of any "source control" like free services which are available for free

as seen on Stack Overflow - Search for 'Stack Overflow'
Me and a friend are going to start development on a project soon and are looking for "source control" like free app to use, the application will be developed using visual studio. Thanks in advance. >>> More

Related posts about search

"Error in the Site Data Web Service." when performing crawl

as seen on Server Fault - Search for 'Server Fault'
Installed SharePoint Services v3 (SP2, october 2009 cumulative updates, Language Pack), attached to a content database I had previously (all works). Installed Search server 2008 Express (with language pack) on top of WSS and crawl does not work. However it works for newly created web application +… >>> More
Search Alternative Search Engines from within Bing’s Search Page

as seen on How to geek - Search for 'How to geek'
So you love using Bing Search but may still be curious to see what another search engine will provide if used. Now you can search using another search engine from within the Bing Search page and enjoy numbered results using two simple user scripts. Note: These user scripts may also be added to other… >>> More
CONVERT(int, (datepart(month, @search)), (datepart(day, @search)), DateAdd(year, Years.Year - (datepart(year, @search)))

as seen on Stack Overflow - Search for 'Stack Overflow'
In the query the top part is getting all the years that will run in the stored procedure. Works fine But at first i just wanted to run the queries for yesterdays date for all the years, but now i realized i want the user to select a date that will be in a parameter @search Booked <= CONVERT(int… >>> More
Am?lioration du Search MOSS: synonyme et Best Bet - La gestion des synonymes dans MOSS Search

as seen on ASP-PHP.net - Search for 'ASP-PHP.net'
Le moteur de recherche de MOSS permet la configuration d'une liste de synonymes. Nous verrons donc dans cet article comment effectuer cette tache et ce que cela peut apporter ? vos utilisateurs. Nous verrons aussi comment automatiser un peu plus cette configuration par l'utilisation de code ou d'outils… >>> More
Utiliser un MOSS 2007 Search avec SPS 2003 - Comment utiliser un MOSS Search avec SPS Portail

as seen on ASP-PHP.net - Search for 'ASP-PHP.net'
Microsoft Office SharePoint Server 2007 (MOSS) fournit de nombreuses fonctionnalit?s qui ne sont pas disponibles sous SharePoint Portal Server 2003 (SPS). C'est particuli?rement vrai pour le moteur de recherche. Ce moteur de recherche peut pourtant ?tre utilis? sans attendre une ?volution du site… >>> More