Lucene search taking TOOO long.
Posted
by Josh Handel
on Stack Overflow
See other posts from Stack Overflow
or by Josh Handel
Published on 2010-06-03T20:26:26Z
Indexed on
2010/06/06
7:02 UTC
Read the original article
Hit count: 390
I;m using Lucene.net (2.9.2.2) on a (currently) 70Gig index.. I can do a fairly complicated search and get all the document IDs back in 1 ~ 2 seconds.. But to actually load up all the hits (about 700 thousand in my test queries) takes 5+ minutes.
We aren't using lucene for UI, this is a datastore between processes where we have hundreds of millions of pre-cached data elements, and the part I am working on exports a few specific fields from each found document. (ergo, pagination doesn't make since as this is an export between processes).
My question is what is the best way to get all of the documents in a search result? currently I am using a custom collector that does a get on the document (with a MapFieldSelector) as its collecting.. I've also tried iterating through the list after the collector has finished.. but that was even worse.
I'm open to ideas :-).
Thanks in advance.
© Stack Overflow or respective owner