How important is index size when searching?

Posted by Michael K on Programmers See other posts from Programmers or by Michael K
Published on 2012-03-27T14:28:51Z Indexed on 2012/03/27 17:42 UTC
Read the original article Hit count: 225

Filed under:
|

My company has recently began using Apache Solr to search its data. As we learn how to use it we have gone down the path of indexing multiple fields to get the results we need. Most of these are either N-Grammed or Edge-N-Grammed.

Gramming by nature takes up a lot of space, which takes more time to search. Space is cheap, but time is less so. Index time is not too important, since a delta-import (only get the changes since last index) is extremely quick and you only pay a penalty on the first import. What we've not been able to determine is what effect the index size has on query times. Obviously a larger index takes longer to search, but the time added by n-gramming a field is difficult to predict.

How do you determine whether a field is worth gramming? Can you predict how much longer a query will take when you gram a field?

© Programmers or respective owner

Related posts about design

Related posts about lucene