Why does tokyo tyrant slow down exponentially even after adjusting bnum?

Posted by HenryL on Stack Overflow See other posts from Stack Overflow or by HenryL
Published on 2009-06-27T00:47:59Z Indexed on 2010/05/05 17:48 UTC
Read the original article Hit count: 215

Filed under:

tokyocabinet

Has anyone successfully used Tokyo Cabinet / Tokyo Tyrant with large datasets? I am trying to upload a subgraph of the Wikipedia datasource. After hitting about 30 million records, I get exponential slow down. This occurs with both the HDB and BDB databases. I adjusted bnum to 2-4x the expected number of records for the HDB case with only a slight speed up. I also set xmsiz to 1GB or so but ultimately I still hit a wall.

It seems that Tokyo Tyrant is basically an in memory database and after you exceed the xmsiz or your RAM, you get a barely usable database. Has anyone else encountered this problem before? Were you able to solve it?

Related posts about tokyocabinet

tokyo cabinet & tyrant tutorial

as seen on Stack Overflow - Search for 'Stack Overflow'
Can somebody refer a good tutorial on tokyo cabinet and tokyo tyrant >>> More
BerkeleyDB vs. Tokyo Cabinet

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm looking for general experiences from people who have used both, particularly on how the two compare on handling large numbers of records, transaction/concurrency/deadlock handling, and juicy stories about database corruption and backup procedures. >>> More
Any chance to get Core Data using Tokyo Cabinet as the persistent store?

as seen on Stack Overflow - Search for 'Stack Overflow'
I watched a free high quality video with Aaron Hillegass about Core Data vs Tokyo Cabinet. Besides that this guy is amazingly funny (really, if you want to laugh now, watch it!), he shows off Tokyo Cabinet beeing about 40x faster than Core Data. I wonder if it's worth thinking about how to attach… >>> More
Ruby Rack: startup and teardown operations (Tokyo Cabinet connection)

as seen on Stack Overflow - Search for 'Stack Overflow'
I have built a pretty simple REST service in Sinatra, on Rack. It's backed by 3 Tokyo Cabinet/Table datastores, which have connections that need to be opened and closed. I have two model classes written in straight Ruby that currently simply connect, get or put what they need, and then disconnect… >>> More
How to store an array as the value in Tokyo Cabinet?

as seen on Stack Overflow - Search for 'Stack Overflow'
Is there any way I can store an array of numbers in a Tokyo Cabinet db? For example, I have predictable arrays of values such as 1 => [1, 2, 444, 0.987], 2 => [2, 23, 123, -0.234], 3 => [3, 1, 34, 1.456] I would like to store the above in a TC fixed length db. Is there a way to store… >>> More

Developer IT