What is the most efficient way to store a mapping "key -> event stream"?

Posted by jkff on Stack Overflow See other posts from Stack Overflow or by jkff
Published on 2010-03-17T09:07:21Z Indexed on 2010/03/17 9:11 UTC
Read the original article Hit count: 256

Suppose there are ~10,000's of keys, where each key corresponds to a stream of events. I'd like to support the following operations:

  • push(key, timestamp, event) - pushes event to the event queue for key, marked with the given timestamp. It is guaranteed that event timestamps for a particular key are pushed in sorted or almost sorted order.
  • tail(key, timestamp) - get all events for key since the given timestamp. Usually the timestamp requests for a given key are almost monotonically increasing, almost synchronously with pushes for the same key.

This stuff has to be persistent (although it is not absolutely necessary to persist pushes immediately and to keep tails with pushes strictly in sync), so I'm going to use some kind of database.

What is the optimal kind of database structure for this task? Would it be better to use a relational database, a key-value storage, or something else?

© Stack Overflow or respective owner

Related posts about database

Related posts about Performance