Storing lots of large strings with frequent "appends" and few reads

Posted by Thiago Moraes on Programmers See other posts from Programmers or by Thiago Moraes
Published on 2012-12-08T00:38:24Z Indexed on 2012/12/08 11:35 UTC
Read the original article Hit count: 335

Filed under:

database

|

strings

|

file-handling

|

file-systems

|

storage

In my current project, I need to store a very long ASCII string to each instance of a given object. This string will receive an 2 appends per minute and will not be retrieved so frequently.

The worst case scenario is a 5-10MB string. I'll have thousands of instances of my object and I'm worried that storing all those strings in the filesystem would not be optimal, but I can't think of a better solution.

Can anyone suggest an alternative? Maybe a key-value store? In this case, which one? Any other thoughts?

© Programmers or respective owner

Related posts about database

SQL SERVER Retrieve and Explore Database Backup without Restoring Database Idera virtual database

as seen on Dot net Slackers - Search for 'Dot net Slackers'
I recently downloaded Ideras SQL virtual database, and tested it. There are a few things about this tool which caught my attention.My ScenarioIt is quite common in real life that sometimes observing or retrieving older data is necessary; however, it had changed as time passed by. The full database… >>> More
Cloning A Database On The Same Server Using Rman Duplicate From Active Database

as seen on Oracle Blogs - Search for 'Oracle Blogs'
To clone a database using Rman we used to require an existing Rman backup, on 11g we can clone databases using the "from active" database option. In this case we do not require an existing backup, the active datafiles will be used as the source for the clone. In order to clone with the source database… >>> More
cPickle ImportError: No module named multiarray

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, I'm using cPickle to save my Database into file. The code looks like that: def Save_DataBase(): import cPickle from scipy import * from numpy import * a=Results.VersionName #filename='D:/results/'+a[a.find('/')+1:-a.find('/')-2]+Results.AssType[:3]+str(random.randint(0,100))+Results.Distribution+"… >>> More
SQL SERVER – 2008 – Introduction to Snapshot Database – Restore From Snapshot

as seen on SQL Authority - Search for 'SQL Authority'
Snapshot database is one of the most interesting concepts that I have used at some places recently. Here is a quick definition of the subject from Book On Line: A Database Snapshot is a read-only, static view of a database (the source database). Multiple snapshots can exist on a source database and… >>> More
OTN ???? ?????? ???????

as seen on Oracle Blogs - Search for 'Oracle Blogs'
Database ?? Database ??????? Database ?????????? Java WebLogic Server/????????·???? SOA/BPM/????? ???????/???? ID??/?????? ?????EPM/BI EPM/BI ??????? EPM/BI ???? OS/??? ???? ????? MySQL Database ?? ???? ?? ????????? ??? ?? ORACLE MASTER… >>> More

Related posts about strings

References to other strings in strings.xml ?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi Is it possible to reference other strings inside of strings.xml ? Something of the form: My string Here it is: android:id="@string/string_one" (If it did exist, there would of course be problems of circular, infinite definitions, etc. to beware of). Thanks. >>> More
Passing List of Strings or Array of strings into Unity Injection Constructor (Config-Based)

as seen on Stack Overflow - Search for 'Stack Overflow'
I cannot seem to get unity working when attempting to pass in an array of strings into a constructor parameter list, while using XML configuration. When I try the following: <typeConfig ...> <constructor ...> <param ... parameterType="System.String[]"> <array>… >>> More
Strings in array are no longer strings after jQuery.each()

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm pretty confused with the behaviour of arrays of strings when I loop them through the jQuery.each() method. Apparently, the strings become jQuery objects inside the callback function. However, I cannot use the this.get() method to obtain the original string; doing so triggers a this.get is not… >>> More
Complete if statement with strings and array of strings

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a page that has 3 variables. They look like this: String[] Headers = new String[] { "Max Width", "Max Length", "Max Height" }; String currentHeader = (String)HttpContext.Current.Request.QueryString["ItemHas"] ?? ""; String checkString = (String)HttpContext.Current.Request.QueryString["ItemIn"]… >>> More
[C++] Index strings by other strings

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, I need to index specific strings with other strings and I can't really find a good way to do so. I tried to use tr1::unordered_map, but I'm having some difficulties using it. If someone could tell me what is the best way to do that I'd be really grateful :) I also need to index objects by a… >>> More