Versioning freindly, extendible binary file format
- by Bas Bossink
In the project I'm currently working on there is a need to save a sizeable data structure to disk. Being in optimist I thought their must be a standard solution for such a problem however upto now I haven't found a solution that satisfies the following requirements:
.net 2.0 support, preferably with a foss implementation
version friendly (this should be interpreted as reading an old version of the format should be relatively simple if the changes in the underlying data structure are simple, say adding/dropping fields)
ability to do some form of random access where part of the data can be extended after initial creation (think of this as extending intermediate results)
space and time efficient (xml has been excluded as option given this requierement)
Options considered so far:
Protocol Buffers : was turned down by verdict of the documentation about Large Data Sets since this comment suggest adding another layer on top, this would call for additional complexity which I wish to have handled by the file format itself.
HDF5,EXI : do not seem to have .net implementations
SQLite : the data structure at hand would result in a pretty complex table structure that seems to heavyweight for the intended use
BSON : does not appear to support requirement 3.
Fast Infoset : only seems to have buyware .net implementations
Any recommendations or pointers are greatly appreciated. Furthermore if you believe any of the information above is not true please provide pointers/examples to proove me wrong.
