binary file formats: need for error correction?

Posted by Jason S on Stack Overflow See other posts from Stack Overflow or by Jason S
Published on 2010-05-27T13:30:28Z Indexed on 2010/05/27 13:31 UTC
Read the original article Hit count: 267

Filed under:

robustness

|

error-detection

|

binary-file

I need to serialize some data in a binary format for efficiency (datalog where 10-100MB files are typical), and I'm working out the formatting details. I'm wondering if realistically I need to worry about file corruption / error correction / etc.

What are circumstances where file corruption can happen? Should I be building robustness to corruption into my binary format? Or should I wrap my nonrobust-to-corruption stream of bytes with some kind of error correcting code? (any suggestions? I'm using Java) Or should I just not worry about this?

© Stack Overflow or respective owner

Related posts about robustness

Improve Microsoft Visual C++ Application Security and Robustness with SafeInt

as seen on Internet.com - Search for 'Internet.com'
In this age of cloud computing, massive parallel systems and complex security threats like identity theft and decentralized botnets, devoting resources to combat the seemingly age-old issue of integer overflow appears distinctly passC). Despite the fact that integer overflow is such a well know problem… >>> More
Linux software raid robustness

as seen on Server Fault - Search for 'Server Fault'
I have a 4 disk 5TB raid5 setup where a disk is showing signs of going down the drain. It is reporting media errors and from dmesg I can see that several read errors are corrected. smartctl does report "notifications" but no panic so far. Since new disks are rather expensive at the moment I am starting… >>> More
NFS robustness or weaknesses

as seen on Server Fault - Search for 'Server Fault'
I have 2 web servers with a load balancer in front of them. They both have mounted a nfs share, so that they can share some common files, like images uploaded from the cms and some run time generated files. Is nfs robust? Are there any specific weaknesses I should now about? I know it does not… >>> More
Linux software raid robustness for raid1 vs other raid levels

as seen on Super User - Search for 'Super User'
I have a raid5 running and now also a raid1 that I set up yesterday. Since raid5 calculates parity it should be able to catch silent data corruption on one disk. However for raid1 the disks are just mirrors. The more I think about it I figure that raid1 is actually quite risky. Sure it will save me… >>> More
Reading data from a socket, considerations for robustness and security

as seen on Stack Overflow - Search for 'Stack Overflow'
I am writing a socket server that will implement small portions of the HTTP and the WebSocket protocol, and I'm wondering what I need to take into consideration in order to make it robust/secure. This is my first time writing a socket-based application so please excuse me if any of my questions are… >>> More

Related posts about error-detection

elisp compile, add a regexp to error detection

as seen on Stack Overflow - Search for 'Stack Overflow'
I am starting with emacs, and don't know much elisp. Nearly nothing, really. I want to use ack as a replacement of grep. These are the instructions I followed to use ack from within emacs: http://www.rooijan.za.net/?q=ack_el Now I don't like the output format that is used in this el file, I would… >>> More
error detection/correction/recovery in serial protocols

as seen on Stack Overflow - Search for 'Stack Overflow'
I have some designing to do for a serial protocol and am running into some questions that I figure must have been considered elsewhere. So I'm wondering if there are some recommendations for best practices in designing serial protocols. (Please either state a fact that is easily verifiable, or cite… >>> More
Unsuccessful error detection of improperly declared method in GCC 4.2 compiler

as seen on Stack Overflow - Search for 'Stack Overflow'
I am using C++ compiler GCC 4.2 in XCode 3.2.2. I have noted that the compiler will successfully compile a method foo even though there are no ellipses. The header and method are properly declared as foo(), but when I do a find and replace either by file or by program-wide it will miss approximately… >>> More
Good memory profiling, leak and error detection for Windows

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm currently looking for a good memory / leak detection tool for Windows. A few years ago, I used Numega's Boundschecker, which was VERY good. Right now it seems to have been sold to Compuware, which apparently sold it again to some other company. Trying to evaluate a demo of the current version… >>> More
Error trapping for a missing data source in a Spring MVC / Spring JDBC web app [migrated]

as seen on Programmers - Search for 'Programmers'
I have written a web app that uses Spring MVC libraries and Spring JDBC to connect to an Oracle DB. (I don't use any ORM type libraries as I create stored procedures on Oracle that do my stuff and I'm quite happy with that.) I use a connection pool to Oracle managed by the Tomcat container The app… >>> More