Is there a compression method for compressing a group of very similar files, without archiving them?

Posted by awiebe on Super User See other posts from Super User or by awiebe
Published on 2012-05-31T20:05:01Z Indexed on 2012/05/31 22:44 UTC
Read the original article Hit count: 241

Filed under:

file-management

|

compression

|

lossless

I want to compress a large nuber of files that have near identical headers, and also some data, however I do not wish to archive them, nor do I wish to zip them individually(because the copression ratio would be much higher if substitutions of similar blocks could be done using a single table). Does a compression method exist to do this already, or should I implement it myself.

Note: Don't say "Disk space is cheap", because I may want to use this on an embedded system.

© Super User or respective owner

Related posts about file-management

An open source FileManagement component helps developing file-required systems easier in .NET

as seen on Code Project - Search for 'Code Project'
The component is designed to save the effort on developing file-required systems through its easy APIs and integration with ASP.NET web development, e.g. a product requires attachments and thumbnails in a product management system. >>> More
An open source FileManagement component helps developing file-required systems easier in .NET

as seen on Dot net Slackers - Search for 'Dot net Slackers'
The component is designed to save the effort on developing file-required systems through its easy APIs and integration with ASP.NET web development, e.g. a product requires attachments and thumbnails in a product management system....Did you know that DotNetSlackers also publishes .net articles written… >>> More
GUI FTP and File Management for Linux VPS

as seen on Super User - Search for 'Super User'
I'm interested in how I could remotely control FTP and file management on my Linux VPS with a GUI. I frequently transfer sites to my VPS for testing, and I'd much rather do it directly on the high bandwidth connection instead of my 10down, 2up Comcrap cable. >>> More
GUI FTP and File Management for Linux VPS

as seen on Server Fault - Search for 'Server Fault'
I'm interested in how I could remotely control FTP and file management on my Linux VPS with a GUI. I frequently transfer sites to my VPS for testing, and I'd much rather do it directly on the high bandwidth connection instead of my 10down, 2up Comcrap cable. >>> More
Help needed implementing a web based file management system with a file hierarchy system, help neede

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello i am trying to create a web application that will allow users to upload files online, i am using gwt while using hibernate for database communication, i am able to upload file to a server , and store them on the server. but what i want is to associate the files with a user. i want the user… >>> More

Related posts about compression

Converting linear colors to SRGB shows banding in FFmpeg

as seen on Super User - Search for 'Super User'
When I convert an EXR file sequence with x264 using FFmpeg and convert the colorspace from linear to SRGB (with gamma 0.45454545) I get some heavy banding issues (most visible on a dark gradient). Here is the ffmpeg command I use: C:/ffmpeg.exe -y -i C:/seq_v001.%04d.exr -vf lutrgb=r=gammaval(0… >>> More
IIS7 Compression CSS files only compressed when dynamic compression is enabled

as seen on Server Fault - Search for 'Server Fault'
If anyone can help it would be appreciated. I would like to enable compression for static files within IIS7 (for the sake of simplicity I'll just refer to static css files for the time being). The problem I'm getting is that css files are only compressed when both dynamic and static compression… >>> More
Built-in GZip/Deflate Compression on IIS 7.x

as seen on West-Wind - Search for 'West-Wind'
IIS 7 improves internal compression functionality dramatically making it much easier than previous versions to take advantage of compression that’s built-in to the Web server. IIS 7 also supports dynamic compression which allows automatic compression of content created in your own applications (ASP… >>> More
Domino Document data compression and design compression

as seen on Server Fault - Search for 'Server Fault'
I was thinking of turning this on some large databases not just mail files - we have around 8 - 10GB of large databases as well as small databases of couple of hundred MB in size. But after reading this post I am not too sure: http://www-10.lotus.com/ldd/nd85forum.nsf/4b9931b774db788c85256bf0006b5e6d/1f4e67b569720e54852576c0003cb8ac… >>> More
I need to choose a compression algorithm

as seen on Stack Overflow - Search for 'Stack Overflow'
I need to choose a compression algorithm to compress some data. I don't know the type of data I'll be compressing in advance (think of it as kinda like the WinRAR program). I've heard of the following algorithms but I don't know which one I should use. Can anyone post a short list of pros and cons… >>> More