Compress snapshot backups with duplicates

Posted by Usavich on Super User See other posts from Super User or by Usavich
Published on 2012-11-10T22:03:08Z Indexed on 2012/11/10 23:03 UTC
Read the original article Hit count: 123

Filed under:

backup

|

compression

I have a set of backups of mostly photos. The directory looks sort of like this:

backup/Day1/photos/1.jpg
               .../2.jpg
backup/Day2/photos/2.jpg
               .../3.jpg
               .../4.jpg
backup/DayN/photos/2.jpg
               .../3.jpg
               .../9.jpg

Files with same name are identical. There are many duplicates. Due to the way the backup system works, it's not possible to create incremental backup directly. I always get the entire dump each day.

If I want to create a compressed archive for a date range, say Day 5~9, what is the best tool/compression algorithm to do that?

© Super User or respective owner

Related posts about backup

backup exec - backup to disk offline

as seen on Server Fault - Search for 'Server Fault'
Hi We are running backup exec 9.1 doing a backup to disk to portable hard disk drives. When we run the backup manually it works fine. But when the backup is setup to run in the evening on a schedule it does not run as the backup to disk folders goes offline and therefore has to be switched back… >>> More
Ideal backup appliance for backup software like Bacula?

as seen on Server Fault - Search for 'Server Fault'
I'm at a small company and we (the IT department of two) manage <100 client computers and a handful of servers. Currently we're using a company's appliance to handle backup; it does a small backup every night and a full backup every weekend, and a guy comes on Wednesday to take an offsite backup… >>> More
Symantec Backup Exec Error on backup

as seen on Server Fault - Search for 'Server Fault'
Recently we have moved some of servers from real servers- into virtual servers. Since then, we are getting errors like the following: Error category : Resource ErrorsError : e000fed1 - A failure occurred querying the Writer status. For additional information regarding this error refer… >>> More
Windows Server Backup - Recover only shows the latest backup

as seen on Server Fault - Search for 'Server Fault'
We're having quite some trouble at work using Windows Server Backup. We have a HyperV server (Win 2008) running 8 virtual web servers, these are running a variety of OS'es: Win 2003, Win 2008 and a lone Debian. Each virtual server has a separate partition on the physical HyperV server, so e.g. E:… >>> More
Failed Backup Job With Backup Exec 12 and AOFO

as seen on Server Fault - Search for 'Server Fault'
I am backing up a Windows 2003 Small Business Server with SP2. We are running Backup Exec 12 with SP4. Recently the backup job started failing on backing up the system state with the following error: V-79-57344-34110 - AOFO: Initialization failure on: "System?State". Advanced Open File Option… >>> More

Related posts about compression

Converting linear colors to SRGB shows banding in FFmpeg

as seen on Super User - Search for 'Super User'
When I convert an EXR file sequence with x264 using FFmpeg and convert the colorspace from linear to SRGB (with gamma 0.45454545) I get some heavy banding issues (most visible on a dark gradient). Here is the ffmpeg command I use: C:/ffmpeg.exe -y -i C:/seq_v001.%04d.exr -vf lutrgb=r=gammaval(0… >>> More
IIS7 Compression CSS files only compressed when dynamic compression is enabled

as seen on Server Fault - Search for 'Server Fault'
If anyone can help it would be appreciated. I would like to enable compression for static files within IIS7 (for the sake of simplicity I'll just refer to static css files for the time being). The problem I'm getting is that css files are only compressed when both dynamic and static compression… >>> More
Built-in GZip/Deflate Compression on IIS 7.x

as seen on West-Wind - Search for 'West-Wind'
IIS 7 improves internal compression functionality dramatically making it much easier than previous versions to take advantage of compression that’s built-in to the Web server. IIS 7 also supports dynamic compression which allows automatic compression of content created in your own applications (ASP… >>> More
Domino Document data compression and design compression

as seen on Server Fault - Search for 'Server Fault'
I was thinking of turning this on some large databases not just mail files - we have around 8 - 10GB of large databases as well as small databases of couple of hundred MB in size. But after reading this post I am not too sure: http://www-10.lotus.com/ldd/nd85forum.nsf/4b9931b774db788c85256bf0006b5e6d/1f4e67b569720e54852576c0003cb8ac… >>> More
I need to choose a compression algorithm

as seen on Stack Overflow - Search for 'Stack Overflow'
I need to choose a compression algorithm to compress some data. I don't know the type of data I'll be compressing in advance (think of it as kinda like the WinRAR program). I've heard of the following algorithms but I don't know which one I should use. Can anyone post a short list of pros and cons… >>> More