Why is piping dd through gzip so much faster than a direct copy?

Posted by Foo Bar on Super User See other posts from Super User or by Foo Bar
Published on 2014-05-29T08:35:55Z Indexed on 2014/05/29 9:32 UTC
Read the original article Hit count: 178

Filed under:
|
|
|
|

I wanted to backup a path from a computer in my network to another computer in the same network over a 100MBit/s line. For this I did

dd if=/local/path of=/remote/path/in/local/network/backup.img

which gave me a very low network transfer speed of something about 50 to 100 kB/s, which would have taken forever. So I stopped it and decided to try gzipping it on the fly to make it much smaller so that the amount to transfer is less. So I did

dd if=/local/folder | gzip > /remote/path/in/local/network/backup.img.gz

But now I get something like 1 MB/s network transfer speed, so a factor of 10 to 20 faster. After noticing this, I tested this on several paths and files and it was always the same.

Why does piping dd through gzip also increase the transfer rates by a large factor instead of only reducing the bytelength of the stream by a large factor? I'd expected even a small decrease in transfer rates instead, due to the higher CPU consumption while compressing, but now I get a double plus. Not that I'm not happy, but just wondering. ;)

© Super User or respective owner

Related posts about backup

Related posts about Performance