Has anyone achieved true differential sync with rsync in ESXi?
- by Julius
Berate me later on the fact that I'm using the service console to do anything in ESXi...
I've got a working rsync binary (v3.0.4) that I can use in ESXi 4.1U1. I tend to use rsync over cp when copying VM's or backups from one local datastore to another local datastore. I've used rsync to copy data from one ESXi box to another but that was just for small files.
In now trying to do true differential syncs of backups taken via ghettoVCB between my primary ESXi machine and a secondary one. But even when I do this locally (one datastore to another datastore on the same ESXi machine) rsync appears to copy the files in their entirety. I've got two VMDK's totally 80GB in size, and rsync still takes anywhere between 1 and 2 hours but the VMDK's aren't growing that much daily.
Below is the rsync command I'm executing. I am copying locally because ultimately these files will get copied onto a datastore created from a LUN on a remote system. Its not an rsync that'll be serviced by an rsync daemon on a remote system.
rsync -avPSI VMBACKUP_2011-06-10_02-27-56/* VMBACKUP_2011-06-01_06-37-11/ --stats --itemize-changes --existing --modify-window=2 --no-whole-file
sending incremental file list
>f..t...... VM-flat.vmdk
42949672960 100% 15.06MB/s 0:45:20 (xfer#1, to-check=5/6)
>f..t...... VM.vmdk
556 100% 4.24kB/s 0:00:00 (xfer#2, to-check=4/6)
>f..t...... VM.vmx
3327 100% 25.19kB/s 0:00:00 (xfer#3, to-check=3/6)
>f..t...... VM_1-flat.vmdk
42949672960 100% 12.19MB/s 0:56:01 (xfer#4, to-check=2/6)
>f..t...... VM_1.vmdk
558 100% 2.51kB/s 0:00:00 (xfer#5, to-check=1/6)
>f..t...... STATUS.ok
30 100% 0.02kB/s 0:00:01 (xfer#6, to-check=0/6)
Number of files: 6
Number of files transferred: 6
Total file size: 85899350391 bytes
Total transferred file size: 85899350391 bytes
Literal data: 2429682778 bytes
Matched data: 83469667613 bytes
File list size: 129
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 2432530094
Total bytes received: 5243054
sent 2432530094 bytes received 5243054 bytes 295648.92 bytes/sec
total size is 85899350391 speedup is 35.24
Is this because ESXi is itself making so many changes to the VMDK's that as far as rsync is concerned the entire file has to be retransmitted?
Has anyone actually achieved actual diff sync with ESXi?
