Cygwin's RSYNC for large data transfer
- by Tim Brigham
I'm using rsync from Cygwin to do a large scale data transfer from an aging HP MSA 1000 to a new DAS attached to a different server. I have a daemon running on the remote server in read only mode and a local copy writing the files to disk.
One of my servers is an image repository with over a million files spread across about 300 directories. Each file averages only a couple hundred kilobytes. More so than any other box this one is proving problematic.
The rsync process will work for a while - some times 20 minutes, some times an hour - and then it simply quits and sits idle at a given file name.
I have verified that the file isn't corrupt on the remote server and that the file is successfully created on the local drive. I ran the rsync client in -vv mode, which returns nothing. I checked out the logs created by the daemon. I looked at the network utilization on the interface, which is sitting idle. I looked at the AV settings to see if anything could pose a problem there. I even updated to the latest release of Cygwin.
What do I need to in order to keep this connection up?
EDIT:
The client system is using the command
rsync.exe server::Drives/f/Repo/ /cygdrive/T/Repo --archive -P -vv
The server is using the command
rsync.exe --daemon --no-detach --config "rsyncd.conf"
The contents of rsyncd.conf:
use chroot = false
strict modes = false
hosts allow = 192.168.100.9
log file = c:/rsyncd.log
uid=0
gid=0
[Drives]
path = /cygdrive
read only = yes
EDIT:
The file server is 2003, the disk type on the array is GPT and the size is of the array is about 4 TB.
EDIT:
Stranger.. It looks like the process is reliably erroring out at about 175,000 files. Rsync runs fine when I pick the same directory it has problems with one at a time.
EDIT:
rsync version 3.0.9 protocol version 30
Copyright (C) 1996-2011 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
no socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, iconv, symtimes
A similar failure occurred when going from the same set of files with Cygwin to a Linux install. It didn't happen until several hours later than normal however.
