Options to efficiently synchronize 1 million files with remote servers?

Posted by Zilvinas on Server Fault See other posts from Server Fault or by Zilvinas
Published on 2012-06-15T09:38:48Z Indexed on 2012/06/15 15:18 UTC
Read the original article Hit count: 174

At a company I work for we have such a thing called "playlists" which are small files ~100-300 bytes each. There's about a million of them. About 100,000 of them get changed every hour. These playlists need to be uploaded to 10 other remote servers on different continents every hour and it needs to happen quick in under 2 mins ideally. It's very important that files that are deleted on the master are also deleted on all the replicas. We currently use Linux for our infrastructure.

I was thinking about trying rsync with the -W option to copy whole files without comparing contents. I haven't tried it yet but maybe people who have more experience with rsync could tell me if it's a viable option?

What other options are worth considering?

© Server Fault or respective owner

Related posts about rsync

Related posts about optimization