One Way Sync of a Bucket With Local Directory

Posted by user48651 on Server Fault See other posts from Server Fault or by user48651
Published on 2012-11-17T19:09:42Z Indexed on 2012/11/17 23:05 UTC
Read the original article Hit count: 274

Filed under:

amazon-s3

|

s3cmd

I have a local directory that I would like to synchronize with an S3 bucket. I have two specific requirements:

If local file is the same as the remote, do not re-transfer it to the bucket.
If some files or directories exist in the bucket but do not exist on local, delete them. Basically the bucket should mirror the local copy and not vice-versa.

I looked into s3cmd sync command, but unfortunately requirement 2 is not fulfilled. If files exists in the bucket but not on local copy, they will be copied to the local instead of being deleted.

© Server Fault or respective owner

Related posts about amazon-s3

Amazon S3 Tips: Quickly Add/Modify HTTP Headers To All Files Recursively

as seen on Tech Dreams - Search for 'Tech Dreams'
Amazon S3 is an dead cheap cloud storage service that offers unlimited storage in pay as you use model. Recently we moved all the images and other static files(scripts & css) of Tech Dreams to Amazon S3 to reduce load on VPS server. Amazon S3 is cheap, but monthly bill will shoot up if images/static… >>> More
Alternative to Amazon's S3 service?

as seen on Super User - Search for 'Super User'
Just wondering if there is good alternative to Amazon's S3 service? I like S3 but the bandwidth cost is high. I looked at CouldFiles from Rackspace but the cost is even higher. I don't mind prepaying or having monthly payment in order to reduce the bandwidth cost greatly. Thank you for any help >>> More
Alternative to Amazon’s S3 service?

as seen on Stack Overflow - Search for 'Stack Overflow'
Just wondering if there is good alternative to Amazon's S3 service? I like S3 but the bandwidth cost is high. I looked at CouldFiles from Rackspace but the cost is even higher. I don't mind prepaying or having monthly payment in order to reduce the bandwidth cost greatly. Thank you for any help >>> More
Saving Python Complex Data Types to Amazon S3

as seen on Stack Overflow - Search for 'Stack Overflow'
Can Python class data be saved to S3 without marshalling? I am trying to cut down of I/O operations until necessary. >>> More
Filesize with SWFUpload and Amazon S3

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello all, I'm currently using SWFUpload to upload files to my S3 bucket. And it's working great. I'm using the script from a website here: http://www.anedix.com/news/article/50 Again, the upload to my S3 works fine, however, I've been running into an issue when attempting to upload larger… >>> More

Related posts about s3cmd

s3cmd setacl to grant 'Authenticated Users'

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm using jgit to create a remote in s3. Problem I'm having is when I do a jgit push s3 master it creates the files in s3 as owned by me. I want to keep the files private, and be read/write by 'Authenticated users'. I'd like to be able to either set acl: authenticated-read in the ~/.jgit file OR… >>> More
s3cmd fails too many times

as seen on Server Fault - Search for 'Server Fault'
It used to be my favorite backup transport agent but now I frequently get this result from s3cmd on the very same Ubuntu server/network: root@server:/home/backups# s3cmd put bkup.tgz s3://mybucket/ bkup.tgz -> s3://mybucket/bkup.tgz [1 of 1] 36864 of 2711541519 0% in 1s 20.95… >>> More
s3cmd run on command line not on cron

as seen on Server Fault - Search for 'Server Fault'
Many have said that the problem is with environment but I still can't seem to solve this problem. BTW I am using Ubuntu 9.10 login as user, then sudo -s using this command: s3cmd put file s3://bucket worked! now here is the simple script intended for testing: #! /bin/bash env >/tmp/cronjob… >>> More
How to sort output of "s3cmd ls"

as seen on Stack Overflow - Search for 'Stack Overflow'
Amazon "s3cmd ls" takes like this output: 2010-02-20 21:01 1458414588 s3://file.tgz.00 2010-02-20 21:10 1458414527 s3://file.tgz.01 2010-02-20 22:01 1458414588 s3://file.tgz.00 2010-02-20 23:10 1458414527 s3://file.tgz.01 2010-02-20 23:20 1458414588 s3://file.tgz.02 How to select all files… >>> More
define software path

as seen on Server Fault - Search for 'Server Fault'
I have an older version already installed. I have upraded the package using setup.py install command. But the path is not correctly set. When I type "s3cmd" is shows the older version of software. # s3cmd s3cmd [options] <command> [arg(s)] version 1.2.6 --help -h --verbose… >>> More