Move files from ftp server to s3
Posted
by
lev
on Server Fault
See other posts from Server Fault
or by lev
Published on 2013-11-11T09:12:07Z
Indexed on
2013/11/11
9:59 UTC
Read the original article
Hit count: 225
I would like to set up an ftp server, where users will upload files, and for each file, put it on s3 storage, and delete it from the ftp server. (the server runs on ec2 ubuntu)
Here are the stuff I already tried, with no success..
Mount s3 bucket using s3fs. I followed those instructions, but there is a bug in the latest version of s3fs, that prevents it from working. The bug was fixed on the develop branch, but I don't want to use unstable version on my production.
Use vsftpd and using s3cmd sync via cron to sync the files periodically. The problem with that approach, is that s3cmd can start running in the middle of a file upload, and start synching the incomplete file. Also s3cmd doesn't give any feedback it the sync fails, so I have no way of knowing if I can delete the files after the sync command finished running.
Use pure-ftpd's upload script feature (which allows to run a script after a file is finished uploading), but I noticed that if the file upload was failed in the middle, the script will run anyway, and I have no way of knowing if the upload was successful or not.
I've been at it for a few days now, and I'm at a loss here. Any suggestions will be welcomed.
© Server Fault or respective owner