Move files from ftp server to s3

Posted by lev on Server Fault See other posts from Server Fault or by lev
Published on 2013-11-11T09:12:07Z Indexed on 2013/11/11 9:59 UTC
Read the original article Hit count: 225

I would like to set up an ftp server, where users will upload files, and for each file, put it on s3 storage, and delete it from the ftp server. (the server runs on ec2 ubuntu)

Here are the stuff I already tried, with no success..

  1. Mount s3 bucket using s3fs. I followed those instructions, but there is a bug in the latest version of s3fs, that prevents it from working. The bug was fixed on the develop branch, but I don't want to use unstable version on my production.

  2. Use vsftpd and using s3cmd sync via cron to sync the files periodically. The problem with that approach, is that s3cmd can start running in the middle of a file upload, and start synching the incomplete file. Also s3cmd doesn't give any feedback it the sync fails, so I have no way of knowing if I can delete the files after the sync command finished running.

  3. Use pure-ftpd's upload script feature (which allows to run a script after a file is finished uploading), but I noticed that if the file upload was failed in the middle, the script will run anyway, and I have no way of knowing if the upload was successful or not.

I've been at it for a few days now, and I'm at a loss here. Any suggestions will be welcomed.

© Server Fault or respective owner

Related posts about ubuntu

Related posts about amazon-ec2