Ruby: Streaming large AWS S3 object freezes
- by Peter
Hi, I am using the ruby aws/s3 library to retrieve files from Amazon S3. I stream an object and write it to file as per the documentation (with debug every 100 chunks to confirm progress)
This works for small files, but randomly freezes downloading large (150MB) files on VPS Ubuntu. Fetching the same files (150MB) from my mac on a much slower connection works just fine.
When it hangs there is no error thrown and the last line of debug output is the 'Finished chunk'. I've seen it write between 100 and 10,000 chunks before freezing.
Anyone come across this or have ideas on what the cause might be?
Thanks
The code that hangs:
i=1
open(local_file, 'w') do |f|
AWS::S3::S3Object.value(key, @s3_bucket) do |chunk|
puts("Writing chunk #{i}")
f.write chunk.read_body
puts("Finished chunk #{i}")
i=i+1
end
end
