Why is uploading to S3 so slow?

Posted by Tom Marthenal on Server Fault See other posts from Server Fault or by Tom Marthenal
Published on 2012-04-08T17:40:03Z Indexed on 2012/11/21 5:04 UTC
Read the original article Hit count: 497

I am using s3cmd to upload to S3:

# s3cmd put 1gb.bin s3://my-bucket/1gb.bin
1gb.bin -> s3://my-bucket/1gb.bin  [1 of 1]
  366706688 of 1073741824    34% in  371s   963.22 kB/s

I am uploading from Linode, which has an outgoing bandwidth cap of 50 Mb/s according to support (roughly 6 MB/s).

Why am I getting such slow upload speeds to S3, and how can I improve them?


Update:

Uploading the same file via SCP to an m1.medium EC2 instance (SCP from my Linode to the instance's EBS drive) gives about 44 Mb/s according to iftop (any compression done by the cipher is not a factor).


Traceroute: Here's a traceroute to the server it's uploading to (according to tcpdump).

# traceroute s3-1-w.amazonaws.com.
traceroute to s3-1-w.amazonaws.com. (72.21.194.32), 30 hops max, 60 byte packets
 1  207.99.1.13 (207.99.1.13)  0.635 ms  0.743 ms  0.723 ms
 2  207.99.53.41 (207.99.53.41)  0.683 ms  0.865 ms  0.915 ms
 3  vlan801.tbr1.mmu.nac.net (209.123.10.9)  0.397 ms  0.541 ms  0.527 ms
 4  0.e1-1.tbr1.tl9.nac.net (209.123.10.102)  1.400 ms  1.481 ms  1.508 ms
 5  0.gi-0-0-0.pr1.tl9.nac.net (209.123.11.62)  1.602 ms  1.677 ms  1.699 ms
 6  equinix02-iad2.amazon.com (206.223.115.35)  9.393 ms  8.925 ms  8.900 ms
 7  72.21.220.41 (72.21.220.41)  32.610 ms  9.812 ms  9.789 ms
 8  72.21.222.141 (72.21.222.141)  9.519 ms  9.439 ms  9.443 ms
 9  72.21.218.3 (72.21.218.3)  10.245 ms  10.202 ms  10.154 ms
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

The latency looks reasonable, at least until the server stopped responding to ping requests.

© Server Fault or respective owner

Related posts about amazon-web-services

Related posts about amazon-s3