Can EC2 instances be set up to come from different IP ranges?

Posted by Joshua Frank on Stack Overflow See other posts from Stack Overflow or by Joshua Frank
Published on 2010-06-03T12:52:51Z Indexed on 2010/06/03 13:24 UTC
Read the original article Hit count: 277

Filed under:
|
|

I need to run a web crawler and I want to do it from EC2 because I want the HTTP requests to come from different IP ranges so I don't get blocked. So I thought distributing this on EC2 instances might help, but I can't find any information about what the outbound IP range will be. I don't want to go to the trouble of figuring out the extra complexity of EC2 and distributed data, only to find that all the instances use the same address block and I get blocked by the server anyway.

NOTE: This isn't for a DoS attack or anything. I'm trying to harvest data for a legitimate business purpose, I'm respecting robots.txt, and I'm only making one request per second, but the host is still shutting me down.

Edit: Commenter Paul Dixon suggests that the act of blocking even my modest crawl indicates that the host doesn't want me to crawl them and therefore that I shouldn't do it (even assuming I can work around the blocking). Do people agree with this?

© Stack Overflow or respective owner

Related posts about webcrawling

Related posts about ec2