Dynamically blocking excessive HTTP bandwidth use?

Posted by Jeff Atwood on Server Fault See other posts from Server Fault or by Jeff Atwood
Published on 2009-06-05T09:50:16Z Indexed on 2012/06/16 21:18 UTC
Read the original article Hit count: 244

Filed under:

networking

|

http

|

bandwidth

We were a little surprised to see this on our Cacti graphs for June 4 web traffic:

We ran Log Parser on our IIS logs and it turns out this was a perfect storm of Yahoo and Google bots indexing us.. in that 3 hour period, we saw 287k hits from 3 different google ips, plus 104k from yahoo. Ouch?

While we don't want to block Google or Yahoo, this has come up before. We have access to a Cisco PIX 515E, and we're thinking about putting that in front so we can dynamically deal with bandwidth offenders without touching our web servers directly.

But is that the best solution? I'm wondering if there is any software or hardware that can help us identify and block excessive bandwidth use, ideally in real time? Perhaps some bit of hardware or open-source software we can put in front of our web servers?

We are mostly a windows shop but we have some linux skills as well; we're also open to buying hardware if the PIX 515E isn't sufficient. What would you recommend?

© Server Fault or respective owner

Related posts about networking

Networking stopped working on Ubuntu

as seen on Super User - Search for 'Super User'
I installed Ubuntu 10.04 through the Wubi installer (Funny, I installed it today and thought I would have gotten 10.10). I had a network connection and everything was working fine. I rebooted my coumputer a couple of times and then suddenly, I could not connect to the network and when I click the… >>> More
Troubles with start up defenition of networking service in Ubuntu

as seen on Super User - Search for 'Super User'
I use Ubuntu 12.04.1. I put attention that networking script starting in runlevel 0: user@comp:/etc/rc0.d$ chkconfig -l networking networking 0:on 1:off 2:off 3:off 4:off 5:off 6:off When I try to move it working to appropriate run levels I get error: user@comp:/etc/rc0… >>> More
Installing the Elgg Social Networking CMS

as seen on Internet.com - Search for 'Internet.com'
One Open Source CMS that stands out above the rest for small businesses and personal networks is the Elgg open source "social engine," which can be used to create social applications. Scott Clark shows you how to install and get started with this social networking CMS. >>> More
OAuth: Token-Based Authorization for the Social Networking Age

as seen on Internet.com - Search for 'Internet.com'
The OAuth token-based authorization system allows users to grant third-party access to their resources without sharing their usernames and passwords. It's perfect for the age of data scattered across many social networking sites. >>> More
OAuth: Token-Based Authorization for the Social Networking Age

as seen on Internet.com - Search for 'Internet.com'
The OAuth token-based authorization system allows users to grant third-party access to their resources without sharing their usernames and passwords. It's perfect for the age of data scattered across many social networking sites. >>> More

Related posts about http

the size of apt-get update lists is too big

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I ran a clean install to Ubuntu 12.04 and so far everything has been working well. I especially commend the Ubuntu team for this release. I only noticed that the size of repository update is now about ~13MB. Normally, it is about this size for the first time you run apt-get update after a clean install… >>> More
Cannot update, apt-get cannot fetch index files

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I have a fresh install of Ubuntu 11.10 from the iso 'ubuntu-11.10-desktop-amd64.iso'. I installed this in VMWare Fusion 4.1.1 running on OSX 10.7.3. When setting up the VM, I allowed easy install to take care of creating my user and installing VMWare tools. No problems during installation, everything… >>> More
Quantal: Broken apt-index, cant fix dependencies

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I can't seem to add/remove/update packages Ubuntu software update has a notice about partial upgrades but fails Seems to be similar to this problem $ sudo apt-get update Ign http://archive.ubuntu.com quantal InRelease Ign http://security.ubuntu.com precise-security InRelease Ign… >>> More
12.04: Apt-Get Update: failure to fetch; can't connect to any sources

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I realize there are dozens of "apt-get update: failure to fetch" questions (I read through all I could find), but my present circumstance is unique to 12.04 and it affects all sources; not just launchpad. Additionally, I've tried several different servers in Europe and the U.S. as well as the "main… >>> More
How can I remove the Translation entries in apt?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
This is the output of aptitude update: Ign http://archive.canonical.com natty InRelease Ign http://extras.ubuntu.com natty InRelease Ign http://dl.google.com stable InRelease Ign http://security.ubuntu.com natty-security InRelease Hit http://deb.torproject.org natty InRelease Get:1 http://dl.google… >>> More