How can I set up a load balancer to direct all Search Engine Bot traffic to one server?

Posted by Ryan on Server Fault See other posts from Server Fault or by Ryan
Published on 2012-11-13T15:41:09Z Indexed on 2012/11/14 11:06 UTC
Read the original article Hit count: 239

Filed under:

Cloud

|

hosting

|

cloud-storage

|

load-balancing

We have a simple load balancer set up on Rackspace to 3 web server nodes. After reviewing our traffic and expenses, the largest bandwidth hog is Google Bot. Since on Rackspace we pay for bandwidth by the byte, we'd like to direct all traffic from GoogleBot to another host (MediaTemple) with unlimited bandwidth.

We think this would cut our hosting bill several thousand dollars a month.

Is this possible? Advisable?

© Server Fault or respective owner

Related posts about Cloud

Understanding: cloud-server, cloud-hosting, cloud-computing, the cloud

as seen on Server Fault - Search for 'Server Fault'
There's a lot of buzz about these subjects and there seems little consensus on the terms. Is that just me not understanding the subject, or is there a clear meaning for each of these terms? Are there more elaborate terms or descriptions that describe what a cloud provider has, is or offers? EDIT:… >>> More
amazon cloud vs rackspace cloud

as seen on Server Fault - Search for 'Server Fault'
Hi, I'm looking to take a dedicated server - in the process I read about Amazon Cloud computing & Rackspace Cloud Servers. Now I'm not sure which one to opt? Could somebody suggest - Performance & Price wise. Regards >>> More
BULL nous parle de Cloud Maker, offre de cloud computing s'appuyant sur l'hyperviseur de Microsoft

as seen on Developper.com - Search for 'Developper.com'
BULL nous parle de Cloud Maker, offre de cloud computing s'appuyant sur l'hyperviseur de Microsoft Lors des Tech Days 2010, organisés par Microsoft, Bull avait annoncé la signature d'un accord de coopération stratégique dans le domaine de l'interopérabilité avec la firme de Redmond. De ce fait… >>> More
Big Data – Role of Cloud Computing in Big Data – Day 11 of 21

as seen on SQL Authority - Search for 'SQL Authority'
In yesterday’s blog post we learned the importance of the NewSQL. In this article we will understand the role of Cloud in Big Data Story What is Cloud? Cloud is the biggest buzzword around from last few years. Everyone knows about the Cloud and it is extremely well defined online. In this article… >>> More
amazon cloud vs rackspace cloud [closed]

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I'm looking to take a dedicated server - in the process I read about Amazon Cloud computing & Rackspace Cloud Servers. Now I'm not sure which one to opt? Could somebody suggest - Performance & Price wise. Regards >>> More

Related posts about hosting

Suitable Hosting for Web Development Company: VPS Hosting, Cloud hosting, Shared Hosting

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Hi, We are web development company here in nepal. We are still in growth phase. Our URL: http://www.outsourcingnepal.com We have our clients demanding for web hosting and are now ready for getting hosting package to host websites of our clients. Since we have multiple clients and they want their… >>> More
Enterprise SharePoint 2010 Hosting, SharePoint Foundation 2010 Hosting, SharePoint Standard 2010 Hos

as seen on Geeks with Blogs - Search for 'Geeks with Blogs'
Enterprise SharePoint 2010 Hosting, SharePoint Foundation 2010 Hosting, SharePoint Standard 2010 Hosting, Michigan Sclera, a Microsoft Hosted Services Provider Partner, is offering key Service Offerings around the Microsoft SharePoint Server 2010 stack. Specifically – if you’re looking for SharePoint… >>> More
Hosting 2 Sites Under 1 DiscountASP.Net Hosting Account

as seen on Server Fault - Search for 'Server Fault'
I've currently got an ASP.Net site hosted with DiscountASP.Net what I would like to do now is put a second site in a subdirectory of this original site and register a new domain to point at the subfolder, so to the end user its 2 completely seperate sites. I've asked DiscountASP support if they could… >>> More
Is virtual hosting any better than shared hosting?

as seen on Stack Overflow - Search for 'Stack Overflow'
Is there any real performance gain on a virtual hosting over a shared hosting? Do you recommend any virtual hosting costing bellow $100? >>> More
moving from shared-hosting to VPS-hosting

as seen on Server Fault - Search for 'Server Fault'
Hi, I'm thinking of moving my social network from shared hosting to VPS hosting. I'm not familiar with VPS and have no experience. Will that be a problem? Thanks >>> More