Normalize Accept-Encoding via HAProxy for optimized Squid hit rate

Posted by Matt Beckman on Server Fault See other posts from Server Fault or by Matt Beckman
Published on 2010-04-05T19:44:33Z Indexed on 2010/04/05 19:53 UTC
Read the original article Hit count: 279

Filed under:

squid

|

haproxy

|

gzip

Our website infrastructure uses HAProxy for load balancing, a Squid cluster for caching, and application data is on an IIS cluster.

We load balance HAProxy by URI to optimize the Squid hit-rate, but we know that Squid is holding different copies of each page based on the Accept-Encoding header passed to it by the browser, and so IE (gzip, deflate) will have a different copy of a cached page than Firefox (gzip,deflate) or Chrome (gzip,deflate,sdch).

We want to normalize the Accept-Encoding headers and I think the best place to do so would be in HAProxy. I'd appreciate it if someone could offer some ideas on how to accomplish this without breaking support for clients without gzip or deflate support.

© Server Fault or respective owner

Related posts about squid

Squid external_acl_type Cannot run process

as seen on Server Fault - Search for 'Server Fault'
I want to restrict uploading for group of the users via squid. So I've choosen to use external_acl_type but after reload of the squid it returns error. WARNING: Cannot run '/usr/local/etc/squid/lists/newupload.sh' process. Permissions of newupload.sh and squid are the same. newupload.sh is executive… >>> More
SquidProxy status:Unknow job :Squid

as seen on Super User - Search for 'Super User'
I have an issue in Centos 6.3. I installed Squid from compressed source archives(.tar.gz), I create an /etc/init.d/squid file, that run well because I view with the command ps -aux, that is running: root 2153 0.0 0.0 8920 1536 ? Ss 13:52 0:00 /opt/squid/squid-3.2/sbin/squid squid 2155 0.0 0… >>> More
squid ssl bump sslv3 enforce to allow old sites

as seen on Server Fault - Search for 'Server Fault'
Important: I have this question on stackoverflow but somebody told me this is more relevant place for this question. Thanks I have configured squid(3.4.2) as ssl bumped proxy. I am setting proxy in firefox(29) to use squid for https/http. Now it works for most sites, but some sites which support… >>> More
Accessing Squid Proxy over internet

as seen on Server Fault - Search for 'Server Fault'
Hi, I recently finished installing Squid on a VPS I have in the US and its working fine locally (I verified by setting http_proxy variable and using lynx). I want to access this proxy over the internet (as an anonymizer) so that I can see how some ads show up for US traffic on my website. I have… >>> More
Accessing Squid Proxy over internet

as seen on Server Fault - Search for 'Server Fault'
Hi, I recently finished installing Squid on a VPS I have in the US and its working fine locally (I verified by setting http_proxy variable and using lynx). I want to access this proxy over the internet (as an anonymizer) so that I can see how some ads show up for US traffic on my website. I have… >>> More

Related posts about haproxy

HA Proxy Stick-table and tcp-connection configuration

as seen on Server Fault - Search for 'Server Fault'
I am using HA Proxy HA-Proxy version 1.4.18 2011/09/16 I am trying to insert the following into /etc/init.d/haproxy.cfg file # Use General Purpose Couter (gpc) 0 in SC1 as a global abuse counter # Monitors the number of request sent by an IP over a period of 10 seconds stick-table type ip size 1m… >>> More
Rotate haproxy logs

as seen on Server Fault - Search for 'Server Fault'
I tried few things but still not able to rotate haproxy logs efficiently. I need to rotate logs when log files crosses 500 MB size. Considering haproxy is serving large no. of static tcp connections, I can not restart haproxy process though a reload is doable. Daily haproxy log file size normally… >>> More
could not bind socket while haproxy restart

as seen on Server Fault - Search for 'Server Fault'
I m restarting HAproxy by following command haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) but i get following message [ALERT] 183/225022 (9278) : Starting proxy appli1-rewrite: cannot bind socket [ALERT] 183/225022 (9278) : Starting proxy appli2-insert:… >>> More
Restarting Haproxy Gracefully

as seen on Server Fault - Search for 'Server Fault'
As per various blogs, HAproxy can be gracefully restarted using the following command: sudo haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) TO verify this, I had set up a apache bench script which contiguously sent message to haproxy. Ideally, whenever… >>> More
HAProxy: Display a "BADREQ" | BADREQ's by the thousands

as seen on Server Fault - Search for 'Server Fault'
My HAProxy Configuration. #HA-Proxy version 1.3.22 2009/10/14 Copyright 2000-2009 Willy Tarreau <[email protected]> global maxconn 10000 spread-checks 50 user haproxy group haproxy daemon stats socket /tmp/haproxy log localhost local0 log localhost local1 notice defaults … >>> More