I am trying to configure HAProxy to do load balancing for a custom webserver I created. Right now I am noticing an increasing delay with HAProxy as the size of the return message increases. For example, I ran four different tests, here are the results:

Response 15kb through HAProxy:
Avg. response time: .34 secs
Transacation rate: 763 trans/sec
Throughput: 11.08 MB/sec

Response 2kb through HAProxy:
Avg. response time: .08 secs
Transaction rate: 1171 trans / sec
Throughput: 2.51 MB/sec

Response 15kb directly to server:
Avg. response time: .11 sec
Transaction rate: 1046 trans/sec
throughput: 15.20 MB/sec

Response 2kb directly to server:
Avg. Response time: .05 secs
Transaction rate: 1158 trans/sec
Throughput: 2.48 MB/sec

All transactions are HTTP requests. As you can see, there seems to be a much bigger difference between response times for when the response is bigger, than when it is smaller. I understand there will be a slight delay when using HAProxy. Not sure if it matters, but the test itself was run using siege. And during the test there was only one server behind the HAProxy(the same that was used in the direct to server tests). Here is my haproxy.config file:

global  
       log 127.0.0.1   local0
       log 127.0.0.1   local1 notice
       maxconn 10000
       user haproxy
       group haproxy
       daemon
       #debug

defaults
       log     global
       mode    http
       option  httplog
       option  dontlognull
       retries 3
       option redispatch
       option httpclose
       maxconn 10000
       contimeout      10000
       clitimeout      50000
       srvtimeout      50000
       balance roundrobin
       stats enable
       stats uri /stats

listen  lb1  10.1.10.26:80
        maxconn 10000
        server  app1 10.1.10.200:8080 maxconn 5000

I couldn't find much in terms of options in this file that would help my problem. I have heard suggestions that I may have to adjust a few of my sysctl settings. I could not find a lot of information on this however, most documentation is for Linux 2.4 and 2.6 on the sysctl stuff, I am running 3.2(Ubuntu server 12.04), which seems to auto tuning, so I have no clue what I should or shouldn't be changing. Most settings changes I tried had no effect or a negative effect on performance.

Just a notice, this is a very preliminary test, and my hope is that at deployment time, my HAProxy will be able to balance 10k-20k requests/sec to many servers, so if anyone could provide information to help me reach that goal, it would be much appreciated.

Thank you very much for any information you can provide. And if you need anymore information from me please let me know, I will get you anything I can.