Proxy / Squid 2.7 / Debian Wheezy 6.7 / lots of TCP Timed-out
- by Maroon Ibrahim
i'm facing a lot of TCP timed-out on a busy cache server and here below my sysctl.conf configuration as well as an output of "netstat -st"
Kernel 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64 GNU/Linux
Any advice or help would be highly appreciated
#################### Sysctl.conf
cat /etc/sysctl.conf
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
fs.file-max = 65536
net.ipv4.tcp_low_latency = 1
net.core.wmem_max = 8388608
net.core.rmem_max = 8388608
net.ipv4.ip_local_port_range = 1024 65000
fs.aio-max-nr = 131072
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 3
kernel.threads-max = 131072
kernel.msgmax = 32768
kernel.msgmni = 64
kernel.msgmnb = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.ip_forward = 1
net.ipv4.tcp_timestamps = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.ip_dynaddr = 1
vm.swappiness = 0
vm.drop_caches = 3
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_ecn = 0
net.ipv4.tcp_max_orphans = 131072
net.ipv4.tcp_orphan_retries = 1
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_max_syn_backlog = 32768
net.core.netdev_max_backlog = 131072
net.ipv4.tcp_mem = 6085248 16227328 67108864
net.ipv4.tcp_wmem = 4096 131072 33554432
net.ipv4.tcp_rmem = 4096 174760 33554432
net.core.rmem_default = 33554432
net.core.rmem_max = 33554432
net.core.wmem_default = 33554432
net.core.wmem_max = 33554432
net.core.somaxconn = 10000
#
################ Netstat results
/# netstat -st
IcmpMsg:
InType0: 2
InType3: 233754
InType8: 56251
InType11: 23192
OutType0: 56251
OutType3: 437
OutType8: 4
Tcp:
20680741 active connections openings
63642431 passive connection openings
1126690 failed connection attempts
2093143 connection resets received
13059 connections established
2649651696 segments received
2195445642 segments send out
183401499 segments retransmited
38299 bad segments received.
14648899 resets sent
UdpLite:
TcpExt:
507 SYN cookies sent
178 SYN cookies received
1376771 invalid SYN cookies received
1014577 resets received for embryonic SYN_RECV sockets
4530970 packets pruned from receive queue because of socket buffer overrun
7233 packets pruned from receive queue
688 packets dropped from out-of-order queue because of socket buffer overrun
12445 ICMP packets dropped because they were out-of-window
446 ICMP packets dropped because socket was locked
33812202 TCP sockets finished time wait in fast timer
622 TCP sockets finished time wait in slow timer
573656 packets rejects in established connections because of timestamp
133357718 delayed acks sent
23593 delayed acks further delayed because of locked socket
Quick ack mode was activated 21288857 times
839 times the listen queue of a socket overflowed
839 SYNs to LISTEN sockets dropped
41 packets directly queued to recvmsg prequeue.
79166 bytes directly in process context from backlog
24 bytes directly received in process context from prequeue
2713742130 packet headers predicted
84 packets header predicted and directly queued to user
1925423249 acknowledgments not containing data payload received
877898013 predicted acknowledgments
16449673 times recovered from packet loss due to fast retransmit
17687820 times recovered from packet loss by selective acknowledgements
5047 bad SACK blocks received
Detected reordering 11 times using FACK
Detected reordering 1778091 times using SACK
Detected reordering 97955 times using reno fast retransmit
Detected reordering 280414 times using time stamp
839369 congestion windows fully recovered without slow start
4173098 congestion windows partially recovered using Hoe heuristic
305254 congestion windows recovered without slow start by DSACK
933682 congestion windows recovered without slow start after partial ack
77828 TCP data loss events
TCPLostRetransmit: 5066
2618430 timeouts after reno fast retransmit
2927294 timeouts after SACK recovery
3059394 timeouts in loss state
75953830 fast retransmits
11929429 forward retransmits
51963833 retransmits in slow start
19418337 other TCP timeouts
2330398 classic Reno fast retransmits failed
2177787 SACK retransmits failed
742371590 packets collapsed in receive queue due to low socket buffer
13595689 DSACKs sent for old packets
50523 DSACKs sent for out of order packets
4658236 DSACKs received
175441 DSACKs for out of order packets received
880664 connections reset due to unexpected data
346356 connections reset due to early user close
2364841 connections aborted due to timeout
TCPSACKDiscard: 1590
TCPDSACKIgnoredOld: 241849
TCPDSACKIgnoredNoUndo: 1636687
TCPSpuriousRTOs: 766073
TCPSackShifted: 74562088
TCPSackMerged: 169015212
TCPSackShiftFallback: 78391303
TCPBacklogDrop: 29
TCPReqQFullDoCookies: 507
TCPChallengeACK: 424921
TCPSYNChallenge: 170388
IpExt:
InBcastPkts: 351510
InOctets: -609466797
OutOctets: -1057794685
InBcastOctets: 75631402
#
