Millions of SYN_RECV connections, no DDoS

Posted by ThomK on Server Fault See other posts from Server Fault or by ThomK
Published on 2011-01-27T08:33:08Z Indexed on 2011/01/28 23:28 UTC
Read the original article Hit count: 396

Filed under:
|
|
|
|

We have such server structure: reverse proxy (nginx) -> worker (uwsgi) -> postgresql / memcached. All servers are in local network behind router, with NATed external ip:ports (http/s 80/443 to proxy, and ssh 22 to all servers).

Problem is, that sometimes proxy server netstat reports MILLIONS of SYN_RECV connections. From same IP / same ports. Like that:

nginx ~ # netstat -n | grep 83.238.153.195
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
[...]

And this is not DDoS, because all IPs affected belongs to our website users. On side note, users says that it's not affecting them. Website is online and working, but... that particular one (from example above) told me that website is down and Firefox can't connect. I've done tcpdump.

19:42:14.826011 IP 83.238.153.195.zephyr-srv > 192.168.1.1.http: Flags [S], seq 1845850583, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0
19:42:14.826042 IP 192.168.1.1.http > 83.238.153.195.zephyr-srv: Flags [S.], seq 2835837547, ack 1845850584, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:42:17.887331 IP 83.238.153.195.zephyr-srv > 192.168.1.1.http: Flags [S], seq 1845850583, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0
19:42:17.887343 IP 192.168.1.1.http > 83.238.153.195.zephyr-srv: Flags [S.], seq 2835837547, ack 1845850584, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:42:19.065497 IP 192.168.1.1.http > 83.238.153.195.zephyr-srv: Flags [S.], seq 2835837547, ack 1845850584, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:42:23.918064 IP 83.238.153.195.zephyr-srv > 192.168.1.1.http: Flags [S], seq 1845850583, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0
19:42:23.918076 IP 192.168.1.1.http > 83.238.153.195.zephyr-srv: Flags [S.], seq 2835837547, ack 1845850584, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:42:25.265499 IP 192.168.1.1.http > 83.238.153.195.zephyr-srv: Flags [S.], seq 2835837547, ack 1845850584, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:42:37.265501 IP 192.168.1.1.http > 83.238.153.195.zephyr-srv: Flags [S.], seq 2835837547, ack 1845850584, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:42:37.758051 IP 83.238.153.195.2107 > 192.168.1.1.http: Flags [S], seq 564208067, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0  
19:42:37.758069 IP 192.168.1.1.http > 83.238.153.195.2107: Flags [S.], seq 3188568660, ack 564208068, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:42:40.714360 IP 83.238.153.195.2107 > 192.168.1.1.http: Flags [S], seq 564208067, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0  
19:42:40.714374 IP 192.168.1.1.http > 83.238.153.195.2107: Flags [S.], seq 3188568660, ack 564208068, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:42:41.665503 IP 192.168.1.1.http > 83.238.153.195.2107: Flags [S.], seq 3188568660, ack 564208068, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:42:46.751073 IP 83.238.153.195.2107 > 192.168.1.1.http: Flags [S], seq 564208067, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0  
19:42:46.751087 IP 192.168.1.1.http > 83.238.153.195.2107: Flags [S.], seq 3188568660, ack 564208068, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:42:47.665498 IP 192.168.1.1.http > 83.238.153.195.2107: Flags [S.], seq 3188568660, ack 564208068, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:42:59.865499 IP 192.168.1.1.http > 83.238.153.195.2107: Flags [S.], seq 3188568660, ack 564208068, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:43:01.265500 IP 192.168.1.1.http > 83.238.153.195.zephyr-srv: Flags [S.], seq 2835837547, ack 1845850584, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:43:13.320382 IP 83.238.153.195.2114 > 192.168.1.1.http: Flags [S], seq 2136055006, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0  
19:43:13.320399 IP 192.168.1.1.http > 83.238.153.195.2114: Flags [S.], seq 3754336171, ack 2136055007, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:43:16.320556 IP 83.238.153.195.2114 > 192.168.1.1.http: Flags [S], seq 2136055006, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0  
19:43:16.320569 IP 192.168.1.1.http > 83.238.153.195.2114: Flags [S.], seq 3754336171, ack 2136055007, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:43:17.665498 IP 192.168.1.1.http > 83.238.153.195.2114: Flags [S.], seq 3754336171, ack 2136055007, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:43:22.250069 IP 83.238.153.195.2114 > 192.168.1.1.http: Flags [S], seq 2136055006, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0  
19:43:22.250080 IP 192.168.1.1.http > 83.238.153.195.2114: Flags [S.], seq 3754336171, ack 2136055007, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:43:23.665500 IP 192.168.1.1.http > 83.238.153.195.2114: Flags [S.], seq 3754336171, ack 2136055007, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:43:23.865501 IP 192.168.1.1.http > 83.238.153.195.2107: Flags [S.], seq 3188568660, ack 564208068, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:43:35.665498 IP 192.168.1.1.http > 83.238.153.195.2114: Flags [S.], seq 3754336171, ack 2136055007, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:43:37.903038 IP 83.238.153.195.2213 > 192.168.1.1.http: Flags [S], seq 2918118729, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0  
19:43:37.903054 IP 192.168.1.1.http > 83.238.153.195.2213: Flags [S.], seq 4145523337, ack 2918118730, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:43:40.772899 IP 83.238.153.195.2213 > 192.168.1.1.http: Flags [S], seq 2918118729, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0  
19:43:40.772912 IP 192.168.1.1.http > 83.238.153.195.2213: Flags [S.], seq 4145523337, ack 2918118730, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:43:41.865500 IP 192.168.1.1.http > 83.238.153.195.2213: Flags [S.], seq 4145523337, ack 2918118730, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0   
19:43:46.793057 IP 83.238.153.195.2213 > 192.168.1.1.http: Flags [S], seq 2918118729, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0  
19:43:46.793069 IP 192.168.1.1.http > 83.238.153.195.2213: Flags [S.], seq 4145523337, ack 2918118730, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:43:47.865500 IP 192.168.1.1.http > 83.238.153.195.2213: Flags [S.], seq 4145523337, ack 2918118730, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  
19:43:49.465503 IP 192.168.1.1.http > 83.238.153.195.zephyr-srv: Flags [S.], seq 2835837547, ack 1845850584, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0  

Anyone have some thoughts on that?

© Server Fault or respective owner

Related posts about linux

Related posts about networking