Improving TCP performance over a gigabit network lots of connections and high traffic for storage and streaming services
- by Linux Guy
I have two servers, Both servers hardware Specification are
Processor : Dual Processor
RAM : over 128 G.B
Hard disk : SSD Hard disk
Outging Traffic bandwidth : 3 Gbps
network cards speed : 10 Gbps
Server A : for Encoding videos
Server B : for storage videos andstream videos over web interface like youtube
The inbound bandwidth between two servers is 10Gbps , the outbound bandwidth internet bandwidth is 500Mpbs
Both servers using public ip addresses in public and private network
Both servers transfer and connection on nginx port , and the server B used for streaming media , like youtube stream videos
Both servers in same network , when i do ping from Server A to Server B i got high time latency above 1.0ms , the time range time=52.7 ms to time=215.7 ms -
This is the output of iftop utility
353Mb 707Mb 1.04Gb 1.38Gb 1.73Gb
mqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqq
server.example.com => ip.address 6.36Mb 4.31Mb 1.66Mb
<= 158Kb 94.8Kb 35.1Kb
server.example.com => ip.address 1.23Mb 4.28Mb 1.12Mb
<= 17.1Kb 83.5Kb 21.9Kb
server.example.com => ip.address 395Kb 3.89Mb 1.07Mb
<= 6.09Kb 109Kb 28.6Kb
server.example.com => ip.address 4.55Mb 3.83Mb 1.04Mb
<= 55.6Kb 45.4Kb 13.0Kb
server.example.com => ip.address 649Kb 3.38Mb 1.47Mb
<= 9.00Kb 38.7Kb 16.7Kb
server.example.com => ip.address 5.00Mb 3.32Mb 1.80Mb
<= 65.7Kb 55.1Kb 29.4Kb
server.example.com => ip.address 387Kb 3.13Mb 1.06Mb
<= 18.4Kb 39.9Kb 15.0Kb
server.example.com => ip.address 3.27Mb 3.11Mb 1.01Mb
<= 81.2Kb 64.5Kb 20.9Kb
server.example.com => ip.address 1.75Mb 3.08Mb 2.72Mb
<= 16.6Kb 35.6Kb 32.5Kb
server.example.com => ip.address 1.75Mb 2.90Mb 2.79Mb
<= 22.4Kb 32.6Kb 35.6Kb
server.example.com => ip.address 3.03Mb 2.78Mb 1.82Mb
<= 26.6Kb 27.4Kb 20.2Kb
server.example.com => ip.address 2.26Mb 2.66Mb 1.36Mb
<= 51.7Kb 49.1Kb 24.4Kb
server.example.com => ip.address 586Kb 2.50Mb 1.03Mb
<= 4.17Kb 26.1Kb 10.7Kb
server.example.com => ip.address 2.42Mb 2.49Mb 2.44Mb
<= 31.6Kb 29.7Kb 29.9Kb
server.example.com => ip.address 2.41Mb 2.46Mb 2.41Mb
<= 26.4Kb 24.5Kb 23.8Kb
server.example.com => ip.address 2.37Mb 2.39Mb 2.40Mb
<= 28.9Kb 27.0Kb 28.5Kb
server.example.com => ip.address 525Kb 2.20Mb 1.05Mb
<= 7.03Kb 26.0Kb 12.8Kb
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
TX: cum: 102GB peak: 1.65Gb rates: 1.46Gb 1.44Gb 1.48Gb
RX: 1.31GB 24.3Mb 19.5Mb 18.9Mb 20.0Mb
TOTAL: 103GB 1.67Gb 1.48Gb 1.46Gb 1.50Gb
I check the transfer speed using iperf utility From Server A to Server B
# iperf -c 0.0.0.2 -p 8777
------------------------------------------------------------
Client connecting to 0.0.0.2, TCP port 8777
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 3] local 0.0.0.1 port 38895 connected with 0.0.0.2 port 8777
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.8 sec 528 KBytes 399 Kbits/sec
My Current Connections in Server B
# netstat -an|grep ":8777"|awk '/tcp/ {print $6}'|sort -nr| uniq -c
2072 TIME_WAIT
28 SYN_RECV
1 LISTEN
189 LAST_ACK
139 FIN_WAIT2
373 FIN_WAIT1
3381 ESTABLISHED
34 CLOSING
Server A Network Card Information
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 10000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: external
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
Server B Network Card Information
Settings for eth2:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: No
Advertised link modes: 10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: 10000Mb/s
Duplex: Full
Port: Direct Attach Copper
PHYAD: 0
Transceiver: external
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
ifconfig server A
eth0 Link encap:Ethernet HWaddr 00:25:90:ED:9E:AA
inet addr:0.0.0.1 Bcast:0.0.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1202795665 errors:0 dropped:64334 overruns:0 frame:0
TX packets:2313161968 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:893413096188 (832.0 GiB) TX bytes:3360949570454 (3.0 TiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:2207544 errors:0 dropped:0 overruns:0 frame:0
TX packets:2207544 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:247769175 (236.2 MiB) TX bytes:247769175 (236.2 MiB)
ifconfig Server B
eth2 Link encap:Ethernet HWaddr 00:25:90:82:C4:FE
inet addr:0.0.0.2 Bcast:0.0.0.2 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:39973046980 errors:0 dropped:1828387600 overruns:0 frame:0
TX packets:69618752480 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3013976063688 (2.7 TiB) TX bytes:102250230803933 (92.9 TiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:1049495 errors:0 dropped:0 overruns:0 frame:0
TX packets:1049495 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:129012422 (123.0 MiB) TX bytes:129012422 (123.0 MiB)
Netstat -i on Server B
# netstat -i
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth2 9000 0 42098629968 0 2131223717 0 73698797854 0 0 0 BMRU
lo 65536 0 1077908 0 0 0 1077908 0 0 0 LRU
I Turn up send/receive buffers on the network card to 2048 and problem still persist
I increase the MTU for server A and problem still persist and i increase the MTU for server B for better connectivity and transfer speed but it couldn't transfer at all
The problem is : as you can see from iperf utility, the transfer speed from server A to server B slow
when i restart network service in server B the transfer in server A at full speed, after 2 minutes , it's getting slow
How could i troubleshoot slow speed issue and fix it in server B ?
Notice : if there any other commands i should execute in servers for more information, so it might help resolve the problem , let me know in comments
