Xen PV packet loss
- by Delphinator
I'm having some serious issues with packetloss with one of my servers. This server is a somewhat old (P4-era) machine running Debian Squeeze and Xen 4.0. There are two domUs running on it (both also Debian Squeeze), one gateway and a fileserver. Unfortunatly the processor has no virtualization extensions, therefore only PV can be used.
While investigating why our network seems to be slower than it should I found some pretty bad packet loss (~25%). After further investigation and several experiments I did a measurment between the dom0 and one of the domUs:
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 110 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to dom0, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 110 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.2(domU) port 33817 connected with 192.168.1.100(dom0) port 5001
[ 4] local 192.168.1.2(domU) port 5001 connected with 192.168.1.100(dom0) port 48606
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 46.3 MBytes 38.7 Mbits/sec
[ 3] Sent 33020 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 46.2 MBytes 38.6 Mbits/sec 0.030 ms 89/33019 (0.27%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
[ 4] 0.0-10.2 sec 43.0 MBytes 35.3 Mbits/sec 13.074 ms 11575/42256 (27%)
tl;dr: 27% packet loss from dom0 to domU with 50Mbit UDP packets.
Same thing happens from anywhere in the network. The problem gets better for smaller bandwidths (0.047% for 5Mbit) and worse for higher (59% for 200Mbit) ones.
I did increase the CPU-weight of the dom0, there is no swapping going on, and actual networking-hardware is not involved. I never expected Xen (or anything related) to drop packets, and I'm completly clueless what to try next.
