TL;DR version: Turns out this was a Windows Server 2008 R2 kernel networking bug. After siccing Microsoft support on it, we (eventually) got an unpublished kernel hotfix from Microsoft to address it. If you, too, are experiencing mysterious low-level network driver failures requiring a reboot/bluescreen cycle, you might want that hotfix (or maybe Service Pack 1 whenever it is released, too.)
We have been using HAProxy along with heartbeat from the Linux-HA project. We are using two linux instances to provide a failover. Each server has with their own public IP and a single IP which is shared between the two using a virtual interface (eth1:1) at IP: 69.59.196.211
The virtual interface (eth1:1) IP 69.59.196.211 is configured as the gateway for the windows servers behind them and we use ip_forwarding to route traffic.
We are experiencing an occasional network outage on one of our windows servers behind our linux gateways. HAProxy will detect the server is offline which we can verify by remoting to the failed server and attempting to ping the gateway:
Pinging 69.59.196.211 with 32 bytes of data:
Reply from 69.59.196.220: Destination host unreachable.
Running arp -a on this failed server shows that there is no entry for the gateway address (69.59.196.211):
Interface: 69.59.196.220 --- 0xa
Internet Address Physical Address Type
69.59.196.161 00-26-88-63-c7-80 dynamic
69.59.196.210 00-15-5d-0a-3e-0e dynamic
69.59.196.212 00-21-5e-4d-45-c9 dynamic
69.59.196.213 00-15-5d-00-b2-0d dynamic
69.59.196.215 00-21-5e-4d-61-1a dynamic
69.59.196.217 00-21-5e-4d-2c-e8 dynamic
69.59.196.219 00-21-5e-4d-38-e5 dynamic
69.59.196.221 00-15-5d-00-b2-0d dynamic
69.59.196.222 00-15-5d-0a-3e-09 dynamic
69.59.196.223 ff-ff-ff-ff-ff-ff static
224.0.0.22 01-00-5e-00-00-16 static
224.0.0.252 01-00-5e-00-00-fc static
225.0.0.1 01-00-5e-00-00-01 static
On our linux gateway instances arp -a shows:
peak-colo-196-220.peak.org (69.59.196.220) at <incomplete> on eth1
stackoverflow.com (69.59.196.212) at 00:21:5e:4d:45:c9 [ether] on eth1
peak-colo-196-215.peak.org (69.59.196.215) at 00:21:5e:4d:61:1a [ether] on eth1
peak-colo-196-219.peak.org (69.59.196.219) at 00:21:5e:4d:38:e5 [ether] on eth1
peak-colo-196-222.peak.org (69.59.196.222) at 00:15:5d:0a:3e:09 [ether] on eth1
peak-colo-196-209.peak.org (69.59.196.209) at 00:26:88:63:c7:80 [ether] on eth1
peak-colo-196-217.peak.org (69.59.196.217) at 00:21:5e:4d:2c:e8 [ether] on eth1
Why would arp occasionally set the entry for this failed server as <incomplete>? Should we be defining our arp entries statically? I've always left arp alone since it works 99% of the time, but in this one instance it appears to be failing. Are there any additional troubleshooting steps we can take help resolve this issue?
THINGS WE HAVE TRIED
I added a static arp entry for testing on one of the linux gateways which still didn't help.
root@haproxy2:~# arp -a
peak-colo-196-215.peak.org (69.59.196.215) at 00:21:5e:4d:61:1a [ether] on eth1
peak-colo-196-221.peak.org (69.59.196.221) at 00:15:5d:00:b2:0d [ether] on eth1
stackoverflow.com (69.59.196.212) at 00:21:5e:4d:45:c9 [ether] on eth1
peak-colo-196-219.peak.org (69.59.196.219) at 00:21:5e:4d:38:e5 [ether] on eth1
peak-colo-196-209.peak.org (69.59.196.209) at 00:26:88:63:c7:80 [ether] on eth1
peak-colo-196-217.peak.org (69.59.196.217) at 00:21:5e:4d:2c:e8 [ether] on eth1
peak-colo-196-220.peak.org (69.59.196.220) at 00:21:5e:4d:30:8d [ether] PERM on eth1
root@haproxy2:~# arp -i eth1 -s 69.59.196.220 00:21:5e:4d:30:8d
root@haproxy2:~# ping 69.59.196.220
PING 69.59.196.220 (69.59.196.220) 56(84) bytes of data.
--- 69.59.196.220 ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6006ms
Rebooting the windows web server solves this issue temporarily with no other changes to the network but our experience shows this issue will come back.
Swapping network cards and switches
I noticed the link light on the port of the switch for the failed windows server was running at 100Mb instead of 1Gb on the failed interface. I moved the cable to several other open ports and the link indicated 100Mb for each port that I tried. I also swapped the cable with the same result. I tried changing the properties of the network card in windows and the server locked up and required a hard reset after clicking apply. This windows server has two physical network interfaces so I have swapped the cables and network settings on the two interfaces to see if the problem follows the interface. If the public interface goes down again we will know that it is not an issue with the network card.
(We also tried another switch we have on hand, no change)
Changing network hardware driver versions
We've had the same problem with the latest Broadcom driver, as well as the built-in driver that ships in Windows Server 2008 R2.
Replacing network cables
As a last ditch effort we remembered another change that occurred was the replacement of all of the patch cords between our servers / switch. We had purchased two sets, one green of lengths 1ft - 3ft for the private interfaces and another set of red cables for the public interfaces. We swapped out all of the public interface patch cables with a different brand and ran our servers without issue for a full week ... aaaaaand then the problem recurred.
Disable checksum offload, remove TProxy
We also tried disabling TCP/IP checksum offload in the driver, no change. We're now pulling out TProxy and moving to a more traditional x-forwarded-for network arrangement without any fancy IP address rewriting. We'll see if that helps.
Switch Virtualization providers
On the off chance this was related to Hyper-V in some way (we do host Linux VMs on it), we switched to VMWare Server. No change.
Switch host model
We've reached the end of our troubleshooting rope and are now formally involving Microsoft support. They recommended changing the host model:
http://en.wikipedia.org/wiki/Host_model
http://technet.microsoft.com/en-us/magazine/2007.09.cableguy.aspx
We did that, and.. we'll see.
