The scenario:
DC running Windows Server 2008 R2 providing DNS + DHCP
Cisco 1811 Router as the gateway
30 Windows XP DHCP clients on the LAN
The problem:
Some workstations are spontaneously switching to an incorrect DNS server. Specifically, ipconfig /all shows that they start using the gateway as a DNS server.
This happens about 5-10 times a day to various computers, sometimes more than once per day.
The workaround:
Repairing the connection on the XP client always fixes the problem, and the correct DNS server address is obtained.
We lost our main DNS/DHCP machine a week ago, and had to bring this one online as a spare. We've been having this issue since then. DHCP leases on the old and new servers are configured for "wired" (8 day) duration. There are definitely no other DHCP servers active on the LAN. So far there is no discernible pattern about which clients will show this problem, or when.
When I ran DCDIAG /test:DNS it came back clean. Manual inspection of the DNS zone shows that all the records are appearing as expected, with no traces of the previous machine in there.
Update Feb 27: Added screenshots.
Here is a screenshot of the DHCP scope options on the 2008 R2 server.
And here is a screenshot of ipconfig /all running on a healthy host. I don't have any ailing hosts at the moment, but will grab a screencap next time it happens.
Update Feb 28: More screenshots.
Here's a screenshot of DHCP and DNS traffic from a healthy client when repairing the local area connection. There's definitely only one server responding, but it does seem strange that the negotiation takes place twice. I'll try to get a similar capture from a sick machine this coming week.
Update Mar 01: Caught a bad ipconfig.
Here's a screenshot of ipconfig /all from a client that had this issue. It says the lease was issued this morning, but it doesn't even have an entry for the secondary DNS I set up yesterday. Both DNS servers were discovered properly when repairing the connection.
Update Mar 01: It even got the sysadmin!
This issue finally affected my personal workstation this morning. Unfortunately I had just rebooted and wasn't running a packet dump at the time. I set up a secondary server yesterday, and was logging all DNS traffic to it. My machine had not contacted the secondary DNS in over half an hour, so that says to me that it's just spontaneously reverting to the gateway without even failing over to secondary DNS first.
Today I swapped the order of the DNS servers in DHCP, so the secondary is primary and vice versa. I will update again once I know how that goes.
