I've been chasing a packet-loss and network stability issue for a handful of end-users on an internal network for the past few days... These issues surfaced last week, however the location was struck by lightning six weeks ago.
I was seeing 5-10% packet loss between a stack of four Cisco 2960's and several PC's and phones on the other side of a 77-meter run. The PC's were run inline with the phones over a trunked link (switchport configuration pastebin). We were seeing dropped calls and interruptions in client-server applications and Microsoft Exchange connectivity.
I tried the usual troubleshooting steps remotely, having a local technician do the following during breaks in user and production activity:
change cables between the wall jack and device.
change patch cables between the patch panel and switch port(s).
try different switch ports within the 2960 stack.
change end-user devices with known-good equipment (new phones, different PC's).
clear switch port interface counters and monitor incrementing errors closely. (Pastebin output of sh int)
Pored over the device logs and Observium RRD graphs. No link up/down issues from the switch side.
change power strips on the end-user side.
test cable runs from the Cisco 2960 using test cable-diagnostics tdr int Gi4/0/9 (clean)*
test cable runs with a Tripp-Lite cable tester. (clean)
run diagnostics on the switch stack members. (clean)
In the end, it took three changes of switch ports to find a stable solution. The only logical conclusion is that a few Cisco 2960 switch ports are bad or flaky... Not dead, but not consistent in behavior either. I'm not used to seeing individual ports die in this manner.
What else can I test or check to determine if these devices are bad?
Is it common for single ports to have problems, rather than a contiguous bank of ports?
BTW - show cable-diagnostics tdr int Gi4/0/14 is very cool...
Interface Speed Local pair Pair length Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi4/0/14 1000M Pair A 79 +/- 0 meters Pair B Normal
Pair B 75 +/- 0 meters Pair A Normal
Pair C 77 +/- 0 meters Pair D Normal
Pair D 79 +/- 0 meters Pair C Normal
