Unicast traffic between hosts on a switch leaving the switch by its uplink. Why?
- by Rich Lafferty
I have a weird thing happening on our network at my office which I can't quite get my head around. In particular I can't tell if it's a problem with a switch, or a problem with configuration.
We have a Cisco SG300-52 switch (sw01) in the top of a rack in our server room, connected to another SG300-28 that acts as our core switch (core01). Both run layer 2 only, our firewalls do routing between VLANs. They have a dozen or so VLANs between them. Gi1 on sw01 is a trunk port connected to gi1 on core01.
(Disclosure: There are other switches in our environment but I'm pretty sure I've isolated the problem down to these two. Happy to provide more info if necessary.)
The behaviour I'm seeing is limited to one VLAN, vlan 12 -- or, at least, it's not happening on the other ones I checked (It's hard to guarantee the absence of packets), and it is: sw01 is forwarding, to core01, traffic which is between two hosts which are both plugged into sw01. (I noticed this because the IDS in our firewall gave a false positive on traffic which should not reach the firewall.)
We noticed this mostly between our two dhcp/dns servers, net01 (10.12.0.10) and net02 (10.12.0.11). net01 is physical hardware and net02 is on a VMware ESX server. net01 is connected to gi44 on sw01 and net02's ESX server to gi11.
[net01]----gi44-[sw01]-gi1----gi1-[core01]
[net02]----gi11/
Let's see some interfaces! Remember, vlan 12 is the problem vlan. Of the others I explicitly verified that vlan 27 was not affected.
Here's the two hosts' ports: esx01 contains net02.
sw01#sh run int gi11
interface gigabitethernet11
description esx01
lldp med disable
switchport trunk allowed vlan add 5-7,11-13,100
switchport trunk native vlan 27
!
sw01#sh run int gi44
interface gigabitethernet44
description net01-1
lldp med disable
switchport mode access
switchport access vlan 12
!
Here's the trunk on sw01.
sw01#sh run int gi1
interface gigabitethernet1
description "trunk to core01"
lldp med disable
switchport trunk allowed vlan add 4-7,11-13,27,100
!
And the other end of the trunk on core01.
interface gigabitethernet1
description sw01
macro description switch
switchport trunk allowed vlan add 2-7,11-16,27,100
!
I have a monitor port on core01, thus:
core01#sh run int gi12
interface gigabitethernet12
description "monitor port"
port monitor GigabitEthernet 1
!
And the monitor port on core01 sees unicast traffic going between net01 and net02, both of which are on sw01! I've verified this with a monitor port on sw01 that sees the net01-net02 unicast traffic leaving via gi1 too.
sw01 knows that both of those hosts are on ports that are not its trunk port:
:) ratchet$ arp -a | grep net
net02.2ndsiteinc.com (10.12.0.11) at 00:0C:29:1A:66:15 [ether] on eth0
net01.2ndsiteinc.com (10.12.0.10) at 00:11:43:D8:9F:94 [ether] on eth0
sw01#sh mac addr addr 00:0C:29:1A:66:15
Aging time is 300 sec
Vlan Mac Address Port Type
-------- --------------------- ---------- ----------
12 00:0c:29:1a:66:15 gi11 dynamic
sw01#sh mac addr addr 00:11:43:D8:9F:94
Aging time is 300 sec
Vlan Mac Address Port Type
-------- --------------------- ---------- ----------
12 00:11:43:d8:9f:94 gi44 dynamic
I also brought up an unused port on sw01 on vlan 12, but the unicast traffic was (as best as I could tell) not coming out that port. So it doesn't look like sw01 is pushing it out all its ports, just the right ports and also gi1!
I've verified that sw01 is not filling up its address-table:
sw01#sh mac addr count
This may take some time.
Capacity : 8192
Free : 7983
Used : 208
The full configs for both core01 and sw01 are available: core01, sw01.
Finally, versions:
sw01#sh ver
SW version 1.1.2.0 ( date 12-Nov-2011 time 23:34:26 )
Boot version 1.0.0.4 ( date 08-Apr-2010 time 16:37:57 )
HW version V01
core01#sh ver
SW version 1.1.2.0 ( date 12-Nov-2011 time 23:34:26 )
Boot version 1.1.0.6 ( date 11-May-2011 time 18:31:00 )
HW version V01
So my understanding is this: sw01 should take unicast traffic for net01 and send it only out net02's port, and vice versa; none of it should go out sw01's uplink. But core01, receiving traffic on gi1 for a host it knows is on gi1, is right in sending it out all of its ports. (That is: sw01 is misbehaving, but core01 is doing what it should given the circumstances.)
My question is: Why is sw01 sending that unicast traffic out its uplink, gi1?
(And pre-emptively: yes, I know SG300s leave much to be desired, and yes, we should have spanning-tree enabled, but that's where I'm at right now.)
