arp problems with transparent bridge on linux
- by Mink
I've been trying to secure my virtual machines on my esx server by putting them behind a transparent bridge with 2 interfaces, one in front, one at the back.
My intention is to put all the firewall rules in one place (instead of on each virtual server).
I've been using as bridge a blank new virtual machine based on arch linux (but I suspect it doesn't matter which brand of linux it is).
What I have is 2 virtual switchs (thus two Virtual Network, VN_front and VN_back), each with 2 types of ports (switched/separated or promiscious/where the machine can see all packets).
On my bridge machine, I've set up 2 virtual NIC, one on VN_front, one on VN_back, both in promisc mode.
I've created a bridge br0 with both NIC in it:
brctl addbr br0
brctl stp br0 off
brctl addif br0 front_if
brctl addif br0 back_if
Then brought them up:
ifconfig front_if 0.0.0.0 promisc
ifconfig back_if 0.0.0.0 promisc
ifconfig br0 0.0.0.0
(I use promisc mode, because I'm not sure I can do without, thinking that maybe the packets don't reach the NICs)
Then I took one of my virtual server sitting on VN_front, and plugged it to VN_back instead (that's the nifty use case I'm thinking about, being able to move my servers around just by changing the VN they are plugged into, without changing anything in the configuration).
Then I looked into the macs "seen" by my addressless bridge using brctl showmacs br0 and it did show my server from both sides:
I get something that looks like this :
port no mac addr is local? ageing timer
2 00:0c:29:e1:54:75 no 9.27
1 00:0c:29:fd:86:0c no 9.27
2 00:50:56:90:05:86 no 73.38
1 00:50:56:90:05:88 no 0.10
2 00:50:56:90:05:8b yes 0.00 << FRONT VN
1 00:50:56:90:05:8c yes 0.00 << BACK VN
2 00:50:56:90:19:18 no 13.55
2 00:50:56:90:3c:cf no 13.57
the thing is that the server that are plugged in front/back are not shown on the correct port.
I suspect some horrible thing happening in the ARP-world... :-/
If I ping from a front virtual server to a back virtual server, I can only see the back machine if that back machine pings something in the front. As soon as I stop the ping from the back machine, the ping from the front machine stops getting through...
I've noticed that if the back machine pings, then its port on the bridge is the correct one...
I've tried to play with the arp_ switch of /proc/sys, but with no clear effect on the end result...
/proc/sys/net/ipv4/ip_forward doesn't seem to be of any use when using a bridge (seems it's all taken care of by brctl)
/proc/sys/net/ipv4/conf//arp_ don't seem to change much either... (tried arp_announce to 2 or 8 - like suggested elsewhere - and arp_ignore to 0 or 1 )
All the examples I've seen have a different subnet on either side like 10.0.1.0/24 and 10.0.2.0/24... In my case I want 10.0.1.0/24 on both side (just like a transparent switch - except it's a hidden fw ).
Turning stp on/off doesn't seem to have any impact on my issue.
It's as if the arp packets where getting through the bridge, corrupting the other side with false data... I've tried to use the -arp on each interface, br0, front, back... it breaks the thing altogether... I suspect it has something to do with both side being on the same subnet...
I've thought about putting all my machine behind the fw, so as to have all the same subnet at the back... but I'm stuck with my provider's gateway standing at the front with part of my subnet (in fact 3 appliance to route the whole subnet), so I'll always have ips from the same subnet on both side, whatever I do... (I'm using fixed front IPs on my delegated subnet).
I'm at a loss... -_-'' Thx for your help.
(As anyone tried something like this? from within ESXi?)
(It's not just a stunt, the idea is to have something like fail2ban running on some servers, sending their banned IP to the bridge/fw so that it too could ban them - saving all the other servers from that same attacker in one go, allowing for some honeypot that would trigger the fw from any kind of suitable response, and stuffs of the sort...
I am aware I could use something like snort, but it addresses some completely different kind of problems, in a completely different way...
)
