A co-worker asked the following question about a SPARC SuperCluster InfiniBand network:
> on the database nodes the RAC nodes communicate over the
cluster_interconnect. This is the > 192.168.10.0 network on bondib0.
(according to ./crs/install/crsconfig_params NETWORKS> setting) > What is bondib1 used for? Is it a HA counterpart in case bondib0 dies?
This is my response:
Summary: bondib1 is currently only being used for outbound cluster interconnect interconnect traffic.
Details:
bondib0 is the cluster_interconnect $ oifcfg getif
bondeth0 10.129.184.0 global public
bondib0 192.168.10.0 global cluster_interconnect
ipmpapp0 192.168.30.0 global public
bondib0 and bondib1 are on 192.168.10.1 and 192.168.10.2 respectively. # ipadm show-addr | grep bondi
bondib0/v4static static ok 192.168.10.1/24
bondib1/v4static static ok 192.168.10.2/24
Hostnames tied to the IPs are node1-priv1 and node1-priv2 # grep 192.168.10 /etc/hosts
192.168.10.1 node1-priv1.us.oracle.com node1-priv1
192.168.10.2 node1-priv2.us.oracle.com node1-priv2
For the 4 node RAC interconnect:
Each node has 2 private IP address on the 192.168.10.0 network.
Each IP address has an active InfiniBand link and a failover InfiniBand link.
Thus, the 4 node RAC interconnect is using a total of 8 IP addresses and 16 InfiniBand links.
bondib1 isn't being used for the Virtual IP (VIP): $ srvctl config vip -n node1
VIP exists: /node1-ib-vip/192.168.30.25/192.168.30.0/255.255.255.0/ipmpapp0, hosting node node1
VIP exists: /node1-vip/10.55.184.15/10.55.184.0/255.255.255.0/bondeth0, hosting node node1
bondib1 is on bondib1_0 and fails over to bondib1_1: # ipmpstat -g
GROUP GROUPNAME STATE FDT INTERFACES
ipmpapp0 ipmpapp0 ok -- ipmpapp_0 (ipmpapp_1)
bondeth0 bondeth0 degraded -- net2 [net5]
bondib1 bondib1 ok -- bondib1_0 (bondib1_1)
bondib0 bondib0 ok -- bondib0_0 (bondib0_1)
bondib1_0 goes over net24 # dladm show-link | grep bond
LINK CLASS MTU STATE OVER
bondib0_0 part 65520 up net21
bondib0_1 part 65520 up net22
bondib1_0 part 65520 up net24
bondib1_1 part 65520 up net23
net24 is IB Partition FFFF # dladm show-ib
LINK HCAGUID PORTGUID PORT STATE PKEYS
net24 21280001A1868A 21280001A1868C 2 up FFFF
net22 21280001CEBBDE 21280001CEBBE0 2 up FFFF,8503
net23 21280001A1868A 21280001A1868B 1 up FFFF,8503
net21 21280001CEBBDE 21280001CEBBDF 1 up FFFF
On Express Module 9 port 2: # dladm show-phys -L
LINK DEVICE LOC
net21 ibp4 PCI-EM1/PORT1
net22 ibp5 PCI-EM1/PORT2
net23 ibp6 PCI-EM9/PORT1
net24 ibp7 PCI-EM9/PORT2
Outbound traffic on the 192.168.10.0 network will be multiplexed between bondib0 & bondib1 # netstat -rn
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
192.168.10.0 192.168.10.2 U 16 6551834 bondib1
192.168.10.0 192.168.10.1 U 9 5708924 bondib0
There is a lot more traffic on bondib0 than bondib1
# /bin/time snoop -I bondib0 -c 100 > /dev/null
Using device ipnet/bondib0 (promiscuous mode)
100 packets captured
real 4.3
user 0.0
sys 0.0
(100 packets in 4.3 seconds = 23.3 pkts/sec) # /bin/time snoop -I bondib1 -c 100 > /dev/null
Using device ipnet/bondib1 (promiscuous mode)
100 packets captured
real 13.3
user 0.0
sys 0.0
(100 packets in 13.3 seconds = 7.5 pkts/sec)
Half of the packets on bondib0 are outbound (from self). The remaining packet are split evenly, from the other nodes in the cluster. # snoop -I bondib0 -c 100 | awk '{print $1}' | sort | uniq -c
Using device ipnet/bondib0 (promiscuous mode)
100 packets captured
49 node1-priv1.us.oracle.com
24 node2-priv1.us.oracle.com
14 node3-priv1.us.oracle.com
13 node4-priv1.us.oracle.com
100% of the packets on bondib1 are outbound (from self), but the headers in the packets indicate that they are from the IP address associated with bondib0: # snoop -I bondib1 -c 100 | awk '{print $1}' | sort | uniq -c
Using device ipnet/bondib1 (promiscuous mode)
100 packets captured
100 node1-priv1.us.oracle.com
The destination of the bondib1 outbound packets are split evenly, to node3 and node 4. # snoop -I bondib1 -c 100 | awk '{print $3}' | sort | uniq -c
Using device ipnet/bondib1 (promiscuous mode)
100 packets captured
51 node3-priv1.us.oracle.com
49 node4-priv1.us.oracle.com
Conclusion: bondib1 is currently only being used for outbound cluster interconnect interconnect traffic.
