A co-worker asked the following question about a SPARC SuperCluster InfiniBand network:

> on the database nodes the RAC nodes communicate over the cluster_interconnect. This is the
> 192.168.10.0 network on bondib0. (according to ./crs/install/crsconfig_params NETWORKS
> setting) 
> What is bondib1 used for? Is it a HA counterpart in case bondib0 dies?

This is my response:

Summary: bondib1 is currently only being used for outbound cluster interconnect interconnect traffic.

Details:

bondib0 is the cluster_interconnect

$ oifcfg getif           
bondeth0  10.129.184.0  global  public
bondib0  192.168.10.0  global  cluster_interconnect
ipmpapp0  192.168.30.0  global  public

bondib0 and bondib1 are on 192.168.10.1 and 192.168.10.2 respectively.

# ipadm show-addr | grep bondi
bondib0/v4static  static   ok           192.168.10.1/24
bondib1/v4static  static   ok           192.168.10.2/24

Hostnames tied to the IPs are node1-priv1 and node1-priv2 

# grep 192.168.10 /etc/hosts
192.168.10.1    node1-priv1.us.oracle.com   node1-priv1
192.168.10.2    node1-priv2.us.oracle.com   node1-priv2

For the 4 node RAC interconnect:

• Each node has 2 private IP address on the 192.168.10.0 network.
• Each IP address has an active InfiniBand link and a failover InfiniBand link.
• Thus, the 4 node RAC interconnect is using a total of 8 IP addresses and 16 InfiniBand links.

bondib1 isn't being used for the Virtual IP (VIP):

$ srvctl config vip -n node1
VIP exists: /node1-ib-vip/192.168.30.25/192.168.30.0/255.255.255.0/ipmpapp0, hosting node node1
VIP exists: /node1-vip/10.55.184.15/10.55.184.0/255.255.255.0/bondeth0, hosting node node1

bondib1 is on bondib1_0 and fails over to bondib1_1:

# ipmpstat -g
GROUP       GROUPNAME   STATE     FDT       INTERFACES
ipmpapp0    ipmpapp0    ok        --        ipmpapp_0 (ipmpapp_1)
bondeth0    bondeth0    degraded  --        net2 [net5]
bondib1     bondib1     ok        --        bondib1_0 (bondib1_1)
bondib0     bondib0     ok        --        bondib0_0 (bondib0_1)

bondib1_0 goes over net24

# dladm show-link | grep bond
LINK                CLASS     MTU    STATE    OVER
bondib0_0           part      65520  up       net21
bondib0_1           part      65520  up       net22
bondib1_0           part      65520  up       net24
bondib1_1           part      65520  up       net23

net24 is IB Partition FFFF

# dladm show-ib
LINK         HCAGUID         PORTGUID        PORT STATE  PKEYS
net24        21280001A1868A  21280001A1868C  2    up     FFFF
net22        21280001CEBBDE  21280001CEBBE0  2    up     FFFF,8503
net23        21280001A1868A  21280001A1868B  1    up     FFFF,8503
net21        21280001CEBBDE  21280001CEBBDF  1    up     FFFF

On Express Module 9 port 2:

# dladm show-phys -L
LINK              DEVICE       LOC
net21             ibp4         PCI-EM1/PORT1
net22             ibp5         PCI-EM1/PORT2
net23             ibp6         PCI-EM9/PORT1
net24             ibp7         PCI-EM9/PORT2

Outbound traffic on the 192.168.10.0 network will be multiplexed between bondib0 & bondib1

# netstat -rn

Routing Table: IPv4
  Destination           Gateway           Flags  Ref     Use     Interface
-------------------- -------------------- ----- ----- ---------- ---------
192.168.10.0         192.168.10.2         U        16    6551834 bondib1  
192.168.10.0         192.168.10.1         U         9    5708924 bondib0  

There is a lot more traffic on bondib0 than bondib1

# /bin/time snoop -I bondib0 -c 100 > /dev/null
Using device ipnet/bondib0 (promiscuous mode)
100 packets captured

real        4.3
user        0.0
sys         0.0

(100 packets in 4.3 seconds = 23.3 pkts/sec)

# /bin/time snoop -I bondib1 -c 100 > /dev/null
Using device ipnet/bondib1 (promiscuous mode)
100 packets captured

real       13.3
user        0.0
sys         0.0

(100 packets in 13.3 seconds = 7.5 pkts/sec)

Half of the packets on bondib0 are outbound (from self). The remaining packet are split evenly, from the other nodes in the cluster.

# snoop -I bondib0 -c 100 | awk '{print $1}' | sort | uniq -c
Using device ipnet/bondib0 (promiscuous mode)
100 packets captured
  49 node1-priv1.us.oracle.com
  24 node2-priv1.us.oracle.com
  14 node3-priv1.us.oracle.com
  13 node4-priv1.us.oracle.com

100% of the packets on bondib1 are outbound (from self), but the headers in the packets indicate that they are from the IP address associated with bondib0:

# snoop -I bondib1 -c 100 | awk '{print $1}' | sort | uniq -c
Using device ipnet/bondib1 (promiscuous mode)
100 packets captured
 100 node1-priv1.us.oracle.com

The destination of the bondib1 outbound packets are split evenly, to node3 and node 4.

# snoop -I bondib1 -c 100 | awk '{print $3}' | sort | uniq -c
Using device ipnet/bondib1 (promiscuous mode)
100 packets captured
  51 node3-priv1.us.oracle.com
  49 node4-priv1.us.oracle.com

Conclusion: bondib1 is currently only being used for outbound cluster interconnect interconnect traffic.