switchless Infiniband between two servers on RHEL 6.3
- by exfizik
I have 2 servers running RHEL 6.3 which have 2 port Infiniband cards
>lspci | grep -i infini
07:00.0 InfiniBand: QLogic Corp. IBA7322 QDR InfiniBand HCA (rev 02)
I'm interested in connecting them directly to each other bypassing an Infiniband switch (which I don't have). Quick googling showed that at least in some configurations it's possible.
I installed all RedHat Infiniband packages with yum groupinstall "Infiniband Support".
However, ibv_devinfo shows that both ports in each card are down, which indicates that cables are not connected. But the cable is connected, although the LEDs are off on the cards (not a good sign).
Another source of confusion for me is that according to this, RedHat doesn't come with OFED packages and I'm slightly hesitant to install them from source due to the lack of RedHat support for them... So where am I going with this?
The questions I have are:
is it possible to have a switchless/direct Infiniband connection between two servers the way I described above?
If it's possible, do I have to use the OFED packages or can I configure everything with just the packages coming with RHEL.
Why are the LEDs off on my servers even though the cable is connected?
Any additional input/advice/pointers would be appreciated.
P.S. I followed this guide for installation instructions. The Infiniband cards are clearly recognized by my OS and the rdma service is running.
Update: I have opensm installed. When I run it it says:
OpenSM 3.3.13
Command Line Arguments:
Log File: /var/log/opensm.log
-------------------------------------------------
OpenSM 3.3.13
Entering DISCOVERING state
Using default GUID 0x1175000076e4c8
SM port is down
and stays at that point.
