How to force IB cards to come up in IB mode?

[Justin Clift]

Hi all,

I'm trying to get a new FreeBSD 10.3-STABLE install to work well with
my MHGH28-XTC (dual port ConnectX DDR) cards.

The problem is they're coming up in eth mode, not ib mode, and I can't
seem to get that fixed. :(

From boot:

  # ifconfig
  mlxen0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
          options=6d07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
          ether 00:02:c9:00:ca:cd
          nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
          media: Ethernet autoselect
          status: no carrier
  mlxen1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
          options=6d07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
          ether 00:02:c9:00:ca:ce
          nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
          media: Ethernet autoselect
          status: no carrier

My loader.conf file seems like it should be correct:

  # cat /boot/loader.conf
  sys.device.mlx4_core0.mlx4_port1="ib"
  sys.device.mlx4_core0.mlx4_port2="ib"

As background, this is a new install (coming from CentOS 6.x, where
it all works well), so I know there's no problem with the cards.  They're
plugged into a 24 port DDR switch (Voltaire ISR 9024D-M) using active
optical cables.

Any ideas?

Regards and best wishes,

Justin Clift

--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi