InfiniBand virtual function fails to attach in FreeBSD guest

[John Nielsen]

I have enabled SR-IOV on a Mellanox Connect-X 3 HCA in a physical server running Linux and KVM.

I am running a virtual machine which includes a virtual function of the HCA and is running FreeBSD 11-CURRENT (r287472).

I rebuilt world and kernel with “WITH_OFED=yes” in make.conf and all the wiki-suggested drivers in the kernel:
options        OFED            # Infiniband protocol stack and support
options        SDP             # Sockets Direct Protocol for infiniband
options        IPOIB_CM        # Use connect mode ipoib
device         ipoib           # IP over IB devices
device         mlx4ib          # ConnectX Infiniband support
device         mlxen           # ConnectX Ethernet support
device         mthca           # Infinihost cards

When the VM boots up, the card is recognized but does not attach:

# dmesg|grep mlx
mlx4_core0: <mlx4_core> mem 0xfe000000-0xfe7fffff at device 5.0 on pci0
mlx4_core: Mellanox ConnectX core driver v2.1 (Sep  4 2015)
mlx4_core: Initializing mlx4_core
mlx4_core0: Missing DCS, aborting.(driver_data: 0x0, pci_resource_flags(pdev, 0):0x0)
device_attach: mlx4_core0 attach returned 19

The same VM running Linux works fine:

# dmesg|grep mlx4
[    9.882096] mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014)
[    9.883305] mlx4_core: Initializing 0000:00:05.0
[    9.886389] mlx4_core 0000:00:05.0: Detected virtual function - running in slave mode
[    9.888431] mlx4_core 0000:00:05.0: Sending reset
[    9.904214] mlx4_core 0000:00:05.0: Sending vhcr0
[    9.906832] mlx4_core 0000:00:05.0: HCA minimum page size:512
[    9.908979] mlx4_core 0000:00:05.0: Timestamping is not supported in slave mode
[    9.985162] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1 (Feb 2014)
[   14.113288] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v2.2-1 (Feb 2014)
[   14.114024] <mlx4_ib> check_flow_steering_support: Device managed flow steering is unavailable for IB port in multifunction env.
[   14.191456] mlx4_core 0000:00:05.0: mlx4_ib: multi-function enabled
[   14.192722] mlx4_core 0000:00:05.0: mlx4_ib: operating in qp1 tunnel mode

# ip a show dev ib0
3: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast state UP qlen 256
    link/infiniband 80:00:0a:12:fe:80:00:00:00:00:00:00:3a:00:f9:de:9d:c0:ee:02 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 172.19.0.253/21 scope global ib0
       valid_lft forever preferred_lft forever
    inet6 fe80::3a00:f9de:9dc0:ee02/64 scope link 
       valid_lft forever preferred_lft forever


The Linux code says "Detected virtual function - running in slave mode.” Does the same functionality exist in FreeBSD?

Thanks,

JN