OFED setup and Mellanox cards

[Tamer Abuelata]

Hi,

I'm trying to get OFED up and running on a FreeBSD 9.1 machine with 2 Mellanox cards (MT26428).
We have compiled the Kernel with the following options (per https://wiki.freebsd.org/InfiniBand ):

options        OFED            # Infiniband protocol stack and support
options        SDP              # Sockets Direct Protocol for infiniband
options        IPOIB_CM    # Use connect mode ipoib

device         ipoib             # IP over IB devices
device         mlx4ib          # ConnectX Infiniband support
device         mlxen           # ConnectX Ethernet support
device         mthca           # Infinihost cards

I gave an IP to each card.
# ifconfig
ib0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 65520
	options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
	lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.0.25.90.ff.ff.7.70.d9
	inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ib1: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 65520
	options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
	lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.0.25.90.ff.ff.7.70.f1
	inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

After doing a build world and build kernel I try to run an example application and get the following warnings/errors:

# ./server 
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
listening on port 29823.

Then I launch the client from another terminal:
# ./client 10.0.0.2 29823
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
[… shows several more times …]
Segmentation fault (core dumped)

So the sample app compiles fine but I get this runtime error which seems to come from the libibverbs. This /sys/class/infiniband_verbs folder does not exist. I'm not sure if I'm missing configuration files. However I have devices at /dev/uverbs0 and /dev/uverbs1.

The example app I am using is from:
http://thegeekinthecorner.wordpress.com/2010/08/14/building-an-rdma-capable-application-with-ib-verbs-part-2-the-server/

The code is linked at the bottom of the page. Direct link:
https://sites.google.com/a/bedeir.com/home/basic-rdma-client-server.tar.gz?attredirects=0&d=1

I would appreciate any help or pointers from someone who managed to get OFED to run on FreeBSD.

Best regards,
Tamer