FBSD to FBSD NFS Mounts over IB.

[Janky Jay, III]

Hello All,

	We're recently been poking around with IB support in FBSD because we 
would really like to use ZFS for cluster storage. We're currently 
testing between two FBSD (10.0-R-p2) systems and two Linux (CentOS 6.5 
2.6.32) systems and we're running into something a little strange...

	I have set up one of the FBSD systems to run OpenSM and also be an NFS 
server which all the systems seem to be able to mount over the IB 
devices without any issue at all. Small reads and writes to and from the 
NFS server to all the other nodes also seems to work without any issue. 
However, if I try to dump large amounts of data using "dd" (in order to 
test speeds and stability), the FBSD NFS client craps out immediately. I 
just get the following message(s) over and over:

newnfs server 10.11.1.1:/data: not responding
newnfs server 10.11.1.1:/data: not responding
newnfs server 10.11.1.1:/data: not responding
newnfs server 10.11.1.1:/data: not responding
newnfs server 10.11.1.1:/data: not responding
newnfs server 10.11.1.1:/data: not responding

	Strangely enough, though, the Linux nodes doesn't seem to have a 
problem at all. I'm unsure if anyone else has seen this problem or if 
it's even an IB issue at all (could be an NFS issue). There just doesn't 
seem to be much (if any) documentation about NFS over IB on FBSD any so 
I'm just curious if someone here may be able to point me in the right 
direction. The output of "ibstat" for both FBSD nodes is as follows:

node1 (NFS Server)
CA 'mlx4_0'
         CA type: MT26428
         Number of ports: 1
         Firmware version: 2.9.1000
         Hardware version: b0
         Node GUID: 0x002590ffff16a3cc
         System image GUID: 0x002590ffff16a3cf
         Port 1:
                 State: Active
                 Physical state: LinkUp
                 Rate: 40
                 Base lid: 1
                 LMC: 0
                 SM lid: 1
                 Capability mask: 0x0251086a
                 Port GUID: 0x002590ffff16a3cd

node2 (NFS Client)
CA 'mlx4_0'
         CA type: MT26428
         Number of ports: 1
         Firmware version: 2.9.1000
         Hardware version: b0
         Node GUID: 0x002590ffff16b3a0
         System image GUID: 0x002590ffff16b3a3
         Port 1:
                 State: Active
                 Physical state: LinkUp
                 Rate: 40
                 Base lid: 3
                 LMC: 0
                 SM lid: 1
                 Capability mask: 0x02510868
                 Port GUID: 0x002590ffff16b3a1

	If anyone has any ideas as to what I can try, it would be greatly 
appreciated!

Regards,
Janky Jay, III