NFS + Infiniband problem

[Rick Macklem]

Andrew Vylegzhanin wrote:
[stuff snipped]
>With this TCP settings same server serve NFS requests via 40G Ethernet on multiply >clients with speed via 1G Eth ~ 105/110 MB/sec  write/read.
>Of course I'll try to change congestion algorithm, but I don't think that will help.
Yes, I doubt changing the congestion algorithm will make much difference.

>Also need to test setup with infniband set from connected mode to datagram mode.
Are you using IPv6 by any chance?
Why I ask is that there was a problem with IPv6 fragmentation re-assembly. If
InfiniBand is using a larger MTU than the ethernet, the switch would probably
fragment the IP datagram. If any fragment is lost (or fragmentation re-assembly
is broken), it isn't going to work well.
"netstat -s | fgrep "fragments dropped after"
will show you the count of fragments dropped after timeout. If this value is
increasing, then fragmentation re-assembly is an issue. (Check on the receiving
end. The server for writes.)

You might want to post on freebsd-net@, since someone there might know more
about InfiniBand (and someone over there will definitely know about the IPv6
fragmentation problem.

rick