IPOIB perforance in 11.0-current

[Chris Reeves]

Hi All,


I've been trying to benchmark performance using IPOIB on FreeBSD 11.0-current, but I'm not getting the speeds I think I should.  The best I'm seeing on a 56 Gb/s link is about 10 Gb/s.  I know I won't see line speed using IPOIB, but this performance still seems slow...


I have two hosts, each with a MLX 354A card, connected to an FDR switch.  The ports show connected at 56 Gb/s.  I'm running in connected mode with an MTU of 65520.

Using iperf as a benchmark, the best performance I can get is about 10 Gb/s one way.  Using the exact same setup under Ubuntu, I've been able to get IPOIB performance of 18 Gb/s.


Does anyone have any advice for improving my performance in FreeBSD?  I've tried datagram mode, but performance is slower (about 5 Gb/s) which I think is expected.  I've tried different window sizes in iperf, but haven't been able to get any higher than 10 Gb/s.  I've also turned off all the debug option in my kernel config.  Should I expect better performance?  Are there any other changes I can make to speed things up?


Thanks very much,

Chris