Infiniband MTU Issue

From: Tom Arnold <xyzzy_at_sysabend.org>
Date: Mon, 24 Mar 2025 19:07:11 UTC
I dont know if Net is the correct place to ask about Infiniband or not.

Machine is FreeBSD 14.0-RELEASE-p5.

The problem we are seeing is our IB MTU is getting set to 2044.  It is not getting the
MTU setting from the subnet manager like our Linux nodes.  It should be getting set to
4092.

ib0: flags=1008043<UP,BROADCAST,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 2044
        options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
        lladdr 0.0.0.7a.fe.80.0.0.0.0.0.0.10.70.fd.3.0.e5.c6.c8
        inet 10.149.200.68 netmask 0xffff0000 broadcast 10.149.255.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

We vaguely remember an obscure Solaris setting to turn "something" off that solves the
issue but can't find anything for FreeBSD and the Solaris cluster nodes are a decade
gone at least.

Any suggestions?

Thanks!

  -Tom

-- 
 --------------------------------------------------------------------
 - Tom Arnold           -  "...is it a virus, a drug, or a religion?"  
 - Sysabend Caretaker   -    Juanita Shrugs. "What's the difference?"
 ------------------------       -- Neal Stephenson,  Snow Crash