Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2

[Hans Petter Selasky]

On 3/4/21 8:00 PM, Pokala, Ravi wrote:
> -----Original Message-----
> From: Hans Petter Selasky <hps at selasky.org>
> Date: 2021-03-04, Thursday at 02:13
> To: "Foster, Greg" <gfoster at panasas.com>, "freebsd-infiniband at freebsd.org" <freebsd-infiniband at freebsd.org>
> Cc: Ravi Pokala <rpokala at panasas.com>
> Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2
> 
>      On 3/4/21 3:49 AM, Foster, Greg wrote:
>      > Hello FreeBSD IB,
>      >
>      > What is the procedure to change between Infiniband Connected Mode and Datagram Mode
>      > at runtime on FreeBSD 12.2?
>      >
>      > I've seen different references on the WEB, but the procedure still seems unclear.
>      >
>      > In one reference https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-May/000032.html
>      > refers to the Linux procedure, which is not applicable on FreeBSD.
>      >
>      > Another reference, https://wiki.freebsd.org/InfiniBand, implies the driver must be recompiled
>      > to change modes.
>      >
>      > Yet another, https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-June/000047.html
>      > Suggests it can be done at runtime to add a tuneable/sysctl but doesn't specify any details...
>      >
>      > Is the solution to maybe compile the drivers twice, once for CM and once for DM, keeping both around
>      > and loading the desired one?
> 
>      Hi,
> 
>      For IPoIB there is an option you need to set at compile time for this
>      feature to be activated:
> 
>      sys/conf/options:IPOIB_CM	opt_ofed.h
> 
>      --HPS
> 
> Hi Hans,
> 
> It looks like IPOIB_CM causes CONFIG_INFINIBAND_IPOIB_CM to be defined. I see that CONFIG_INFINIBAND_IPOIB_CM adds fields to structures, which is why it can't be a run-time or boot-time tunable. :-(
> 
> But it looks like the changes associated with CONFIG_INFINIBAND_IPOIB_CM are all additions, not replacing datagram code with connected-mode code. Other than having unnecessary fields in structures, would there be any harm in building with IPOIB_CM, but running in datagram mode? Or are you saying that we first need to build with IPOIB_CM, and then we can use a tunable to switch modes?
> 

Hi,

At the moment there is no tunable for this feature, as I'm aware of. The 
easiest would be to build ipoib two times, one time with CM enabled and 
one time without CM enabled.

--HPS