Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2

[Pokala, Ravi]

-----Original Message-----
From: Hans Petter Selasky <hps at selasky.org>
Date: 2021-03-04, Thursday at 02:13
To: "Foster, Greg" <gfoster at panasas.com>, "freebsd-infiniband at freebsd.org" <freebsd-infiniband at freebsd.org>
Cc: Ravi Pokala <rpokala at panasas.com>
Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2

    On 3/4/21 3:49 AM, Foster, Greg wrote:
    > Hello FreeBSD IB,
    > 
    > What is the procedure to change between Infiniband Connected Mode and Datagram Mode
    > at runtime on FreeBSD 12.2?
    > 
    > I've seen different references on the WEB, but the procedure still seems unclear.
    > 
    > In one reference https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-May/000032.html
    > refers to the Linux procedure, which is not applicable on FreeBSD.
    > 
    > Another reference, https://wiki.freebsd.org/InfiniBand, implies the driver must be recompiled
    > to change modes.
    > 
    > Yet another, https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-June/000047.html
    > Suggests it can be done at runtime to add a tuneable/sysctl but doesn't specify any details...
    > 
    > Is the solution to maybe compile the drivers twice, once for CM and once for DM, keeping both around
    > and loading the desired one?

    Hi,

    For IPoIB there is an option you need to set at compile time for this 
    feature to be activated:

    sys/conf/options:IPOIB_CM	opt_ofed.h

    --HPS

Hi Hans,

It looks like IPOIB_CM causes CONFIG_INFINIBAND_IPOIB_CM to be defined. I see that CONFIG_INFINIBAND_IPOIB_CM adds fields to structures, which is why it can't be a run-time or boot-time tunable. :-(

But it looks like the changes associated with CONFIG_INFINIBAND_IPOIB_CM are all additions, not replacing datagram code with connected-mode code. Other than having unnecessary fields in structures, would there be any harm in building with IPOIB_CM, but running in datagram mode? Or are you saying that we first need to build with IPOIB_CM, and then we can use a tunable to switch modes?

Thanks,

Ravi