Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2

[Pokala, Ravi]

-----Original Message-----
From: Hans Petter Selasky <hps at selasky.org>
Date: 2021-03-04, Thursday at 11:11
To: Ravi Pokala <rpokala at panasas.com>, "Foster, Greg" <gfoster at panasas.com>, "freebsd-infiniband at freebsd.org" <freebsd-infiniband at freebsd.org>
Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2

    On 3/4/21 8:00 PM, Pokala, Ravi wrote:
    > -----Original Message-----
    > From: Hans Petter Selasky <hps at selasky.org>
    > Date: 2021-03-04, Thursday at 02:13
    > To: "Foster, Greg" <gfoster at panasas.com>, "freebsd-infiniband at freebsd.org" <freebsd-infiniband at freebsd.org>
    > Cc: Ravi Pokala <rpokala at panasas.com>
    > Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2
    > 
    >      On 3/4/21 3:49 AM, Foster, Greg wrote:
    >      > Hello FreeBSD IB,
    >      >
    >      > What is the procedure to change between Infiniband Connected Mode and Datagram Mode
    >      > at runtime on FreeBSD 12.2?
    >      >
    >      > I've seen different references on the WEB, but the procedure still seems unclear.
    >      >
    >      > In one reference https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-May/000032.html
    >      > refers to the Linux procedure, which is not applicable on FreeBSD.
    >      >
    >      > Another reference, https://wiki.freebsd.org/InfiniBand, implies the driver must be recompiled
    >      > to change modes.
    >      >
    >      > Yet another, https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-June/000047.html
    >      > Suggests it can be done at runtime to add a tuneable/sysctl but doesn't specify any details...
    >      >
    >      > Is the solution to maybe compile the drivers twice, once for CM and once for DM, keeping both around
    >      > and loading the desired one?
    > 
    >      Hi,
    > 
    >      For IPoIB there is an option you need to set at compile time for this
    >      feature to be activated:
    > 
    >      sys/conf/options:IPOIB_CM	opt_ofed.h
    > 
    >      --HPS
    > 
    > Hi Hans,
    > 
    > It looks like IPOIB_CM causes CONFIG_INFINIBAND_IPOIB_CM to be defined. I see that CONFIG_INFINIBAND_IPOIB_CM adds fields to structures, which is why it can't be a run-time or boot-time tunable. :-(
    > 
    > But it looks like the changes associated with CONFIG_INFINIBAND_IPOIB_CM are all additions, not replacing datagram code with connected-mode code. Other than having unnecessary fields in structures, would there be any harm in building with IPOIB_CM, but running in datagram mode? Or are you saying that we first need to build with IPOIB_CM, and then we can use a tunable to switch modes?
    > 

    Hi,

    At the moment there is no tunable for this feature, as I'm aware of. The 
    easiest would be to build ipoib two times, one time with CM enabled and 
    one time without CM enabled.

    --HPS

Hi Hans,

Couldn't we do the following:

(a) Get rid of IPOIB_CM as a compile-time option
(b) Remove CONFIG_INFINIBAND_IPOIB_CM guards when defining structures
(c) Change CONFIG_INFINIBAND_IPOIB_CM guards on the code, to being conditional on a new tunable.

It seems like that would do what Greg and I are looking for.

Alternatively, we could *not* do (a) or (b), just add the check for the tunable described in (c), and then build with IPOIB_CM. That would minimize the change, and still give us control of the code via the tunable.

If either of those sound okay, I'm happy to make the change myself and put you down at the reviewer in Phabricator.

Thanks,

Ravi