Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2
Hans Petter Selasky
hps at selasky.org
Fri Mar 5 08:25:35 UTC 2021
On 3/4/21 8:37 PM, Pokala, Ravi wrote:
> -----Original Message-----
> From: Hans Petter Selasky <hps at selasky.org>
> Date: 2021-03-04, Thursday at 11:11
> To: Ravi Pokala <rpokala at panasas.com>, "Foster, Greg" <gfoster at panasas.com>, "freebsd-infiniband at freebsd.org" <freebsd-infiniband at freebsd.org>
> Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2
> On 3/4/21 8:00 PM, Pokala, Ravi wrote:
> > -----Original Message-----
> > From: Hans Petter Selasky <hps at selasky.org>
> > Date: 2021-03-04, Thursday at 02:13
> > To: "Foster, Greg" <gfoster at panasas.com>, "freebsd-infiniband at freebsd.org" <freebsd-infiniband at freebsd.org>
> > Cc: Ravi Pokala <rpokala at panasas.com>
> > Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2
> > On 3/4/21 3:49 AM, Foster, Greg wrote:
> > > Hello FreeBSD IB,
> > >
> > > What is the procedure to change between Infiniband Connected Mode and Datagram Mode
> > > at runtime on FreeBSD 12.2?
> > >
> > > I've seen different references on the WEB, but the procedure still seems unclear.
> > >
> > > In one reference https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-May/000032.html
> > > refers to the Linux procedure, which is not applicable on FreeBSD.
> > >
> > > Another reference, https://wiki.freebsd.org/InfiniBand, implies the driver must be recompiled
> > > to change modes.
> > >
> > > Yet another, https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-June/000047.html
> > > Suggests it can be done at runtime to add a tuneable/sysctl but doesn't specify any details...
> > >
> > > Is the solution to maybe compile the drivers twice, once for CM and once for DM, keeping both around
> > > and loading the desired one?
> > Hi,
> > For IPoIB there is an option you need to set at compile time for this
> > feature to be activated:
> > sys/conf/options:IPOIB_CM opt_ofed.h
> > --HPS
> > Hi Hans,
> > It looks like IPOIB_CM causes CONFIG_INFINIBAND_IPOIB_CM to be defined. I see that CONFIG_INFINIBAND_IPOIB_CM adds fields to structures, which is why it can't be a run-time or boot-time tunable. :-(
> > But it looks like the changes associated with CONFIG_INFINIBAND_IPOIB_CM are all additions, not replacing datagram code with connected-mode code. Other than having unnecessary fields in structures, would there be any harm in building with IPOIB_CM, but running in datagram mode? Or are you saying that we first need to build with IPOIB_CM, and then we can use a tunable to switch modes?
> At the moment there is no tunable for this feature, as I'm aware of. The
> easiest would be to build ipoib two times, one time with CM enabled and
> one time without CM enabled.
> Hi Hans,
> Couldn't we do the following:
> (a) Get rid of IPOIB_CM as a compile-time option
> (b) Remove CONFIG_INFINIBAND_IPOIB_CM guards when defining structures
> (c) Change CONFIG_INFINIBAND_IPOIB_CM guards on the code, to being conditional on a new tunable.
> It seems like that would do what Greg and I are looking for.
Sounds good to me.
If you want to work on it, please go ahead and make a review.
> Alternatively, we could *not* do (a) or (b), just add the check for the tunable described in (c), and then build with IPOIB_CM. That would minimize the change, and still give us control of the code via the tunable.
> If either of those sound okay, I'm happy to make the change myself and put you down at the reviewer in Phabricator.
More information about the freebsd-infiniband