Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2
rpokala at panasas.com
Thu Mar 4 19:00:47 UTC 2021
From: Hans Petter Selasky <hps at selasky.org>
Date: 2021-03-04, Thursday at 02:13
To: "Foster, Greg" <gfoster at panasas.com>, "freebsd-infiniband at freebsd.org" <freebsd-infiniband at freebsd.org>
Cc: Ravi Pokala <rpokala at panasas.com>
Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2
On 3/4/21 3:49 AM, Foster, Greg wrote:
> Hello FreeBSD IB,
> What is the procedure to change between Infiniband Connected Mode and Datagram Mode
> at runtime on FreeBSD 12.2?
> I've seen different references on the WEB, but the procedure still seems unclear.
> In one reference https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-May/000032.html
> refers to the Linux procedure, which is not applicable on FreeBSD.
> Another reference, https://wiki.freebsd.org/InfiniBand, implies the driver must be recompiled
> to change modes.
> Yet another, https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-June/000047.html
> Suggests it can be done at runtime to add a tuneable/sysctl but doesn't specify any details...
> Is the solution to maybe compile the drivers twice, once for CM and once for DM, keeping both around
> and loading the desired one?
For IPoIB there is an option you need to set at compile time for this
feature to be activated:
It looks like IPOIB_CM causes CONFIG_INFINIBAND_IPOIB_CM to be defined. I see that CONFIG_INFINIBAND_IPOIB_CM adds fields to structures, which is why it can't be a run-time or boot-time tunable. :-(
But it looks like the changes associated with CONFIG_INFINIBAND_IPOIB_CM are all additions, not replacing datagram code with connected-mode code. Other than having unnecessary fields in structures, would there be any harm in building with IPOIB_CM, but running in datagram mode? Or are you saying that we first need to build with IPOIB_CM, and then we can use a tunable to switch modes?
More information about the freebsd-infiniband