Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2

[Hans Petter Selasky]

On 3/4/21 8:37 PM, Pokala, Ravi wrote:
> -----Original Message-----
> From: Hans Petter Selasky <hps at selasky.org>
> Date: 2021-03-04, Thursday at 11:11
> To: Ravi Pokala <rpokala at panasas.com>, "Foster, Greg" <gfoster at panasas.com>, "freebsd-infiniband at freebsd.org" <freebsd-infiniband at freebsd.org>
> Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2
> 
>      On 3/4/21 8:00 PM, Pokala, Ravi wrote:
>      > -----Original Message-----
>      > From: Hans Petter Selasky <hps at selasky.org>
>      > Date: 2021-03-04, Thursday at 02:13
>      > To: "Foster, Greg" <gfoster at panasas.com>, "freebsd-infiniband at freebsd.org" <freebsd-infiniband at freebsd.org>
>      > Cc: Ravi Pokala <rpokala at panasas.com>
>      > Subject: Re: Changing Between Infiniband Connected Mode and Datagram Mode at Runtime on FreeBSD 12.2
>      >
>      >      On 3/4/21 3:49 AM, Foster, Greg wrote:
>      >      > Hello FreeBSD IB,
>      >      >
>      >      > What is the procedure to change between Infiniband Connected Mode and Datagram Mode
>      >      > at runtime on FreeBSD 12.2?
>      >      >
>      >      > I've seen different references on the WEB, but the procedure still seems unclear.
>      >      >
>      >      > In one reference https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-May/000032.html
>      >      > refers to the Linux procedure, which is not applicable on FreeBSD.
>      >      >
>      >      > Another reference, https://wiki.freebsd.org/InfiniBand, implies the driver must be recompiled
>      >      > to change modes.
>      >      >
>      >      > Yet another, https://lists.freebsd.org/pipermail/freebsd-infiniband/2013-June/000047.html
>      >      > Suggests it can be done at runtime to add a tuneable/sysctl but doesn't specify any details...
>      >      >
>      >      > Is the solution to maybe compile the drivers twice, once for CM and once for DM, keeping both around
>      >      > and loading the desired one?
>      >
>      >      Hi,
>      >
>      >      For IPoIB there is an option you need to set at compile time for this
>      >      feature to be activated:
>      >
>      >      sys/conf/options:IPOIB_CM	opt_ofed.h
>      >
>      >      --HPS
>      >
>      > Hi Hans,
>      >
>      > It looks like IPOIB_CM causes CONFIG_INFINIBAND_IPOIB_CM to be defined. I see that CONFIG_INFINIBAND_IPOIB_CM adds fields to structures, which is why it can't be a run-time or boot-time tunable. :-(
>      >
>      > But it looks like the changes associated with CONFIG_INFINIBAND_IPOIB_CM are all additions, not replacing datagram code with connected-mode code. Other than having unnecessary fields in structures, would there be any harm in building with IPOIB_CM, but running in datagram mode? Or are you saying that we first need to build with IPOIB_CM, and then we can use a tunable to switch modes?
>      >
> 
>      Hi,
> 
>      At the moment there is no tunable for this feature, as I'm aware of. The
>      easiest would be to build ipoib two times, one time with CM enabled and
>      one time without CM enabled.
> 
>      --HPS
> 
> Hi Hans,
> 
> Couldn't we do the following:
> 
> (a) Get rid of IPOIB_CM as a compile-time option
> (b) Remove CONFIG_INFINIBAND_IPOIB_CM guards when defining structures
> (c) Change CONFIG_INFINIBAND_IPOIB_CM guards on the code, to being conditional on a new tunable.
> 
> It seems like that would do what Greg and I are looking for.

Hi,

Sounds good to me.

If you want to work on it, please go ahead and make a review.

--HPS

> 
> Alternatively, we could *not* do (a) or (b), just add the check for the tunable described in (c), and then build with IPOIB_CM. That would minimize the change, and still give us control of the code via the tunable.
> 
> If either of those sound okay, I'm happy to make the change myself and put you down at the reviewer in Phabricator.
> 
> Thanks,
> 
> Ravi
>