RE: Ifconfig limitations

Hi Mina!

How about using just ibv_devices and/or ibv_devinfo?

On the other hand, if you are only after RoCE devices (though there are also devices that support iwarp, or both iwarp and roce), you may probably map ib device with ifconfig device using sysctl:

sys.class.infiniband.{ib_device}.ports.1.gid_attrs.ndevs.{id}: {if_name}

For Intel E810 card it looks like this:
sysctl sys.class.infiniband | grep ndev
sys.class.infiniband.irdma-ice0.ports.1.gid_attrs.ndevs.2: ice0
sys.class.infiniband.irdma-ice0.ports.1.gid_attrs.ndevs.1: ice0
sys.class.infiniband.irdma-ice0.ports.1.gid_attrs.ndevs.0: ice0
I believe for mlx5 devices it should work the same.

Note:
It does not work for E810 in iwarp mode. There are 2 other options then:
1) ib_device name for e810 contains the if name - irdma-ice0, where 'irdma' stands for the driver, and ice0 is the interface name
2) ibv_devices return node GUID which could be converted back to mac address:   ww:yy:zz:ff:fe:ss:tt:uu -> (ww^0x02):yy:zz:ss:tt:uu, mlx might have different type of conversion.

Hope this is accurate and helps.

Bartek

-----Original Message-----
From: owner-freebsd-infiniband@freebsd.org <owner-freebsd-infiniband@freebsd.org> On Behalf Of Mina Galić
Sent: Tuesday, April 18, 2023 02:19
To: freebsd-net@freebsd.org; freebsd-infiniband@FreeBSD.org
Subject: Ifconfig limitations

Hi folks!

I've been working on getting better FreeBSD support into cloud-init.
My working thesis has been that all I really need to replicate what cloud-init on Linux is doing with ip addr/ip link and its reading of `/sys/class/net/<device>/*` files is BSD's ifconfig(8)

This ifconfig parser is the implementation of that thesis:
https://github.com/canonical/cloud-init/blob/main/cloudinit/distros/parsers/ifconfig.py

However, trying to port cloud-init's Infiniband code seems to have falsified my hypothesis.

In some cases, it's easily possible to tell if an interface is an Infiniband interface: For example, it'll start with `ib`, or have a GUID as `lladdr`:

https://github.com/canonical/cloud-init/pull/2003/files#diff-f143a014d8daaa1671a7feef90cc2b3f5c5759e234e5cd3382e438dde3703fd3

but in other cases, such as when the device is in Ethernet mode, it's impossible to tell.

It means i would have to parse the highly verbose ibstat(8), then map the Infiniband device name to the ifconfig names and only then can I decide if the devices in ifconfig contain infiniband devices, which ones they are, and what state / mode they are in.

(if they even show up in ifconfig to begin with! tho this might be a skill / knowledge issue, see: https://lists.freebsd.org/archives/freebsd-infiniband/2023-April/000011.html )

On Linux, cloud-init checks `/sys/net/mce0/type`.
If that type is 32, it's infiniband, if it's 1, it's ethernet.

We don't seem to have such a thing.
Sometimes we have useful groups, but not as many useful groups as OpenBSD does, for instance. So I have determined that on FreeBSD, the absence of groups marks a physical device.

But beyond that, there's not much overlap between Infiniband and ifconfig, and that's a real shame.

Let's compare that to wlan!
the actual wifi devices don't show up in ifconfig, but they can be easily found via `sysctl net.wlan.devices`.
Once plumbed in, a wlanX device shows a parent device and belonging to the wlan group.

How hard would it be to emulate that behaviour for Infiniband?
What tooling do we have right now, that I may have missed?

Kind regards,

Mina Galić
Don't Try *my* PkgBase Repository: https://alpha.pkgbase.live/ — until i have a new hardware sponsor…

---------------------------------------------------------------------
Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
Spolka oswiadcza, ze posiada status duzego przedsiebiorcy w rozumieniu ustawy z dnia 8 marca 2013 r. o przeciwdzialaniu nadmiernym opoznieniom w transakcjach handlowych.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.