Re: Chelsio NIC with RSS - Traffic distribution to different Queues

Sorry for the spam, I do see the values with sysctl now. It seems that FreeBSD always loads the if_cxgbe.ko from /boot/kernel/if_cxgbe.ko. So what I’ve done is, I renamed the original file and copied the newly compiled if_cxgbe.ko from /boot/module to /boot/kernel. Is there a cleaner way to get it work? Btw. Do I need t5fw_cfg.ko as well, I haven’t found any documentation what exactly it does…

Quiet difficult for someone who isn’t familiar with FreeBSD at all :-P.

So I’ve retested again. Sadly it does still share the load over all configured CPUs (0-3) for RSS (there are 8 cores plus 8 HT cores per physical CPU). As I already mentioned, the new sysctl values are now visible, so I think the driver should be fine. My expectation with that configuration was, that CPU0 shouldn’t be used for RSS (as all values are smaller than the available CPUs), but it has been used, so the network traffic does flapping as hell.

hw.cxgbe.cong_drop="1"
hw.cxgbe.nrxq="3"
hw.cxgbe.pause_settings="0"
hw.cxgbe.rsrv_noflowq="1"
hw.cxgbe.rsrv_norssq="1"
hw.cxgbe.rx_budget="128"
if_cxgbe_load="yes"
net.inet.rss.bits="2"
net.inet.rss.enabled="1"
net.isr.bindthreads="1"
net.isr.maxthreads="-1"
t5fw_cfg_load="yes"

Do I miss anything?

Cheers Josef


From: Zahner Josef, GSA-REP-LI <Josef.Zahner1@swisscom.com>
Date: Monday, 3 July 2023 at 16:50
To: Navdeep Parhar <np@freebsd.org>
Cc: freebsd-net@freebsd.org <freebsd-net@freebsd.org>
Subject: Re: Chelsio NIC with RSS - Traffic distribution to different Queues
Again me, I tried to build the chelsio driver (if_cxgbe) on my machine.

root@fw-94:/usr/src/sys/modules/cxgbe # make ; make install -> no errors so far and I see the new kernel module in /boot/modules:

root@fw-94:/usr/src/sys/modules/cxgbe # ls -alh /boot/modules/
total 4584
drwxr-xr-x   2 root  wheel    16B Jul  3 16:37 .
drwxr-xr-x  15 root  wheel    68B Jul  3 16:37 ..
-r-xr-xr-x   1 root  wheel    46K Jul  3 16:37 ccr.ko
-r-xr-xr-x   1 root  wheel    47K Jul  3 16:37 cxgbei.ko
-r-xr-xr-x   1 root  wheel    11K Jul  3 16:37 if_cc.ko
-r-xr-xr-x   1 root  wheel    11K Jul  3 16:37 if_ccv.ko
-r-xr-xr-x   1 root  wheel   746K Jul  3 16:37 if_cxgbe.ko
-r-xr-xr-x   1 root  wheel    41K Jul  3 16:37 if_cxgbev.ko
-r-xr-xr-x   1 root  wheel    11K Jul  3 16:37 if_cxl.ko
-r-xr-xr-x   1 root  wheel    11K Jul  3 16:37 if_cxlv.ko
-r-xr-xr-x   1 root  wheel   154K Jul  3 16:37 iw_cxgbe.ko
-rw-r--r--   1 root  wheel   1.3K Jul  3 16:37 linker.hints
-r-xr-xr-x   1 root  wheel   131K Jul  3 16:37 t4_tom.ko
-r-xr-xr-x   1 root  wheel   596K Jul  3 16:37 t4fw_cfg.ko
-r-xr-xr-x   1 root  wheel   725K Jul  3 16:37 t5fw_cfg.ko
-r-xr-xr-x   1 root  wheel   777K Jul  3 16:37 t6fw_cfg.ko

However still the same result, the new sysctl values are not visible. Any help would be appreciated.

Btw.: reboot has been done.

Cheers Josef

From: Zahner Josef, GSA-REP-LI <Josef.Zahner1@swisscom.com>
Date: Monday, 3 July 2023 at 15:45
To: Navdeep Parhar <np@freebsd.org>
Cc: freebsd-net@freebsd.org <freebsd-net@freebsd.org>
Subject: Re: Chelsio NIC with RSS - Traffic distribution to different Queues
Hi Navdeep

Nevermind, I found the following command to apply your patch diff:
root@fw-94:/sys/dev/cxgbe # patch < /root/chelsio_rss_patch.diff

There was an inconsistency with “adapter.h”, your change on line 219 was in my case a few lines further below, so I decided to manually edit those two changes in the file.

However I tried to load the new sysctl values like “hw.cxgbe.rsrv_norssq» in loader.conf, how can I verify that it works as it should? It seems that they don’t work as the don’t show up when I try to print out the new sysctl values. You mentioned that I’ve to patch the kernel, do you mean execute the patch diff? Or is there anything else I’ve to do except to execute your patch diff? What about my driver, do I need to change anything there?

Thank you, cheers
Josef