Re: Chelsio NIC with RSS - Traffic distribution to different Queues

On Mon, Jul 3, 2023 at 8:48 AM <Josef.Zahner1@swisscom.com> wrote:
>
> Sorry for the spam, I do see the values with sysctl now. It seems that FreeBSD always loads the if_cxgbe.ko from /boot/kernel/if_cxgbe.ko. So what I’ve done is, I renamed the original file and copied the newly compiled if_cxgbe.ko from /boot/module to /boot/kernel. Is there a cleaner way to get it work? Btw. Do I need t5fw_cfg.ko as well, I haven’t found any documentation what exactly it does…
>
>
>
> Quiet difficult for someone who isn’t familiar with FreeBSD at all :-P.
>
>
>
> So I’ve retested again. Sadly it does still share the load over all configured CPUs (0-3) for RSS (there are 8 cores plus 8 HT cores per physical CPU). As I already mentioned, the new sysctl values are now visible, so I think the driver should be fine. My expectation with that configuration was, that CPU0 shouldn’t be used for RSS (as all values are smaller than the available CPUs), but it has been used, so the network traffic does flapping as hell.

Each rx queue gets its own interrupt and I verified with "vmstat -i"
that only non-RSS traffic (ARP) resulted in activity on queue 0, TCP
and UDP traffic were seen on other queues, as expected.  Note that rx
queue 0 does not necessarily mean CPU0.

> hw.cxgbe.cong_drop="1"
> hw.cxgbe.nrxq="3"
> hw.cxgbe.pause_settings="0"
> hw.cxgbe.rsrv_noflowq="1"
> hw.cxgbe.rsrv_norssq="1"
> hw.cxgbe.rx_budget="128"
> if_cxgbe_load="yes"

okay.

> net.inet.rss.bits="2"
> net.inet.rss.enabled="1"
> net.isr.bindthreads="1"
> net.isr.maxthreads="-1"

What are all these for?  Please go back to defaults and retry with
only the driver tunables above.  It looks like OPNsense ships with
"options RSS" in the kernel.  That complicates things a bit because I
tested my patch on FreeBSD GENERIC kernel which does not have this
option.  Run "top -SHIPzt" during the test to see exactly what is
hogging CPU0.

Regards,
Navdeep