Re: Chelsio NIC with RSS - Traffic distribution to different Queues

On 7/10/23 2:36 AM, Josef.Zahner1@swisscom.com wrote:
> Ok we are getting closer to a solution. But I’m a little bit confused, 
> I’ll try to explain why.
> 
> I’ve removed the following sysctl values, as you stated that they are 
> not official FreeBSD values:
> 
> -net.inet.rss.bits="2"
> -net.inet.rss.enabled="1"
> -net.isr.bindthreads="1"
> -net.isr.maxthreads="-1"
> 
> Without the setting above and without your proposed sysctl values, my 
> system was stable and the traffic distributed over all CPUs (even HT 
> cores). So no more drops on CARP if CPU0 has had high load. It looks 
> like my problem is clearly related to the “Custom” OPNsense RSS values 
> above. As the RSS sysctl values are noted in the OPNsense guide 
> (https://docs.opnsense.org/troubleshooting/performance.html 
> <https://docs.opnsense.org/troubleshooting/performance.html>), my 
> intention was, that those values needs to be set anyway for RSS. But it 
> seems that those 4 values are making the thing worse!

The defaults are defaults for a reason :-) and it's always a good idea 
to get a baseline measurement for your workload before any tuning.  It's 
not possible to assess the effect of any tuning without a baseline to 
compare with.  Looks like everything would have worked for you even with 
the OPNsense defaults although they are different from FreeBSD.

> Back to your topic. You are right, queue-0 does only control traffic if 
> I’m enabling your sysctl values. Where can I see which protocols are 
> falling into control traffic (queue 0)? So your change works as 
> expected, thanks a lot!

There is no direct way to observe the traffic on each queue.  Each rx 
queue happens to have its own interrupt so I looked at the interrupt 
counts (as shown by vmstat -i) and the queues' consumer index (sysctl 
dev.cxl.<port>.rxq.<qid>.cidx) to test my patch.

> 
> As you stated, queue 0 is never CPU 0, which is important to know. >
> I’m pretty sure that I needed the RSS values above to get RSS work on 
> intel NICs and with “netstat -Q” I could see the RSS queues. With 
> Chelsio it seems that the FreeBSD process “netisr” doesn’t do the RSS 
> stuff and “netstat -Q” shows nothing about RSS. For me this means that 
> your driver directly does the RSS magic, is this correct? Please help to 
> shed light on this topic.
All modern multiqueue NICs do RSS internally whether you have "options 
RSS" in the kernel or not.  "options RSS" enables extra code in the 
kernel that gives it some control over how RSS is configured the NICs 
and how many queues they should use.  The net.inet.rss.* sysctls show up 
only if you have "options RSS" in the kernel.  Note that cxgbe does 
support "options RSS" so I'm not sure why the output of "netstat -Q" 
wouldn't show that.  Was this the if_cxgbe.ko that you built yourself? 
You must use the same configuration as the kernel to build the module.

 > -net.isr.bindthreads="1"
 > -net.isr.maxthreads="-1"

This doesn't show the ISR dispatch policy (net.isr.dispatch) but 
whatever the policy is it applies to all the NICs in the system.  If 
you're comparing runs with different NICs please do it with the exact 
same configuration.  All NIC drivers call if_input (ether_input) to 
submit work to the kernel and ether_input uses netisr to dispatch the 
work.  The driver doesn't make netisr calls directly.

Regards,
Navdeep