Re: Chelsio NIC with RSS - Traffic distribution to different Queues

Hello,

Please try this patch: https://people.freebsd.org/~np/norssq.diff

It adds these sysctls to the driver.
1) hw.cxgbe.rsrv_norssq.  This is what you originally asked for.
2) hw.cxgbe.rx_budget.  This can be used to force the driver's RX to
yield periodically.

What kind of system (cores, memory, etc.) is this?  Control packets
are either getting dropped or the threads/timers responsible for
sending or processing these packets are starved of CPU.  It would be
useful to monitor interface activity with "netstat -d -I <ifnet>"
during the test.

# sysctl hw.model hw.ncpu hw.physmem
# netstat -dw1 -I cxl0

Try the settings listed below.  nrxq=X might help in case the driver
RX threads are hogging all the cores because all rx queues are heavily
loaded.  Set nrxq to something less than the number of cores in the
system.  rx_budget can be changed any time (try 64, 128, 256) and
might improve the responsiveness of the rest of the system during
load.

(in loader.conf)
hw.cxgbe.nrxq=2                 (3 if you've patched the kernel and set norssq)
hw.cxgbe.rsrv_noflowq=1
hw.cxgbe.pause_settings=0
hw.cxgbe.cong_drop=1            (2 would be better but needs a recent driver)
hw.cxgbe.rsrv_norssq=1          (needs patch)
hw.cxgbe.rx_budget=128          (needs patch)

Let us know how it goes.

Regards,
Navdeep

On Thu, Jun 29, 2023 at 5:53 AM <Josef.Zahner1@swisscom.com> wrote:
>
> Can you tell me which netstat command you have in mind? I tried “netstat -Q”, it shows a few drops but not that much that it would explain the CARP drops. What I can tell you is, that especially CARP on the corresponding server is just sending out packets as long as it is the master box and CPU0 load is below 100%. It doesn’t receive any CARP traffic at all, just normal network traffic. What I see is, that those CARP packets are not sent anymore if CPU0 has 100% load -> if that happens the server switches to standby and the traffic is gone on the machine. So because of this behavior we would like to have an option, which allows us to have Control Plane Traffic (LACP, CARP,…?) in RSS RX queue 0 and nothing else. Question is, what would Control Plane traffic be. Hopefully as well CARP/VRRP,…
>
> We tried hw.cxgbe.cong_drops=1, but it doesn’t help in our case.
>
> Can you explain a bit what your patch will do? Am I right that you will post the link later on here?
>
> Cheers Josef