From nobody Thu Jul 06 07:58:07 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QxTRd6YHCz4mCg6 for ; Thu, 6 Jul 2023 07:58:21 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-ej1-f43.google.com (mail-ej1-f43.google.com [209.85.218.43]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QxTRc2w8Fz3DPb for ; Thu, 6 Jul 2023 07:58:20 +0000 (UTC) (envelope-from nparhar@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of nparhar@gmail.com designates 209.85.218.43 as permitted sender) smtp.mailfrom=nparhar@gmail.com; dmarc=none Received: by mail-ej1-f43.google.com with SMTP id a640c23a62f3a-9891c73e0fbso72770066b.1 for ; Thu, 06 Jul 2023 00:58:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688630299; x=1691222299; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=76muZbzhDitsoJ4iHmnRsJe5yrdvHozKgjhi0QMVogk=; b=mEfTzxHTx0SgTX7o4ln7OOGgMb5RhnxgV6jA4qu2Wz4AtC6WL6Em7HAwuhM69KQ0pM qAZtNx6xcYC9cELhmR2HgCU2Tem9jbcB761F5J3Hwb1fjM9xg/mahtjQ3p/HFpQRo/Xz swFirx5tqvlQr4QM046wVPpPeTxgK7TD/W3axP0uoUo0Q/poORIK1Xhr0gisM2FlYrPa pl9o7gNglRbCwE2Jpkv0Q8kqcQFlIOuv6eUQV26jr6246msGBlDlTete6xfniAxjPP6c D9LgEgDORd5Z9oP7ZLy10Sg/2cv5BoS3o+zA4dw6Dkdk8Bk5Smnj2VnCtvZe5y2scFcm H+Mw== X-Gm-Message-State: ABy/qLbyxla9R708YbhlY09VizCw535TIZ9Nmo8/j/lZT19czPVsdQrY 1fBPeB1+oS7tC8gNVHovNkOI/aD3UQi5DOSIzgI= X-Google-Smtp-Source: APBJJlER7AZaJEwEJSazVQbLC/zI1iKRLeLfDvdD5o3FGcrS7dwcP26SB9W730rwvaqjmntX3dkAyvhFb2I3hR2m3O0= X-Received: by 2002:a17:906:73cb:b0:993:74b5:1359 with SMTP id n11-20020a17090673cb00b0099374b51359mr1239751ejl.27.1688630298716; Thu, 06 Jul 2023 00:58:18 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 References: <8fe00cbc-f218-a587-48d8-1612223ccd49@FreeBSD.org> In-Reply-To: From: Navdeep Parhar Date: Thu, 6 Jul 2023 00:58:07 -0700 Message-ID: Subject: Re: Chelsio NIC with RSS - Traffic distribution to different Queues To: Josef.Zahner1@swisscom.com Cc: freebsd-net@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.995]; FORGED_SENDER(0.30)[np@freebsd.org,nparhar@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.218.43:from]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_IN_DNSWL_NONE(0.00)[209.85.218.43:from]; RCVD_TLS_LAST(0.00)[]; FROM_NEQ_ENVFROM(0.00)[np@freebsd.org,nparhar@gmail.com]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4QxTRc2w8Fz3DPb X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On Mon, Jul 3, 2023 at 8:48=E2=80=AFAM wrote: > > Sorry for the spam, I do see the values with sysctl now. It seems that Fr= eeBSD always loads the if_cxgbe.ko from /boot/kernel/if_cxgbe.ko. So what I= =E2=80=99ve done is, I renamed the original file and copied the newly compi= led if_cxgbe.ko from /boot/module to /boot/kernel. Is there a cleaner way t= o get it work? Btw. Do I need t5fw_cfg.ko as well, I haven=E2=80=99t found = any documentation what exactly it does=E2=80=A6 > > > > Quiet difficult for someone who isn=E2=80=99t familiar with FreeBSD at al= l :-P. > > > > So I=E2=80=99ve retested again. Sadly it does still share the load over a= ll configured CPUs (0-3) for RSS (there are 8 cores plus 8 HT cores per phy= sical CPU). As I already mentioned, the new sysctl values are now visible, = so I think the driver should be fine. My expectation with that configuratio= n was, that CPU0 shouldn=E2=80=99t be used for RSS (as all values are small= er than the available CPUs), but it has been used, so the network traffic d= oes flapping as hell. Each rx queue gets its own interrupt and I verified with "vmstat -i" that only non-RSS traffic (ARP) resulted in activity on queue 0, TCP and UDP traffic were seen on other queues, as expected. Note that rx queue 0 does not necessarily mean CPU0. > hw.cxgbe.cong_drop=3D"1" > hw.cxgbe.nrxq=3D"3" > hw.cxgbe.pause_settings=3D"0" > hw.cxgbe.rsrv_noflowq=3D"1" > hw.cxgbe.rsrv_norssq=3D"1" > hw.cxgbe.rx_budget=3D"128" > if_cxgbe_load=3D"yes" okay. > net.inet.rss.bits=3D"2" > net.inet.rss.enabled=3D"1" > net.isr.bindthreads=3D"1" > net.isr.maxthreads=3D"-1" What are all these for? Please go back to defaults and retry with only the driver tunables above. It looks like OPNsense ships with "options RSS" in the kernel. That complicates things a bit because I tested my patch on FreeBSD GENERIC kernel which does not have this option. Run "top -SHIPzt" during the test to see exactly what is hogging CPU0. Regards, Navdeep