From nobody Mon May 08 01:46:38 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QF40G3mcwz49rjn for ; Mon, 8 May 2023 01:46:54 +0000 (UTC) (envelope-from ccfreebsd@gmail.com) Received: from mail-lf1-f41.google.com (mail-lf1-f41.google.com [209.85.167.41]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QF40F5QkZz4796 for ; Mon, 8 May 2023 01:46:53 +0000 (UTC) (envelope-from ccfreebsd@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of ccfreebsd@gmail.com designates 209.85.167.41 as permitted sender) smtp.mailfrom=ccfreebsd@gmail.com; dmarc=none Received: by mail-lf1-f41.google.com with SMTP id 2adb3069b0e04-4f139de8cefso25216998e87.0 for ; Sun, 07 May 2023 18:46:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683510412; x=1686102412; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=oiP64lcOSLuBZ6yIhj1OWf94EAcD3K+jZaWxU58bhvM=; b=L07Yj3MPRph2uo+znCllfLBbbEm5SFUYvRZbHjMnEqPWo2ghUGxISTBq1QX3FLZMYL 3X/fME4cZ8MofYwavz0qB21z91XHubXrIDMq8PHi1pRXvRel51uS/609oQpQ5qPvhSps 9nyfzXR0++sI7d+3SUF4//rX3/gG8diQJsLMtqjdz9UPP3o0Hb53lsTcezJKFFP8JELA c0MpUvk8gTkBR9G/rVu9UmN6wStXFuheOD4pptoZCITc5GNV5dC9ggy2QTu2oPiZ7kVi Aihx+hkfx4de60mS8QxCOqkBaaNpHNJgGn0eSlolgrljpmLR7GB1p3eO2jE6/8QS6w6B PIFA== X-Gm-Message-State: AC+VfDwRarr1292MvNXQyNfq7TFlb6+jDaGduHsZARGGPR94vXpI9ovu 8mFNrNJpkEyPXRJcm0mUy9IsHrmQXBF/RA== X-Google-Smtp-Source: ACHHUZ5NzT2M+5jIBf/JKkQlyPibEZz5d99AK/at0FzZqfk4YBeflqJdEiLPLcv28sCOez5t280P+w== X-Received: by 2002:a2e:b55a:0:b0:2a9:f9e0:a820 with SMTP id a26-20020a2eb55a000000b002a9f9e0a820mr2313621ljn.11.1683510411263; Sun, 07 May 2023 18:46:51 -0700 (PDT) Received: from mail-lf1-f41.google.com (mail-lf1-f41.google.com. [209.85.167.41]) by smtp.gmail.com with ESMTPSA id j14-20020a2eb70e000000b002a8a77f4d03sm1000099ljo.59.2023.05.07.18.46.50 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 07 May 2023 18:46:51 -0700 (PDT) Received: by mail-lf1-f41.google.com with SMTP id 2adb3069b0e04-4f139de8cefso25216940e87.0 for ; Sun, 07 May 2023 18:46:50 -0700 (PDT) X-Received: by 2002:a05:6512:281e:b0:4eb:1315:eaea with SMTP id cf30-20020a056512281e00b004eb1315eaeamr4812314lfb.3.1683510410329; Sun, 07 May 2023 18:46:50 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Cheng Cui Date: Sun, 7 May 2023 21:46:38 -0400 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: missing hw.em.msix support in FreeBSD 14.0-CURRENT? To: Yuri Cc: freebsd-net@freebsd.org Content-Type: multipart/alternative; boundary="000000000000fbc44d05fb24cfe7" X-Spamd-Result: default: False [0.19 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.990]; NEURAL_SPAM_SHORT(0.55)[0.548]; NEURAL_HAM_LONG(-0.36)[-0.364]; FORGED_SENDER(0.30)[cc@freebsd.org,ccfreebsd@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.167.41:from]; RCVD_IN_DNSWL_NONE(0.00)[209.85.167.41:from,209.85.167.41:received]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; R_DKIM_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_COUNT_THREE(0.00)[4]; FROM_NEQ_ENVFROM(0.00)[cc@freebsd.org,ccfreebsd@gmail.com]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[freebsd.org]; FREEMAIL_ENVFROM(0.00)[gmail.com] X-Rspamd-Queue-Id: 4QF40F5QkZz4796 X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N --000000000000fbc44d05fb24cfe7 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Found the RSS support, only if the hardware is ">=3D e1000_82571" from the if_em.c. if (hw->mac.type >=3D em_mac_min) // #define em_mac_min e1000_82571 > ... > scctx->isc_txrx =3D &em_txrx; > ... > > Best Regards, Cheng Cui On Sun, May 7, 2023 at 7:28=E2=80=AFPM Cheng Cui wrote: > Hi Yuri, > > Thanks for the update. I am using an Emulab testbed node under the same > type of hardware. Below are the new outputs. > It is supposed to have multiple tx/rx queues under msix. Why are there > only rxq0/txq0? Is it because the CPU has only one core? How can I check = if > RSS is supported? > > root@s1:~ # sysctl -d dev.em.0.iflib.disable_msix > dev.em.0.iflib.disable_msix: disable MSI-X (default 0) > root@s1:~ # sysctl dev.em.0.iflib.disable_msix > dev.em.0.iflib.disable_msix: 0 > root@s1:~ # sysctl dev.em.0.iflib > dev.em.0.iflib.rxq0.rxq_fl0.buf_size: 2048 > dev.em.0.iflib.rxq0.rxq_fl0.credits: 1023 > dev.em.0.iflib.rxq0.rxq_fl0.cidx: 959 > dev.em.0.iflib.rxq0.rxq_fl0.pidx: 958 > dev.em.0.iflib.rxq0.cpu: 1 > dev.em.0.iflib.txq0.r_abdications: 0 > dev.em.0.iflib.txq0.r_restarts: 0 > dev.em.0.iflib.txq0.r_stalls: 0 > dev.em.0.iflib.txq0.r_starts: 110443 > dev.em.0.iflib.txq0.r_drops: 0 > dev.em.0.iflib.txq0.r_enqueues: 110444 > dev.em.0.iflib.txq0.ring_state: pidx_head: 1903 pidx_tail: 1903 cidx: 190= 3 > state: IDLE > dev.em.0.iflib.txq0.txq_cleaned: 115208 > dev.em.0.iflib.txq0.txq_processed: 115251 > dev.em.0.iflib.txq0.txq_in_use: 42 > dev.em.0.iflib.txq0.txq_cidx_processed: 565 > dev.em.0.iflib.txq0.txq_cidx: 525 > dev.em.0.iflib.txq0.txq_pidx: 568 > dev.em.0.iflib.txq0.no_tx_dma_setup: 0 > dev.em.0.iflib.txq0.txd_encap_efbig: 0 > dev.em.0.iflib.txq0.tx_map_failed: 0 > dev.em.0.iflib.txq0.no_desc_avail: 0 > dev.em.0.iflib.txq0.mbuf_defrag_failed: 0 > dev.em.0.iflib.txq0.m_pullups: 0 > dev.em.0.iflib.txq0.mbuf_defrag: 0 > dev.em.0.iflib.txq0.cpu: 0 > dev.em.0.iflib.override_nrxds: 0 > dev.em.0.iflib.override_ntxds: 0 > dev.em.0.iflib.use_logical_cores: 0 > dev.em.0.iflib.separate_txrx: 0 > dev.em.0.iflib.core_offset: 0 > dev.em.0.iflib.tx_abdicate: 0 > dev.em.0.iflib.rx_budget: 0 > dev.em.0.iflib.disable_msix: 0 > dev.em.0.iflib.override_qs_enable: 0 > dev.em.0.iflib.override_nrxqs: 0 > dev.em.0.iflib.override_ntxqs: 0 > dev.em.0.iflib.driver_version: 7.7.8-fbsd > root@s1:~ # > > > Best Regards, > Cheng Cui > > > On Sun, May 7, 2023 at 4:18=E2=80=AFPM Yuri wrote: > >> Cheng Cui wrote: >> > Hello, >> > >> > I am using this em(4) driver for some tests. According to the man page >> > below, there should be a "hw.em.msix" sysctl to enable/disable MSI-X >> > style interrupts. But why is it missing in my system below? >> > https://man.freebsd.org/cgi/man.cgi?em(4) >> > >> > >> > root@s1:~ # dmesg | egrep "8254" >> > Timecounter "i8254" frequency 1193182 Hz quality 0 >> > Event timer "i8254" frequency 1193182 Hz quality 100 >> > em0: port 0xdcc0-0xdcff mem >> > 0xdfae0000-0xdfafffff irq 64 at device 7.0 on pci6 >> > em1: port 0xccc0-0xccff mem >> > 0xdf8e0000-0xdf8fffff irq 65 at device 8.0 on pci7 >> > em2: port 0xbcc0-0xbcff >> > mem 0xdf5e0000-0xdf5fffff irq 106 at device 4.0 on pci9 >> > em3: port 0xbc80-0xbcbf >> > mem 0xdf5c0000-0xdf5dffff irq 107 at device 4.1 on pci9 >> > em4: port 0xacc0-0xacff >> > mem 0xdf3e0000-0xdf3fffff irq 101 at device 3.0 on pci10 >> > em5: port 0xac80-0xacbf >> > mem 0xdf3c0000-0xdf3dffff irq 102 at device 3.1 on pci10 >> > >> > root@s1:~ # ifconfig em5 >> > em5: flags=3D8843 metric 0 mtu >> 1500 >> > >> options=3D481009b >> > ether 00:04:23:b7:1e:1d >> > inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255 >> > media: Ethernet 1000baseT >> > status: active >> > nd6 options=3D29 >> > >> > root@s1:~ # sysctl hw.em >> > hw.em.max_interrupt_rate: 8000 >> > hw.em.eee_setting: 1 >> > hw.em.rx_process_limit: 100 >> > hw.em.sbp: 0 >> > hw.em.smart_pwr_down: 0 >> > hw.em.rx_abs_int_delay: 66 >> > hw.em.tx_abs_int_delay: 66 >> > hw.em.rx_int_delay: 0 >> > hw.em.tx_int_delay: 66 >> > hw.em.disable_crc_stripping: 0 >> > root@s1:~ # >> >> As if_em was converted to iflib, I guess the related sysctl would now be >> dev.em.0.iflib.disable_msix (I could be wrong, don't have h/w to check). >> > --000000000000fbc44d05fb24cfe7 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Found the RSS support, only if the hardware is "= >=3D e1000_82571" from the if_em.c.

if (hw->mac.t=
ype >=3D em_mac_min)        // #define em_mac_min e1000_82571
    ...
    scctx->isc_txrx =3D &em_txrx;
    ...
<= br>
Best Regards,
Cheng Cui


On Su= n, May 7, 2023 at 7:28=E2=80=AFPM Cheng Cui <cc@freebsd.org> wrote:
Hi Yuri,

Thanks for the=C2=A0update. I am using an Emulab testbed node under the s= ame type of hardware. Below are the new outputs.
It is suppos= ed to have multiple tx/rx queues under msix. Why are there only rxq0/txq0? = Is it because the CPU has only one core? How can I check if RSS is supporte= d?

root@s1:~ # sysctl -d dev.em.0.iflib.disabl= e_msix
dev.em.0.iflib.disable_msix: disable MSI-X (default 0)
root@s1= :~ # sysctl dev.em.0.iflib.disable_msix
dev.em.0.iflib.disable_msix: 0root@s1:~ # sysctl dev.em.0.iflib
dev.em.0.iflib.rxq0.rxq_fl0.buf_size= : 2048
dev.em.0.iflib.rxq0.rxq_fl0.credits: 1023
dev.em.0.iflib.rxq0.= rxq_fl0.cidx: 959
dev.em.0.iflib.rxq0.rxq_fl0.pidx: 958
dev.em.0.ifli= b.rxq0.cpu: 1
dev.em.0.iflib.txq0.r_abdications: 0
dev.em.0.iflib.txq= 0.r_restarts: 0
dev.em.0.iflib.txq0.r_stalls: 0
dev.em.0.iflib.txq0.r= _starts: 110443
dev.em.0.iflib.txq0.r_drops: 0
dev.em.0.iflib.txq0.r_= enqueues: 110444
dev.em.0.iflib.txq0.ring_state: pidx_head: 1903 pidx_ta= il: 1903 cidx: 1903 state: IDLE
dev.em.0.iflib.txq0.txq_cleaned: 115208<= br>dev.em.0.iflib.txq0.txq_processed: 115251
dev.em.0.iflib.txq0.txq_in_= use: 42
dev.em.0.iflib.txq0.txq_cidx_processed: 565
dev.em.0.iflib.tx= q0.txq_cidx: 525
dev.em.0.iflib.txq0.txq_pidx: 568
dev.em.0.iflib.txq= 0.no_tx_dma_setup: 0
dev.em.0.iflib.txq0.txd_encap_efbig: 0
dev.em.0.= iflib.txq0.tx_map_failed: 0
dev.em.0.iflib.txq0.no_desc_avail: 0
dev.= em.0.iflib.txq0.mbuf_defrag_failed: 0
dev.em.0.iflib.txq0.m_pullups: 0dev.em.0.iflib.txq0.mbuf_defrag: 0
dev.em.0.iflib.txq0.cpu: 0
dev.e= m.0.iflib.override_nrxds: 0
dev.em.0.iflib.override_ntxds: 0
dev.em.0= .iflib.use_logical_cores: 0
dev.em.0.iflib.separate_txrx: 0
dev.em.0.= iflib.core_offset: 0
dev.em.0.iflib.tx_abdicate: 0
dev.em.0.iflib.rx_= budget: 0
dev.em.0.iflib.disable_msix: 0
dev.em.0.iflib.override_qs_e= nable: 0
dev.em.0.iflib.override_nrxqs: 0
dev.em.0.iflib.override_ntx= qs: 0
dev.em.0.iflib.driver_version: 7.7.8-fbsd
root@s1:~ #

=

Best Regards,
Cheng Cui


On Su= n, May 7, 2023 at 4:18=E2=80=AFPM Yuri <yuri@aetern.org> wrote:
Cheng Cui wrote:
> Hello,
>
> I am using this em(4) driver for some tests. According to the man page=
> below, there should be a "hw.em.msix" sysctl to enable/disab= le MSI-X
> style interrupts. But why is it missing in my system below?
> https://man.freebsd.org/cgi/man.cgi?em(4)
> <https://man.freebsd.org/cgi/man.cgi?em(4)><= br> >
> root@s1:~ # dmesg | egrep "8254"
> Timecounter "i8254" frequency 1193182 Hz quality 0
> Event timer "i8254" frequency 1193182 Hz quality 100
> em0: <Intel(R) Legacy PRO/1000 MT 82541GI> port 0xdcc0-0xdcff me= m
> 0xdfae0000-0xdfafffff irq 64 at device 7.0 on pci6
> em1: <Intel(R) Legacy PRO/1000 MT 82541GI> port 0xccc0-0xccff me= m
> 0xdf8e0000-0xdf8fffff irq 65 at device 8.0 on pci7
> em2: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xbcc0-= 0xbcff
> mem 0xdf5e0000-0xdf5fffff irq 106 at device 4.0 on pci9
> em3: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xbc80-= 0xbcbf
> mem 0xdf5c0000-0xdf5dffff irq 107 at device 4.1 on pci9
> em4: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xacc0-= 0xacff
> mem 0xdf3e0000-0xdf3fffff irq 101 at device 3.0 on pci10
> em5: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xac80-= 0xacbf
> mem 0xdf3c0000-0xdf3dffff irq 102 at device 3.1 on pci10
>
> root@s1:~ # ifconfig em5
> em5: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric= 0 mtu 1500
> options=3D481009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM= ,VLAN_HWFILTER,NOMAP>
> ether 00:04:23:b7:1e:1d
> inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255
> media: Ethernet 1000baseT <full-duplex>
> status: active
> nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>
> root@s1:~ # sysctl hw.em
> hw.em.max_interrupt_rate: 8000
> hw.em.eee_setting: 1
> hw.em.rx_process_limit: 100
> hw.em.sbp: 0
> hw.em.smart_pwr_down: 0
> hw.em.rx_abs_int_delay: 66
> hw.em.tx_abs_int_delay: 66
> hw.em.rx_int_delay: 0
> hw.em.tx_int_delay: 66
> hw.em.disable_crc_stripping: 0
> root@s1:~ #

As if_em was converted to iflib, I guess the related sysctl would now be dev.em.0.iflib.disable_msix (I could be wrong, don't have h/w to check)= .
--000000000000fbc44d05fb24cfe7--