From nobody Sun May 07 23:28:34 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QF0wy2Nlmz49hK3 for ; Sun, 7 May 2023 23:28:50 +0000 (UTC) (envelope-from ccfreebsd@gmail.com) Received: from mail-lj1-f175.google.com (mail-lj1-f175.google.com [209.85.208.175]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QF0wx5SY5z3rjk for ; Sun, 7 May 2023 23:28:49 +0000 (UTC) (envelope-from ccfreebsd@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lj1-f175.google.com with SMTP id 38308e7fff4ca-2ac87e7806aso31131021fa.3 for ; Sun, 07 May 2023 16:28:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683502127; x=1686094127; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=8B2haAXDRaYBNZls8AjPVp0P3aVRVRHVKAkndw58gAw=; b=Td/5i9sN2fIR0bKIrg2+4/570y67AkcUUUu4A+mSOctEfc0JfhsDT1KgiHdr2YeTqz +rvKJY15/m+vixjYmlnyA4FLQnLSd1X5OJhLf2BmAbqlmaKpq1/ekgSZQx3LcKIN6pzc Lp7UxCSCrIyyOTjHIv7ddsge9oayxm7f0pFhMe2WpBtIk25smM1VumCIZjFgjF4OfpMD 0iwWnV2IwVwTXbI5LXlfHeHdVf0VMea6Xettdi0FNQVla/CIGAz7zsAOK8238LOWpI6F JMHztrQwJT7mGgpD9xXr59Jhk1dzuz+dngOjMMXbzHu1rZC9SgVw3NRP2l4Pm9DNArlk IfNQ== X-Gm-Message-State: AC+VfDzkxKTNsVM/fSVHoFY5ggAUgJsIRexFZr43v9MAAyDAfP5eIZc3 HQicX4+ox9bN1QRqAtC0/MV4dazJKuDR5g== X-Google-Smtp-Source: ACHHUZ4+hCK7DmgNTHZfvnmRwELVA6yV+cu8/eUCZSBY3RU255kMCkrwkWNazpQt4nF3XoeMEwH2DA== X-Received: by 2002:a2e:b6c6:0:b0:2ab:4399:708b with SMTP id m6-20020a2eb6c6000000b002ab4399708bmr2254969ljo.40.1683502127309; Sun, 07 May 2023 16:28:47 -0700 (PDT) Received: from mail-lj1-f174.google.com (mail-lj1-f174.google.com. [209.85.208.174]) by smtp.gmail.com with ESMTPSA id a16-20020a2eb170000000b002a8adb2b2d6sm961925ljm.128.2023.05.07.16.28.46 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 07 May 2023 16:28:47 -0700 (PDT) Received: by mail-lj1-f174.google.com with SMTP id 38308e7fff4ca-2ac8091351eso42500181fa.2 for ; Sun, 07 May 2023 16:28:46 -0700 (PDT) X-Received: by 2002:a19:f804:0:b0:4ec:8816:f4fc with SMTP id a4-20020a19f804000000b004ec8816f4fcmr2080201lff.6.1683502126275; Sun, 07 May 2023 16:28:46 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Cheng Cui Date: Sun, 7 May 2023 19:28:34 -0400 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: missing hw.em.msix support in FreeBSD 14.0-CURRENT? To: Yuri Cc: freebsd-net@freebsd.org Content-Type: multipart/alternative; boundary="0000000000003723e605fb22e2d3" X-Rspamd-Queue-Id: 4QF0wx5SY5z3rjk X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --0000000000003723e605fb22e2d3 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Yuri, Thanks for the update. I am using an Emulab testbed node under the same type of hardware. Below are the new outputs. It is supposed to have multiple tx/rx queues under msix. Why are there only rxq0/txq0? Is it because the CPU has only one core? How can I check if RSS is supported? root@s1:~ # sysctl -d dev.em.0.iflib.disable_msix dev.em.0.iflib.disable_msix: disable MSI-X (default 0) root@s1:~ # sysctl dev.em.0.iflib.disable_msix dev.em.0.iflib.disable_msix: 0 root@s1:~ # sysctl dev.em.0.iflib dev.em.0.iflib.rxq0.rxq_fl0.buf_size: 2048 dev.em.0.iflib.rxq0.rxq_fl0.credits: 1023 dev.em.0.iflib.rxq0.rxq_fl0.cidx: 959 dev.em.0.iflib.rxq0.rxq_fl0.pidx: 958 dev.em.0.iflib.rxq0.cpu: 1 dev.em.0.iflib.txq0.r_abdications: 0 dev.em.0.iflib.txq0.r_restarts: 0 dev.em.0.iflib.txq0.r_stalls: 0 dev.em.0.iflib.txq0.r_starts: 110443 dev.em.0.iflib.txq0.r_drops: 0 dev.em.0.iflib.txq0.r_enqueues: 110444 dev.em.0.iflib.txq0.ring_state: pidx_head: 1903 pidx_tail: 1903 cidx: 1903 state: IDLE dev.em.0.iflib.txq0.txq_cleaned: 115208 dev.em.0.iflib.txq0.txq_processed: 115251 dev.em.0.iflib.txq0.txq_in_use: 42 dev.em.0.iflib.txq0.txq_cidx_processed: 565 dev.em.0.iflib.txq0.txq_cidx: 525 dev.em.0.iflib.txq0.txq_pidx: 568 dev.em.0.iflib.txq0.no_tx_dma_setup: 0 dev.em.0.iflib.txq0.txd_encap_efbig: 0 dev.em.0.iflib.txq0.tx_map_failed: 0 dev.em.0.iflib.txq0.no_desc_avail: 0 dev.em.0.iflib.txq0.mbuf_defrag_failed: 0 dev.em.0.iflib.txq0.m_pullups: 0 dev.em.0.iflib.txq0.mbuf_defrag: 0 dev.em.0.iflib.txq0.cpu: 0 dev.em.0.iflib.override_nrxds: 0 dev.em.0.iflib.override_ntxds: 0 dev.em.0.iflib.use_logical_cores: 0 dev.em.0.iflib.separate_txrx: 0 dev.em.0.iflib.core_offset: 0 dev.em.0.iflib.tx_abdicate: 0 dev.em.0.iflib.rx_budget: 0 dev.em.0.iflib.disable_msix: 0 dev.em.0.iflib.override_qs_enable: 0 dev.em.0.iflib.override_nrxqs: 0 dev.em.0.iflib.override_ntxqs: 0 dev.em.0.iflib.driver_version: 7.7.8-fbsd root@s1:~ # Best Regards, Cheng Cui On Sun, May 7, 2023 at 4:18=E2=80=AFPM Yuri wrote: > Cheng Cui wrote: > > Hello, > > > > I am using this em(4) driver for some tests. According to the man page > > below, there should be a "hw.em.msix" sysctl to enable/disable MSI-X > > style interrupts. But why is it missing in my system below? > > https://man.freebsd.org/cgi/man.cgi?em(4) > > > > > > root@s1:~ # dmesg | egrep "8254" > > Timecounter "i8254" frequency 1193182 Hz quality 0 > > Event timer "i8254" frequency 1193182 Hz quality 100 > > em0: port 0xdcc0-0xdcff mem > > 0xdfae0000-0xdfafffff irq 64 at device 7.0 on pci6 > > em1: port 0xccc0-0xccff mem > > 0xdf8e0000-0xdf8fffff irq 65 at device 8.0 on pci7 > > em2: port 0xbcc0-0xbcff > > mem 0xdf5e0000-0xdf5fffff irq 106 at device 4.0 on pci9 > > em3: port 0xbc80-0xbcbf > > mem 0xdf5c0000-0xdf5dffff irq 107 at device 4.1 on pci9 > > em4: port 0xacc0-0xacff > > mem 0xdf3e0000-0xdf3fffff irq 101 at device 3.0 on pci10 > > em5: port 0xac80-0xacbf > > mem 0xdf3c0000-0xdf3dffff irq 102 at device 3.1 on pci10 > > > > root@s1:~ # ifconfig em5 > > em5: flags=3D8843 metric 0 mtu = 1500 > > > options=3D481009b > > ether 00:04:23:b7:1e:1d > > inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255 > > media: Ethernet 1000baseT > > status: active > > nd6 options=3D29 > > > > root@s1:~ # sysctl hw.em > > hw.em.max_interrupt_rate: 8000 > > hw.em.eee_setting: 1 > > hw.em.rx_process_limit: 100 > > hw.em.sbp: 0 > > hw.em.smart_pwr_down: 0 > > hw.em.rx_abs_int_delay: 66 > > hw.em.tx_abs_int_delay: 66 > > hw.em.rx_int_delay: 0 > > hw.em.tx_int_delay: 66 > > hw.em.disable_crc_stripping: 0 > > root@s1:~ # > > As if_em was converted to iflib, I guess the related sysctl would now be > dev.em.0.iflib.disable_msix (I could be wrong, don't have h/w to check). > --0000000000003723e605fb22e2d3 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Yuri,

Thanks for the=C2= =A0update. I am using an Emulab testbed node under the same type of hardwar= e. Below are the new outputs.
It is supposed to have multiple= tx/rx queues under msix. Why are there only rxq0/txq0? Is it because the C= PU has only one core? How can I check if RSS is supported?
root@s1:~ # sysctl -d dev.em.0.iflib.disable_msix
dev.em.0.= iflib.disable_msix: disable MSI-X (default 0)
root@s1:~ # sysctl dev.em.= 0.iflib.disable_msix
dev.em.0.iflib.disable_msix: 0
root@s1:~ # sysct= l dev.em.0.iflib
dev.em.0.iflib.rxq0.rxq_fl0.buf_size: 2048
dev.em.0.= iflib.rxq0.rxq_fl0.credits: 1023
dev.em.0.iflib.rxq0.rxq_fl0.cidx: 959dev.em.0.iflib.rxq0.rxq_fl0.pidx: 958
dev.em.0.iflib.rxq0.cpu: 1
de= v.em.0.iflib.txq0.r_abdications: 0
dev.em.0.iflib.txq0.r_restarts: 0
= dev.em.0.iflib.txq0.r_stalls: 0
dev.em.0.iflib.txq0.r_starts: 110443
= dev.em.0.iflib.txq0.r_drops: 0
dev.em.0.iflib.txq0.r_enqueues: 110444dev.em.0.iflib.txq0.ring_state: pidx_head: 1903 pidx_tail: 1903 cidx: 1903= state: IDLE
dev.em.0.iflib.txq0.txq_cleaned: 115208
dev.em.0.iflib.t= xq0.txq_processed: 115251
dev.em.0.iflib.txq0.txq_in_use: 42
dev.em.0= .iflib.txq0.txq_cidx_processed: 565
dev.em.0.iflib.txq0.txq_cidx: 525dev.em.0.iflib.txq0.txq_pidx: 568
dev.em.0.iflib.txq0.no_tx_dma_setup: = 0
dev.em.0.iflib.txq0.txd_encap_efbig: 0
dev.em.0.iflib.txq0.tx_map_f= ailed: 0
dev.em.0.iflib.txq0.no_desc_avail: 0
dev.em.0.iflib.txq0.mbu= f_defrag_failed: 0
dev.em.0.iflib.txq0.m_pullups: 0
dev.em.0.iflib.tx= q0.mbuf_defrag: 0
dev.em.0.iflib.txq0.cpu: 0
dev.em.0.iflib.override_= nrxds: 0
dev.em.0.iflib.override_ntxds: 0
dev.em.0.iflib.use_logical_= cores: 0
dev.em.0.iflib.separate_txrx: 0
dev.em.0.iflib.core_offset: = 0
dev.em.0.iflib.tx_abdicate: 0
dev.em.0.iflib.rx_budget: 0
dev.em= .0.iflib.disable_msix: 0
dev.em.0.iflib.override_qs_enable: 0
dev.em.= 0.iflib.override_nrxqs: 0
dev.em.0.iflib.override_ntxqs: 0
dev.em.0.i= flib.driver_version: 7.7.8-fbsd
root@s1:~ #
<= br clear=3D"all">

Best Regards,
Cheng Cui
=

On Sun, May 7, 2023 at 4:18=E2=80=AFPM Yuri <yuri@aetern.org> wrote:
Cheng Cui wrote:
> Hello,
>
> I am using this em(4) driver for some tests. According to the man page=
> below, there should be a "hw.em.msix" sysctl to enable/disab= le MSI-X
> style interrupts. But why is it missing in my system below?
> https://man.freebsd.org/cgi/man.cgi?em(4)
> <https://man.freebsd.org/cgi/man.cgi?em(4)><= br> >
> root@s1:~ # dmesg | egrep "8254"
> Timecounter "i8254" frequency 1193182 Hz quality 0
> Event timer "i8254" frequency 1193182 Hz quality 100
> em0: <Intel(R) Legacy PRO/1000 MT 82541GI> port 0xdcc0-0xdcff me= m
> 0xdfae0000-0xdfafffff irq 64 at device 7.0 on pci6
> em1: <Intel(R) Legacy PRO/1000 MT 82541GI> port 0xccc0-0xccff me= m
> 0xdf8e0000-0xdf8fffff irq 65 at device 8.0 on pci7
> em2: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xbcc0-= 0xbcff
> mem 0xdf5e0000-0xdf5fffff irq 106 at device 4.0 on pci9
> em3: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xbc80-= 0xbcbf
> mem 0xdf5c0000-0xdf5dffff irq 107 at device 4.1 on pci9
> em4: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xacc0-= 0xacff
> mem 0xdf3e0000-0xdf3fffff irq 101 at device 3.0 on pci10
> em5: <Intel(R) Legacy PRO/1000 MT 82546EB (Copper)> port 0xac80-= 0xacbf
> mem 0xdf3c0000-0xdf3dffff irq 102 at device 3.1 on pci10
>
> root@s1:~ # ifconfig em5
> em5: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric= 0 mtu 1500
> options=3D481009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM= ,VLAN_HWFILTER,NOMAP>
> ether 00:04:23:b7:1e:1d
> inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255
> media: Ethernet 1000baseT <full-duplex>
> status: active
> nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>
> root@s1:~ # sysctl hw.em
> hw.em.max_interrupt_rate: 8000
> hw.em.eee_setting: 1
> hw.em.rx_process_limit: 100
> hw.em.sbp: 0
> hw.em.smart_pwr_down: 0
> hw.em.rx_abs_int_delay: 66
> hw.em.tx_abs_int_delay: 66
> hw.em.rx_int_delay: 0
> hw.em.tx_int_delay: 66
> hw.em.disable_crc_stripping: 0
> root@s1:~ #

As if_em was converted to iflib, I guess the related sysctl would now be dev.em.0.iflib.disable_msix (I could be wrong, don't have h/w to check)= .
--0000000000003723e605fb22e2d3--