From nobody Wed May 25 15:43:11 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 9CE3D1B44BF4 for ; Wed, 25 May 2022 15:43:29 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-vk1-xa32.google.com (mail-vk1-xa32.google.com [IPv6:2607:f8b0:4864:20::a32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4L7b283Yh2z4brP for ; Wed, 25 May 2022 15:43:28 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-vk1-xa32.google.com with SMTP id bs5so10080749vkb.4 for ; Wed, 25 May 2022 08:43:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=O8ovHhiikfTGACHr0z6AM4rU+9ZU3h67oUrzqoWjH5Q=; b=uuKCZOjA7cogrRG+vMIE5c9MKFYRH4pJfi9Y6/dOPnlLxBOQenZkLdUT+tVpMG8JKX PNiDGel1rSpsYhFdB1CyOXfWK6a/fCioMJufZ64piWtvcNQlpokrZXYfyR7/JfgHYa5z GlGNAexfNprw8/4qGr3bKv6nihw0QgpTZw0Lnm+K4Luh6vLxz64moeBEgeRHhmISEGsl Xc6S1g1Qc3/ZWQSX8hG/ol+zK2su0nvBYvAi5XLrNOgiAU65KOIrfHjTCdNom+UuzwnQ RepMP+AfMqhACE5zCREyQAyFXDaC2CbDmf/i88tZnHLlE//YwGHmvALuNRh3qRT5x2LG AxSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=O8ovHhiikfTGACHr0z6AM4rU+9ZU3h67oUrzqoWjH5Q=; b=n/q3yqFGQPPZAmrbh4Ox5NK/MmlbgWlrWrHGRWL1RUZF3VsHOv5iLsYBeqJP9dMhDD 2aZSmxBdhKBFYwKIUTgewaCSE3vTJ5ntYc3yVKafDtyogvtvw2NDnFYf32//+VAqRixV nOhzUfHSGplAlrWLlf7i/Kwn5L7e7ubwgWim+kapA7UI7CxNw9jN3Poef8DxKHnqrcL2 2rnv060m6FWcciYmzGvp2gp07P+FtMmeeciuOTewOtuG9Wjc+v7NFMbMGvSycw1Y5GRH BurWe8BkGwPUO9p7xTZyiodiNo8yBUkXMFN2iIJGdGAuWQyjLZFqnlJr4bH9AdZO1/BJ vxbQ== X-Gm-Message-State: AOAM532oEEiCujUJoC0Z0s7PhelJzx54snJl9Y0shFy2oU1JPZj40+Pf zWHrCX9ngMBaKWHMlXLqz/72AMd9xDTQygIMPGK2dQ== X-Google-Smtp-Source: ABdhPJwXgVQ4B04f5+GtkN3FFEYuT9MmV8so0y6HSz/YUzuBZjWshKNJ1XCdjeF8VFDW4FCyo9/CrFB/hBZX1lje42Y= X-Received: by 2002:a05:6122:2229:b0:32d:1642:b58b with SMTP id bb41-20020a056122222900b0032d1642b58bmr11940935vkb.27.1653493402414; Wed, 25 May 2022 08:43:22 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <20220525122529.t2kwfg2q65dfiyyt@host-ubertino-mac-88e9fe7361f5.eduroam.ssid.10net.amherst.edu> <20220526001715.4ffee96a@ws1.wobblyboot.net> In-Reply-To: From: Warner Losh Date: Wed, 25 May 2022 09:43:11 -0600 Message-ID: Subject: Re: nvme INVALID_FIELD in dmesg.boot To: matti k Cc: Alexander Motin , Matteo Riondato , FreeBSD Current , Jim Harris Content-Type: multipart/alternative; boundary="000000000000e3c1a105dfd7ee8b" X-Rspamd-Queue-Id: 4L7b283Yh2z4brP X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20210112.gappssmtp.com header.s=20210112 header.b=uuKCZOjA; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::a32) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20210112.gappssmtp.com:s=20210112]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20210112.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::a32:from]; MLMMJ_DEST(0.00)[freebsd-current]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; R_SPF_NA(0.00)[no SPF record]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N --000000000000e3c1a105dfd7ee8b Content-Type: text/plain; charset="UTF-8" Here's a patch that might fix it diff --git a/sys/dev/nvme/nvme_ctrlr.c b/sys/dev/nvme/nvme_ctrlr.c index 2c5d521ecaa1..72c511de3be8 100644 --- a/sys/dev/nvme/nvme_ctrlr.c +++ b/sys/dev/nvme/nvme_ctrlr.c @@ -854,8 +854,9 @@ nvme_ctrlr_configure_aer(struct nvme_controller *ctrlr) NVME_CRIT_WARN_ST_READ_ONLY | NVME_CRIT_WARN_ST_VOLATILE_MEMORY_BACKUP; if (ctrlr->cdata.ver >= NVME_REV(1, 2)) - ctrlr->async_event_config |= NVME_ASYNC_EVENT_NS_ATTRIBUTE | - NVME_ASYNC_EVENT_FW_ACTIVATE; + ctrlr->async_event_config |= + ctrlr->cdata.oaes & (NVME_ASYNC_EVENT_NS_ATTRIBUTE | + NVME_ASYNC_EVENT_FW_ACTIVATE); status.done = 0; nvme_ctrlr_cmd_get_feature(ctrlr, NVME_FEAT_TEMPERATURE_THRESHOLD, Warner On Wed, May 25, 2022 at 9:29 AM Warner Losh wrote: > > > On Wed, May 25, 2022 at 8:18 AM matti k wrote: > >> On Wed, 25 May 2022 09:58:54 -0400 >> Alexander Motin wrote: >> >> > On 25.05.2022 08:25, Matteo Riondato wrote: >> > > My dmesg.boot contains the following entries containing >> > > "INVALID_FIELD" about nvme (I use nda(4) for my nvme disks, with >> > > hw.nvme.use_nvd=0 in loader.conf): >> > > >> > > trismegistus ~ % grep -e 'nvme[0-9]\?' /var/run/dmesg.boot >> > > nvme0: mem 0xb8610000-0xb8613fff irq 40 at device >> > > 0.0 numa-domain 0 on pci7 >> > > nvme1: mem 0xb8510000-0xb8513fff irq 47 at device >> > > 0.0 numa-domain 0 on pci8 >> > > nvme2: mem 0xc5e10000-0xc5e13fff irq 48 at device >> > > 0.0 numa-domain 0 on pci10 >> > > nvme3: mem 0xc5d10000-0xc5d13fff irq 55 at device >> > > 0.0 numa-domain 0 on pci11 >> > > nvme0: SET FEATURES (09) sqid:0 cid:15 nsid:0 cdw10:0000000b >> > > cdw11:0000031f nvme0: INVALID_FIELD (00/02) sqid:0 cid:15 cdw0:0 >> > > nvme1: SET FEATURES (09) sqid:0 cid:15 nsid:0 cdw10:0000000b >> > > cdw11:0000031f nvme1: INVALID_FIELD (00/02) sqid:0 cid:15 cdw0:0 >> > > nvme2: SET FEATURES (09) sqid:0 cid:15 nsid:0 cdw10:0000000b >> > > cdw11:0000031f nvme2: INVALID_FIELD (00/02) sqid:0 cid:15 cdw0:0 >> > > nvme3: SET FEATURES (09) sqid:0 cid:15 nsid:0 cdw10:0000000b >> > > cdw11:0000031f nvme3: INVALID_FIELD (00/02) sqid:0 cid:15 cdw0:0 >> > > nda0 at nvme0 bus 0 scbus16 target 0 lun 1 >> > > nda0: nvme version 1.2 x4 (max x4) lanes PCIe Gen3 (max Gen3) link >> > > nda1 at nvme1 bus 0 scbus17 target 0 lun 1 >> > > nda1: nvme version 1.2 x4 (max x4) lanes PCIe Gen3 (max Gen3) link >> > > nda2 at nvme2 bus 0 scbus18 target 0 lun 1 >> > > nda2: nvme version 1.2 x4 (max x4) lanes PCIe Gen3 (max Gen3) link >> > > nda3 at nvme3 bus 0 scbus19 target 0 lun 1 >> > > nda3: nvme version 1.2 x4 (max x4) lanes PCIe Gen3 (max Gen3) link >> > > >> > > The disks seem to work fine, from what I can tell. >> > > >> > > Are the "INVALID_FIELD" messages harmless, or can they be avoided >> > > with some tuning, or maybe with some patch? >> > >> > Those messages mean that driver tried to enable certain types of >> > asynchronous events, but probably the hardware does not support some >> > of those. If you wish to experiment we could try to mask some of the >> > bits in nvme_ctrlr_configure_aer() function to find out which one >> > exactly, but for discontinued drives 4-5 years old it might not have >> > too much sense. It should not be critical unless you either overheat >> > them, or somehow else they fail and wish to report it. >> > >> >> I am intrigued to how you guru's know this, is it because you know >> the code well enough? >> > > SET FEATURES (opcode 9) feature 0xb is indeed async event configuration. > 0x31f is: > SMART WARNING for available spares (0x1) > SMART warning for temperature (0x2) > SMART WARNING for device reliability (0x4) > SMART WARNING for being read only (0x8) > SMART WARNING for volatile memory backup (0x10) > Namespace attribute change events (0x100) > Firmware activation events (0x200) > > I wonder which one of those it doesn't like. My reading of the standard > suggests that those > should always be supported for a 1.2 and later drive... Thought maybe with > the possible > exception of the volatile memory backup, so let me do some digging here... > > We can get the last two items from OAES field of the controller > identificaiton data. This is bytes 95:92, > which if I'm counting right is the last word on the 040: line in the > nvmecontrol identify -x nvmeX command: > > 040: 4e474e4b 30303150 000cca07 00230000 00010200 005b8d80 0030d400 > 00000100 > > ----------------------------------------------------------------------------------------------------------^^^^^^^^^ > > It looks like we don't currently test these bits before we add the last > two (we do it unconditionally > for >= 1.2, and maybe we should check these bits >= 1.2). > > Would you be able to test a fix for this? > > Warner > --000000000000e3c1a105dfd7ee8b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Here's a patch that might fix it

di= ff --git a/sys/dev/nvme/nvme_ctrlr.c b/sys/dev/nvme/nvme_ctrlr.c
index 2= c5d521ecaa1..72c511de3be8 100644
--- a/sys/dev/nvme/nvme_ctrlr.c
+++ = b/sys/dev/nvme/nvme_ctrlr.c
@@ -854,8 +854,9 @@ nvme_ctrlr_configure_aer= (struct nvme_controller *ctrlr)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 NVME_CRIT_WARN_ST_READ_ONLY |
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 NVME_CRIT_WARN_ST_VOLATILE_MEMORY_BACKUP;
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 if (ctrlr->cdata.ver >=3D NVME_REV(1, 2))
- =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ctrlr->async_event_config |=3D NVME_ASYN= C_EVENT_NS_ATTRIBUTE |
- =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 NVME_ASYNC_EVENT_FW_ACTIVATE;
+ =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 ctrlr->async_event_config |=3D
+ =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ctrlr->cdata.oae= s & (NVME_ASYNC_EVENT_NS_ATTRIBUTE |
+ =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 NVME_ASYNC_EVENT_FW_ACTIVA= TE);

=C2=A0 =C2=A0 =C2=A0 =C2=A0 status.done =3D 0;
=C2=A0 =C2=A0= =C2=A0 =C2=A0 nvme_ctrlr_cmd_get_feature(ctrlr, NVME_FEAT_TEMPERATURE_THRE= SHOLD,

Warner

On Wed, May 25, 2022 at 9:2= 9 AM Warner Losh <imp@bsdimp.com&g= t; wrote:


On Wed, May 25, 2022 at 8:18 AM matti k <= ;mattik@gwsit.com.= au> wrote: