From nobody Sun Mar 20 10:43:58 2022 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3EEFB1A23EB7 for ; Sun, 20 Mar 2022 10:44:36 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-yw1-x1134.google.com (mail-yw1-x1134.google.com [IPv6:2607:f8b0:4864:20::1134]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KLvWl4B6Zz3n3G for ; Sun, 20 Mar 2022 10:44:35 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: by mail-yw1-x1134.google.com with SMTP id 00721157ae682-2e62c790cbdso4232977b3.0 for ; Sun, 20 Mar 2022 03:44:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=WGkDvo4dfZ7GwPSu5MOAFFD+CbohjIERt4TxU2IN9gk=; b=YZKXcUGFSJKjqtD486EiINPRTZ94LE4LvDYHWskJzQ51HX1Y/LQKaIIFs4jrzOHvV2 m9Kl/aJPopJEGLubIRTI6nlkfHH3eWx0PerN3Ve7bzmuWBvg2RTdDEHQp354E0jAb1+G K57snxUNx0l0EhuITmrIxcDi2L3hELEQ9K+9oRGPFrRJrzI5CrSFyWC4om2ITsjpnfqY iVbPlNrAcb4irNenFPuQtee94dTSbHiHN2TIKssljyNgyGrcnvXbp3oUYrji/89ys3b7 Ck30TSx3d+pKrKyYrdJQm/uodHJKdoCEOZai7QfaC5IymPJZWmMGE1i+JSOumNXAwudd JMRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=WGkDvo4dfZ7GwPSu5MOAFFD+CbohjIERt4TxU2IN9gk=; b=ZCIFncvABpnXuMyqBtQ87g+lgM8lY9/4JSyCvXCksZLTSsOAlEBA+OiWVAGE3spgNx WpWqcefpD+DFNKY0sGQa8PiPBDd1+wxLJkhC2nyyvMy1i+2PbfkifQL0DE5dykPnQv05 KHRwiS+bRgBt60Z77amQvdYp2iUlUTVRk1itKOJ653BNHl9eeUx5hTF+C3vUeyCEvqbo iUIv1/3wSm0L+MUrZM8wRZNIVASYIUSeHgy+0l59AiGCgPwJKEuyRgsfQAkmL2j3mSmc g3xC2aMl1G5mkrKUNPvx/KxU2NITw4n4qx9pJ/aO02QZ8VLHhwj4Jk8YPIB3qiXJxV9P 0IGg== X-Gm-Message-State: AOAM533N8dZJHQLduqbRC6qAZdHcU4W4Zs/JMzTu/heABD0xSB3vWKLe dIYGbeY+uJrVQ30C9B/qGdaJrCHl1KbWUgUQt5RtcaQ42VUPBw== X-Google-Smtp-Source: ABdhPJx1jho/gH4j4Hf53atBYF6ZoldWpYFgRNGTixWdLGs6RIeDP2/GTIsnWSV5ZHGnOfjZA7n4+kwuFDo2hFrMQJ4= X-Received: by 2002:a81:4643:0:b0:2e2:af31:13c2 with SMTP id t64-20020a814643000000b002e2af3113c2mr19301518ywa.274.1647773074823; Sun, 20 Mar 2022 03:44:34 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <00bf01d80104$e6ba5de0$b42f19a0$@tubnor.net> <082b01d80697$64e95030$2ebbf090$@tubnor.net> In-Reply-To: <082b01d80697$64e95030$2ebbf090$@tubnor.net> From: Mario Marietto Date: Sun, 20 Mar 2022 11:43:58 +0100 Message-ID: Subject: Re: bhyve NVMe 1.4 support To: jason@tubnor.net Cc: Chuck Tuffli , FreeBSD virtualization Content-Type: multipart/alternative; boundary="000000000000cb9fe205daa41016" X-Rspamd-Queue-Id: 4KLvWl4B6Zz3n3G X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=YZKXcUGF; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::1134 as permitted sender) smtp.mailfrom=marietto2008@gmail.com X-Spamd-Result: default: False [-1.96 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[1]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.96)[-0.958]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1134:from]; HTTP_TO_IP(1.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --000000000000cb9fe205daa41016 Content-Type: text/plain; charset="UTF-8" Please check this situation : (I'm on : FreeBSD marietto 13.0-RELEASE-p8 FreeBSD 13.0-RELEASE-p8 #6 n244863-45b90a014c11: Thu Mar 17 18:42:4 0 CET 2022 marietto@marietto:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64) ubuntu21-10-nvidia495-vm0.sh : bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H -A \ -s 0,hostbridge \ -s 1,virtio-blk,/mnt/$vmdisk'p2'/bhyve/os/Linux/impish-cuda-11-5-nvidia-495.img \ -s 2,nvme,/dev/nvd0 \ -s 3,passthru,5/0/0 \ -s 4,passthru,1/0/0 \ -s 8,virtio-net,tap0 \ -s 9,virtio-9p,sharename=/ \ -s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900 \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ vm0 < /dev/null & sleep 2 && vncviewer 0:0 on the Ubuntu 21.10 guest os : mario@marietto-BHYVE:/home/marietto# fdisk -l Disk /dev/nvme0n1: 931,51 GiB, 1000204886016 bytes, 1953525168 sectors Disk model: bhyve-NVMe Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes The nvme disk is not recognized. This is how it looks on FreeBSD : 1. Name: nvd0 Mediasize: 1000204886016 (932G) Sectorsize: 512 Mode: r0w0e0 descr: CT1000P1SSD8 lunid: 000000000000000100a07520285f1175 ident: 2022285F1175 rotationrate: 0 fwsectors: 0 fwheads: 0 => 34 1953525101 nvd0 GPT (932G) 34 2014 - free - (1.0M) 2048 1748992 1 efi (854M) 1751040 921985024 2 ms-basic-data (440G) 923736064 191522816 - free - (91G) 1115258880 833185547 7 ms-basic-data (397G) 1948444427 245 - free - (123K) 1948444672 1318912 3 ms-recovery (644M) 1949763584 2048 - free - (1.0M) 1949765632 1310720 4 ms-recovery (640M) 1951076352 2048 - free - (1.0M) 1951078400 1265657 5 ms-basic-data (618M) 1952344057 7 - free - (3.5K) 1952344064 1179641 6 ms-basic-data (576M) 1953523705 1430 - free - (715K) Il giorno mar 11 gen 2022 alle ore 04:01 ha scritto: > > -----Original Message----- > > From: Chuck Tuffli > > Sent: Tuesday, 11 January 2022 10:58 AM > > To: jason@tubnor.net > > Cc: FreeBSD virtualization > > Subject: Re: bhyve NVMe 1.4 support > > > > On Wed, Jan 5, 2022 at 4:33 PM Chuck Tuffli wrote: > > > > > > On Mon, Jan 3, 2022 at 4:49 PM wrote: > > > > > > > > Hi Chuck, > > > > > > > > Testing on 14.0-CURRENT shows the following: > > > > > > > > OpenBSD 6.9 - NVMe read/write OK > > > > Windows 10 - NVMe read OK, write FAIL Windows Server 2022 - NVMe > > > > read OK, write FAIL Alma Linux 8.5 - NVMe read OK, write FAIL. > > > > > > Thanks for the report, Jason. I have Alma 8.5 installed and am > investigating. > > > > OK, I found and fixed the regression. New version of the file in the same > > place is up for folks who are interested. > > > > --chuck > > This version looks good. No regression on the following guest platforms > under 14.0-CURRENT: > > OpenBSD 6.9 - NVMe read/write OK > Windows 10 - NVMe read/write OK > Windows Server 2022 - NVMe read/write OK > Alma Linux 8.5 - NVMe read/write OK > FreeBSD 13.0 - NVMe read/write OK > > From a storage presentation layer in a production perspective, I can't > find any issues with the proposed update to pci_nvme.c > > Some of our tests included: > > The removal and addition of partitions on the NVMe presentation layer > Installation of all the operating system listed above > Where supported by the guest operating system, the TRIM command executed > on the presentation layer and guest rebooted to ensure no data corruption > was caused by TRIM > > Cheers, > > Jason. > > > -- Mario. --000000000000cb9fe205daa41016 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Please check this situation :

(I'm on :

FreeBSD marietto 13.0-RELEASE-p8 FreeBSD 13.0-RELEASE-p8 #6 n244863-45b90a= 014c11: Thu Mar 17 18:42:4
0 CET 2022 =C2=A0=C2=A0=C2=A0=C2=A0mar= ietto@marietto:/usr/obj/usr/src/amd64.amd64/sys/GENERIC =C2=A0amd64)


ubuntu21-10-nvidia495-vm0.sh :=
=09 =09
=09 

bhyve -S -c sockets=
=3D2,cores=3D2,threads=3D2 -m 8G -w -H -A \
-s 0,hostbridge \
-s 1,virtio-blk,/mnt/$vmdisk'p2'/bhyve/os/Linux/impish-cuda-11-5-nv=
idia-495.img \
-s 2,nvme,/dev/nvd0 \
-s 3,passthru,5/0/0 \
-s 4,passthru,1/0/0 \
-s 8,virtio-net,tap0 \
-s 9,virtio-9p,sharename=3D/ \
-s 29,fbuf,tcp=3D0.0.0.0:5900,w=3D1440,=
h=3D900 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0 < /dev/null & sleep 2 && vncviewer 0:0

on the Ubuntu 21.10 guest os :

=09 =09
=09 
mario@marietto-BHYVE:/h=
ome/marietto# fdisk -l

Disk /dev/nvme0n1: 931,51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: bhyve-NVMe                           =20
Units: sectors of 1 * 512 =3D 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

The nvme disk is not recognized. This is how it looks on FreeBSD :

1. Name: nvd0
=C2=A0=C2=A0Mediasize: 1000204886016 (932G)
=C2=A0=C2=A0Sectorsize: 512
=C2=A0=C2=A0Mode: r0w0e0
=C2=A0=C2=A0descr: CT1000P1SSD8
=C2=A0=C2=A0lunid: 000000000000000100a07520285f1175
=C2=A0=C2=A0ident: 2022285F1175
=C2=A0=C2=A0rotationrate: 0
=C2=A0=C2=A0fwsectors: 0
=C2=A0=C2=A0fwheads: 0

=3D> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A034 =C2=A01953525101 =C2=A0nvd0 =C2=A0GPT =C2=A0(932G)=C2=A0

=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 34 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A020= 14 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0- free - =C2=A0(1.0M)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02048 =C2=A0=C2=A0=C2=A0=C2= =A01748992 =C2=A0=C2=A0=C2=A0=C2=A01 =C2=A0efi =C2=A0(854M)
=C2=A0=C2=A0=C2=A0=C2=A01751040 =C2=A0=C2=A0921985024 =C2=A0=C2=A0=C2= =A0=C2=A02 =C2=A0ms-basic-data =C2=A0(440G)
=C2=A0=C2=A0923736064 =C2=A0=C2=A0191522816 =C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0- free - =C2=A0(91G)
=C2=A01115258880 =C2=A0=C2=A0833185547 =C2=A0=C2=A0=C2=A0=C2=A07 =C2= =A0ms-basic-data =C2=A0(397G)
=C2=A01948444427 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0245 = =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0- free - =C2=A0(123K)
=C2=A01948444672 =C2=A0=C2=A0=C2=A0=C2=A01318912 =C2=A0=C2=A0=C2=A0=C2= =A03 =C2=A0ms-recovery =C2=A0(644M)
=C2=A01949763584 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02048 =C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0- free - =C2=A0(1.0M)
=C2=A01949765632 =C2=A0=C2=A0=C2=A0=C2=A01310720 =C2=A0=C2=A0=C2=A0=C2= =A04 =C2=A0ms-recovery =C2=A0(640M)
=C2=A01951076352 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02048 =C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0- free - =C2=A0(1.0M)
=C2=A01951078400 =C2=A0=C2=A0=C2=A0=C2=A01265657 =C2=A0=C2=A0=C2=A0=C2= =A05 =C2=A0ms-basic-data =C2=A0(618M)
=C2=A01952344057 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A07 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0- free - =C2=A0(3.5K)
=C2=A01952344064 =C2=A0=C2=A0=C2=A0=C2=A01179641 =C2=A0=C2=A0=C2=A0=C2= =A06 =C2=A0ms-basic-data =C2=A0(576M)
=C2=A01953523705 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A01430 =C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0- free - =C2=A0(715K)




Il giorno mar 11 gen 2022 alle ore 04:01 &l= t;jason@tubnor.net> ha scritto:<= br>
> -----Origin= al Message-----
> From: Chuck Tuffli <chuck@tuffli.net>
> Sent: Tuesday, 11 January 2022 10:58 AM
> To: jason@tubnor= .net
> Cc: FreeBSD virtualization <freebsd-virtualization@freebsd.org>=
> Subject: Re: bhyve NVMe 1.4 support
>
> On Wed, Jan 5, 2022 at 4:33 PM Chuck Tuffli <chuck@tuffli.net> wrote:
> >
> > On Mon, Jan 3, 2022 at 4:49 PM <jason@tubnor.net> wrote:
> > >
> > > Hi Chuck,
> > >
> > > Testing on 14.0-CURRENT shows the following:
> > >
> > > OpenBSD 6.9 - NVMe read/write OK
> > > Windows 10 - NVMe read OK, write FAIL Windows Server 2022 - = NVMe
> > > read OK, write FAIL Alma Linux 8.5 - NVMe read OK, write FAI= L.
> >
> > Thanks for the report, Jason. I have Alma 8.5 installed and am in= vestigating.
>
> OK, I found and fixed the regression. New version of the file in the s= ame
> place is up for folks who are interested.
>
> --chuck

This version looks good. No regression on the following guest platforms und= er 14.0-CURRENT:

OpenBSD 6.9 - NVMe read/write OK
Windows 10 - NVMe read/write OK
Windows Server 2022 - NVMe read/write OK
Alma Linux 8.5 - NVMe read/write OK
FreeBSD 13.0 - NVMe read/write OK

From a storage presentation layer in a production perspective, I can't = find any issues with the proposed update to pci_nvme.c

Some of our tests included:

The removal and addition of partitions on the NVMe presentation layer
Installation of all the operating system listed above
Where supported by the guest operating system, the TRIM command executed on= the presentation layer and guest rebooted to ensure no data corruption was= caused by TRIM

Cheers,

Jason.




--
Mario.
--000000000000cb9fe205daa41016--