From nobody Sun Mar 20 18:35:33 2022 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E29CE1A29171 for ; Sun, 20 Mar 2022 18:35:45 +0000 (UTC) (envelope-from chuck@tuffli.net) Received: from mail-yb1-xb2a.google.com (mail-yb1-xb2a.google.com [IPv6:2607:f8b0:4864:20::b2a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KM5zN6qHrz4gyV for ; Sun, 20 Mar 2022 18:35:44 +0000 (UTC) (envelope-from chuck@tuffli.net) Received: by mail-yb1-xb2a.google.com with SMTP id g138so4141457ybf.5 for ; Sun, 20 Mar 2022 11:35:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuffli-net.20210112.gappssmtp.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Bh8WqNmq+SP38W8hQeAFgPr1X9dU/oqq4DiLdJcxQiY=; b=C8DezydiIM3QQUs/7xeg26biMP6t9zhEWUIBxw4Jl0uzgcqRdK0Sk9M6g07KTGnr4h DW8Gf8fL2xiKKfo73NZf9VrR+2CTdQawYH7ZHbrfOgPWFtVAhUBajKnZcCM7OzrU1fLs uDTDlUh7bAAY/xqQB2oFaf0y+xcrwrXxb6XmfLpD7AWyVYtqkxSRXroMMs+Gllwftz15 /TZ6XUh6NXKDcfmGzPZFn6HA84j4NWjxHGFnpyZFVgHRBXcXaUPrq4NxtIWiyQWsp3jD 0sWGM9EP3t4AZEKgUmfPeiTG78VWRlX8kOyLb77qa1VifIFZQvoDfwgJQLWC7N6/mjaC AYGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Bh8WqNmq+SP38W8hQeAFgPr1X9dU/oqq4DiLdJcxQiY=; b=VWu7lVCJztqAwVGgAdAJSo+V0ajobowVMi7nYE22Z3C6L5w6mKe2kp/IJt7e6dQ23v bll+x8z6AEK1S6wwc5jGESfeCrjzeCDp99lhqX55N6oPXcHU42+ARrmiM/pjU9/L/GHQ oRUxyIxcaYuvnXNULyEkcZRfVpzn5XMCRt7N7jyMTNnzgFJk4MOY0lZyQLVrPdRTD2Mv x1KhiurUAvJWYtzWSr3gTDiLWbJVETbjZ4T6myreGC3x4d7fcd4a/MoWDFPy0/ELsEKv PwLAwscL8fdlC12KbGbCt6aad8v+URcEOXlHMgzlpW65S/nC0DRSpNjrj56r1YbADXAX 4Rdw== X-Gm-Message-State: AOAM533PMsGMX9JenwA9lZosvcIJfw0k6Oz07ChGFCFpzg3rO6JJNw3W xZQ4F6pCtw8Z0dID6oix/xzjuAORRxWDONdbJz+yzA== X-Google-Smtp-Source: ABdhPJwfZNw0TmRrPhq1Bjpfs9b+2HqrC7wnrtTO3VUqx4udNvReQzQg9FPc5s4QxgeTRv1B9BzG39JMdS5A58kYmUI= X-Received: by 2002:a25:e78b:0:b0:633:bccb:751a with SMTP id e133-20020a25e78b000000b00633bccb751amr10788887ybh.95.1647801344207; Sun, 20 Mar 2022 11:35:44 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <00bf01d80104$e6ba5de0$b42f19a0$@tubnor.net> <082b01d80697$64e95030$2ebbf090$@tubnor.net> In-Reply-To: From: Chuck Tuffli Date: Sun, 20 Mar 2022 11:35:33 -0700 Message-ID: Subject: Re: bhyve NVMe 1.4 support To: Mario Marietto Cc: jason@tubnor.net, FreeBSD virtualization Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4KM5zN6qHrz4gyV X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tuffli-net.20210112.gappssmtp.com header.s=20210112 header.b=C8Dezydi; dmarc=none; spf=permerror (mx1.freebsd.org: domain of chuck@tuffli.net uses mechanism not recognized by this client) smtp.mailfrom=chuck@tuffli.net X-Spamd-Result: default: False [0.70 / 15.00]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[tuffli-net.20210112.gappssmtp.com:s=20210112]; FREEFALL_USER(0.00)[chuck]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; DMARC_NA(0.00)[tuffli.net]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; NEURAL_SPAM_SHORT(1.00)[1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[tuffli-net.20210112.gappssmtp.com:+]; R_SPF_PERMFAIL(0.00)[empty SPF record]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::b2a:from]; MLMMJ_DEST(0.00)[freebsd-virtualization]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On Sun, Mar 20, 2022 at 8:13 AM Mario Marietto wrote: > > ----> Help me understand what "not recognized" means. The device nvme0n1 : > > I don't see the partitions that are stored inside the disk nvme. And I'm not able to mount the NTFS partition that's mapped as nvd0p2 under FreeBSD. Thank you, I understand the question now. As an experiment, I created a zvol, copied a FreeBSD disk image to it, and verified that fdisk showed what I expected on the guest. I.e. : # zfs create -V 20G zroot/vmvol/gptdisk # dd if=/vms/.img/FreeBSD-14.0-CURRENT-amd64.raw of=/dev/zvol/zroot/vmvol/gptdisk bs=1m # gpart recover zvol/zroot/vmvol/gptdisk < add /dev/zvol/zroot/vmvol/gptdisk to test-vm configuration > # vm start test-vm # ssh root@test-vm lsb_release -a Distributor ID: Ubuntu Description: Ubuntu 21.04 Release: 21.04 Codename: hirsute No LSB modules are available. # ssh root@test-vm fdisk -l /dev/nvme1n1 Disk /dev/nvme1n1: 20 GiB, 21474836480 bytes, 41943040 sectors Disk model: bhyve-NVMe Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 1B73327C-EAE2-11EB-90A0-002590EC5BF2 Device Start End Sectors Size Type /dev/nvme1n1p1 3 129 127 63.5K FreeBSD boot /dev/nvme1n1p2 130 66713 66584 32.5M EFI System /dev/nvme1n1p3 66714 2163865 2097152 1G FreeBSD swap /dev/nvme1n1p4 2163866 10552473 8388608 4G FreeBSD UFS The bhyve invocation is: bhyve -c 2 -m 2G -Hw \ -s 0,hostbridge \ -s 4:0,virtio-blk,/dev/zvol/zroot/vms/test-vm/disk0 \ -s 5:0,nvme,/dev/zvol/zroot/vmvol/disk0 \ -s 6:0,nvme,/dev/zvol/zroot/vmvol/gptdisk \ -s 7:0,virtio-net,tap0,mac=58:9c:fc:0b:ed:d6 -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ -l com1,/dev/nmdm-test-vm.1A \ test-vm Note that my experiment is different from what you are doing. But I would expect a Zvol and raw block device to behave the same as a backing-store for an emulated NVMe drive in bhyve. My experiment ran on -current, but the behavior in this area should be identical to the 13.0-p8 version you are using. Does the output on the host of # hd -n 256 /dev/nvd0 match the output on the guest of # hd -n 256 /dev/nvme0n1 ?