From nobody Fri Feb 16 17:55:27 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tc0394jlNz53wM1 for ; Fri, 16 Feb 2024 17:55:49 +0000 (UTC) (envelope-from chuck@tuffli.net) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tc0392j1sz3x8f for ; Fri, 16 Feb 2024 17:55:49 +0000 (UTC) (envelope-from chuck@tuffli.net) Authentication-Results: mx1.freebsd.org; none Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailout.nyi.internal (Postfix) with ESMTP id 680925C00AE; Fri, 16 Feb 2024 12:55:48 -0500 (EST) Received: from imap51 ([10.202.2.101]) by compute7.internal (MEProxy); Fri, 16 Feb 2024 12:55:48 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuffli.net; h=cc :cc:content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1708106148; x=1708192548; bh=N6shIJCxz7 cMT0xOjez70/QlT/pqNM+UlU8mhfCYi8Y=; b=rfmyFIL28tddL3XX2mfV2r8Xo6 GZUtO7qdEE6V6Pz/havVF2P1ziWvA4IKTNPebzVosHQLE36tySG8Mne5fSOX9B+N qpPHfdWcgybvz5gxKZvJGnbOwKCEtUtj4sk+OLEDSPufmJWrFDSG03qxnveGUFBx ipT2fO2GCM1hyxTfpOt4vKcHXEPE8UF4DmWb1f5A+OHZUK/RSINUi+z0ffpqFiSz LYZsLw+WPd7MvznMubnLW+xcvUypwqju3ZEAUaJteLwtkGmxfQW1Y7KQZdKeViax 9tyftKuy/8S5AF8GiXaXlV3zb7vtAtvayVNH4afjlbaSG5wm82bDQcpyAB8Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1708106148; x=1708192548; bh=N6shIJCxz7cMT0xOjez70/QlT/pq NM+UlU8mhfCYi8Y=; b=d7CKlPvdUYqdpmWESjDmpS1n7LG3MaYj29oMmyHVBrO/ pHJHCJ7rZmI0KeiNFmpDqkwD9/ENb3YvbZug+8IzZxccWBMsPMUg/sNEcRB3oTx9 Bm8oP3cXvSAYLgEo4okJrwnYOaBRFqLJLkw8ewRahDZpDtOkuphOUaSmNOQANuEJ eVr39pMladn9+qEz3BHjF6tPR6oPHSJs46fcC0mvNnpPI7557UegGvKmV8YX/ZyL e7ResoZ378Eo58vrZ/mvjbyiDaBtK8xfRxACKGKjSshzkCFxowMMt6dpqOMN45V6 7GFTQIqoBi2Wha5731xtvfInRUVS/SFW5eKd80sLhA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddvgddutdeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvfevufgtsegrtderreerreejnecuhfhrohhmpedfvehh uhgtkhcuvfhufhhflhhifdcuoegthhhutghksehtuhhffhhlihdrnhgvtheqnecuggftrf grthhtvghrnhepudeuteehiefgtdehheduveevgeehieevveeuvddthedvueevjeehfeet heefjeffnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomh eptghhuhgtkhesthhufhhflhhirdhnvght X-ME-Proxy: Feedback-ID: ib6f94606:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 0E4F9B6008D; Fri, 16 Feb 2024 12:55:47 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-144-ge5821d614e-fm-20240125.002-ge5821d61 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Message-Id: <5fc8b9d9-da94-4694-9134-d0cc22df2eaf@app.fastmail.com> In-Reply-To: References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> Date: Fri, 16 Feb 2024 09:55:27 -0800 From: "Chuck Tuffli" To: "Paul Procacci" Cc: "Matthew Grooms" , "FreeBSD virtualization" Subject: Re: bhyve disk performance issue Content-Type: multipart/alternative; boundary=edaaeeb4c44c4b3f8141d4116aa98709 X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4Tc0392j1sz3x8f X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US] --edaaeeb4c44c4b3f8141d4116aa98709 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable On Fri, Feb 16, 2024, at 9:45 AM, Paul Procacci wrote: >=20 >=20 > On Fri, Feb 16, 2024 at 12:43=E2=80=AFPM Chuck Tuffli wrote: >> __ >> On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote: >>> Hi All, >>>=20 >>>=20 >>>=20 >>> I'm in the middle of a project that involves building out a handful = of servers to host virtual Linux instances. Part of that includes testin= g bhyve to see how it performs. The intent is to compare host storage op= tions such as raw vs zvol block devices and ufs vs zfs disk images using= hardware raid vs zfs managed disks. It would also involve >>>=20 >>>=20 >> =E2=80=A6 >>> Here is a list of a few other things I'd like to try: >>>=20 >>>=20 >>> 1) Wiring guest memory ( unlikely as it's 32G of 256G ) >>> 2) Downgrading the host to 13.2-RELEASE >>=20 >> FWIW we recently did a similar exercise and saw significant performan= ce differences on ZFS backed disk images when comparing 14.0 and 13.2. W= e didn=E2=80=99t have time to root cause the difference, so it could sim= ply be some tuning difference needed for 14.=20 >>=20 >> =E2=80=94chuck > I myself am actually doing something very very similar. > I was seeing atrocious disk performance until I set the disk type to n= vme. > Now it's screaming fast. >=20 > disk0_type=3D"nvme" > Not sure what yours is set at, but it might be worth looking into. Similar to Matthew, we were testing both virtio and nvme and saw perform= ance differences for both emulation types between 13 and 14.=20 --edaaeeb4c44c4b3f8141d4116aa98709 Content-Type: text/html;charset=utf-8 Content-Transfer-Encoding: quoted-printable
On Fri, Feb 16,= 2024, at 9:45 AM, Paul Procacci wrote:


On Fri, Feb 16, 2024 at 12:43=E2=80=AFPM Chuck Tuffli <= ;chuck@tuffli.net> wrote:
=

On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote:

Hi All,=


I'm in the middle of a project that involves bui= lding out a handful of servers to host virtual Linux instances. Part of that includes testing bhyve to see how it performs. The intent is to compare host storage options such as raw vs zvol block devices and ufs vs zfs disk images using hardware raid vs zfs managed disks. It would also involve


=E2=80= =A6

Here is a list of a few othe= r things I'd like to try:


1) Wiring gu= est memory ( unlikely as it's 32G of 256G )
2) Downgrading= the host to 13.2-RELEASE

FWIW= we recently did a similar exercise and saw significant performance diff= erences on ZFS backed disk images when comparing 14.0 and 13.2. We didn=E2= =80=99t have time to root cause the difference, so it could simply be so= me tuning difference needed for 14. 

=E2= =80=94chuck
I myself = am actually doing something very very similar.
I was seein= g atrocious disk performance until I set the disk type to nvme.
Now it's screaming fast.

disk0_t= ype=3D"nvme"
Not sure what yours is set at, but it m= ight be worth looking into.
Similar to = Matthew, we were testing both virtio and nvme and saw performance differ= ences for both emulation types between 13 and 14. 

--edaaeeb4c44c4b3f8141d4116aa98709--