From nobody Sat Feb 17 21:53:10 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TcjGm2CVDz59ntP for ; Sat, 17 Feb 2024 21:53:20 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx1.shrew.net (mx1.shrew.net [204.27.62.57]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TcjGl2DKCz4xw9 for ; Sat, 17 Feb 2024 21:53:19 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=shrew.net header.s=default header.b=hEJv+ixX; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 204.27.62.57 as permitted sender) smtp.mailfrom=mgrooms@shrew.net; dmarc=none Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx1.shrew.net (8.17.1/8.17.1) with ESMTP id 41HLrBKl062450 for ; Sat, 17 Feb 2024 15:53:11 -0600 (CST) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1708206791; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=eLmpFBk/n8cFe/W+Qcl5WTr01GVr3tnc4szjIB4dzKY=; b=hEJv+ixXFFelKKtRkN/boD5CEo9h6XbW3kslYyz+tMLfg7rg3VAn+nkHvp0vM7AcwSH+5a IBLn6aFovHtxaWpuNkUo25NxUfgnXakv9ZGF0Z7zTYLFjO4duWshXeLOzMC3A0DUPNg46k 4xtpiNJaNJzMFGKQSjL8k2TCVJeyUTS5//Zv4NxdrSZkPF96YYDcC/CYfMbZPtp9JSxMYW UITv9WO39lkz+HMlDlciqE6i58iBzHgscWabQGmWil97IiSnYUFbTGYqB6T9SW2TjwLvo6 Dyx7cZCcGE3R4dtLPUYgN9yyqP8Ybi/N73f/SOydpZpKhaSYXWLlBa5FQuMAPw== Received: from [10.22.200.32] (unknown [136.62.156.42]) by mail.shrew.net (Postfix) with ESMTPSA id 68E713AB37 for ; Sat, 17 Feb 2024 15:53:11 -0600 (CST) Content-Type: multipart/alternative; boundary="------------rzBtn40afBj06nBx93HWKb1k" Message-ID: <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> Date: Sat, 17 Feb 2024 15:53:10 -0600 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bhyve disk performance issue Content-Language: en-US To: virtualization@freebsd.org References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> From: Matthew Grooms In-Reply-To: <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> X-Rspamd-Queue-Id: 4TcjGl2DKCz4xw9 X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.45 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.959]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[shrew.net:s=default]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DMARC_NA(0.00)[shrew.net]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; DKIM_TRACE(0.00)[shrew.net:+] This is a multi-part message in MIME format. --------------rzBtn40afBj06nBx93HWKb1k Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2/16/24 12:00, Matthew Grooms wrote: > On 2/16/24 11:42, Chuck Tuffli wrote: >> On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote: >>> >>> Hi All, >>> >>> >>> I'm in the middle of a project that involves building out a handful >>> of servers to host virtual Linux instances. Part of that includes >>> testing bhyve to see how it performs. The intent is to compare host >>> storage options such as raw vs zvol block devices and ufs vs zfs >>> disk images using hardware raid vs zfs managed disks. It would also >>> involve >>> >>> >> … >>> >>> Here is a list of a few other things I'd like to try: >>> >>> >>> 1) Wiring guest memory ( unlikely as it's 32G of 256G ) >>> 2) Downgrading the host to 13.2-RELEASE >> >> FWIW we recently did a similar exercise and saw significant >> performance differences on ZFS backed disk images when comparing 14.0 >> and 13.2. We didn’t have time to root cause the difference, so it >> could simply be some tuning difference needed for 14. > > Hi Chuck, > > That's very helpful feedback. I'll start by downgrading the host to > 13.2 and report back here. > Unfortunately same story with 13.2. I'm going to try and downgrade to 12.4 and see if it gets any better. ================================================================================ Begin @ Sat Feb 17 11:00:01 CST 2024 Version  2.00       ------Sequential Output------ --Sequential Input- --Random-                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP localhost.lo 63640M  690k  99  1.5g  97  727m  78  950k  99  1.3g 68 +++++ +++ Latency             11759us   29114us    8098us    8649us 23413us    4540us Version  2.00       ------Sequential Create------ --------Random Create-------- localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency              7791us     131us    1671us     464us 15us    1811us End @ Sat Feb 17 11:03:13 CST 2024 ================================================================================ Begin @ Sat Feb 17 11:10:01 CST 2024 Version  2.00       ------Sequential Output------ --Sequential Input- --Random-                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP localhost.lo 63640M  667k  99  449m  99  313m  94  940k  99  398m 99 16204 563 Latency             12147us    1079us   24470us    8795us 17853us    4384us Version  2.00       ------Sequential Create------ --------Random Create-------- localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP                  16     0  93 +++++ +++ +++++ +++     0  96 +++++ +++ +++++ +++ Latency               118us     159us     269us     164us 54us     844us End @ Sat Feb 17 11:18:43 CST 2024 --------------rzBtn40afBj06nBx93HWKb1k Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 2/16/24 12:00, Matthew Grooms wrote:
On 2/16/24 11:42, Chuck Tuffli wrote:
On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote:

Hi All,


I'm in the middle of a project that involves building out a handful of servers to host virtual Linux instances. Part of that includes testing bhyve to see how it performs. The intent is to compare host storage options such as raw vs zvol block devices and ufs vs zfs disk images using hardware raid vs zfs managed disks. It would also involve


…

Here is a list of a few other things I'd like to try:


1) Wiring guest memory ( unlikely as it's 32G of 256G )
2) Downgrading the host to 13.2-RELEASE

FWIW we recently did a similar exercise and saw significant performance differences on ZFS backed disk images when comparing 14.0 and 13.2. We didn’t have time to root cause the difference, so it could simply be some tuning difference needed for 14.

Hi Chuck,

That's very helpful feedback. I'll start by downgrading the host to 13.2 and report back here.


Unfortunately same story with 13.2. I'm going to try and downgrade to 12.4 and see if it gets any better.

================================================================================
Begin @ Sat Feb 17 11:00:01 CST 2024

Version  2.00       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
localhost.lo 63640M  690k  99  1.5g  97  727m  78  950k  99  1.3g  68 +++++ +++
Latency             11759us   29114us    8098us    8649us   23413us    4540us
Version  2.00       ------Sequential Create------ --------Random Create--------
localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency              7791us     131us    1671us     464us      15us    1811us

End @ Sat Feb 17 11:03:13 CST 2024
================================================================================
Begin @ Sat Feb 17 11:10:01 CST 2024

Version  2.00       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
localhost.lo 63640M  667k  99  449m  99  313m  94  940k  99  398m  99 16204 563
Latency             12147us    1079us   24470us    8795us   17853us    4384us
Version  2.00       ------Sequential Create------ --------Random Create--------
localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16     0  93 +++++ +++ +++++ +++     0  96 +++++ +++ +++++ +++
Latency               118us     159us     269us     164us      54us     844us

End @ Sat Feb 17 11:18:43 CST 2024

--------------rzBtn40afBj06nBx93HWKb1k--