From nobody Wed Feb 28 21:31:39 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TlSGq4qzWz5CSNR for ; Wed, 28 Feb 2024 21:31:47 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx1.shrew.net (mx1.shrew.net [204.27.62.57]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TlSGp5y0Tz4gFj for ; Wed, 28 Feb 2024 21:31:46 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=shrew.net header.s=default header.b=EOMbAPKp; dmarc=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 204.27.62.57 as permitted sender) smtp.mailfrom=mgrooms@shrew.net Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx1.shrew.net (8.17.1/8.17.1) with ESMTP id 41SLVeIj006909; Wed, 28 Feb 2024 15:31:40 -0600 (CST) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1709155900; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=hIYOrKlheX1G4yxRxytbAE4RMn8PtinUlq28NWLbP+8=; b=EOMbAPKpfI9P0D8TptuM8jAgD0Lk82+3Dn2tWy5JZXVjtCmuKLmloG+Om0YMMbipBYvONX RVqjtbQE8Md6uwzp6dT5B6NK1kxF1JBMlRJ5YBhqPhhIxrUXjYB2hnxLaD8wAcdoQrKf7a gqjwo+v+eXi2MJ7/s9eJrDQFLhSqg0tJi3lLj487dlRAz5e9kFw24qI8uqo3tbGsQtymgY zQKO/jT0i8OjhOrOnph+sL2thSwNXH0IWUHQZHJmFeFSO8r1lkUhpzyw+cgy6imUFt4uYP ZrDQ3NxrgQoT/Drl/nirFWbF5d2Nb2dD+jHF5SpImT0Kz5qEVPHkSeknaDg1bg== Received: from [10.22.200.32] (unknown [136.62.156.42]) by mail.shrew.net (Postfix) with ESMTPSA id 0FE5F3AB37; Wed, 28 Feb 2024 15:31:40 -0600 (CST) Content-Type: multipart/alternative; boundary="------------uA3ah2p5aRXCMQ7fXwU07F9H" Message-ID: <3738b08a-7841-4d18-9439-5f1c73a5c9e1@shrew.net> Date: Wed, 28 Feb 2024 15:31:39 -0600 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bhyve disk performance issue Content-Language: en-US To: Vitaliy Gusev Cc: virtualization@freebsd.org References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net> <6a4e7e1d-cca5-45d4-a268-1805a15d9819@shrew.net> <25ddf43d-f700-4cb5-af2a-1fe669d1e24b@shrew.net> <1DAEB435-A613-4A04-B63F-D7AF7A0B7C0A@gmail.com> <3850080E-EBD1-4414-9C4E-DD89611C9F58@gmail.com> From: Matthew Grooms In-Reply-To: <3850080E-EBD1-4414-9C4E-DD89611C9F58@gmail.com> X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.47 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.979]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[shrew.net:s=default]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; XM_UA_NO_VERSION(0.01)[]; ARC_NA(0.00)[]; DMARC_NA(0.00)[shrew.net]; TAGGED_RCPT(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[shrew.net:+] X-Rspamd-Queue-Id: 4TlSGp5y0Tz4gFj This is a multi-part message in MIME format. --------------uA3ah2p5aRXCMQ7fXwU07F9H Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2/28/24 15:02, Vitaliy Gusev wrote: > > >> On 28 Feb 2024, at 23:03, Matthew Grooms wrote: >> >> ... >> The virtual disks were provisioned with either a 128G disk image or a >> 1TB raw partition, so I don't think space was an issue. >> >> Trim is definitely not an issue. I'm using a tiny fraction of the >> 32TB array have tried both heavily under-provisioned HW RAID10 and SW >> RAID10 using GEOM. The latter was tested after sending full trim >> resets to all drives individually. >> > It could be then TRIM/UNMAP is not used, zvol (for the instance) > becomes full for the while. ZFS considers it as all blocks are used > and write operations could  have troubles. I believe it was recently > fixed. > > Also look at this one: > > GuestFS->UNMAP->bhyve->Host-FS->PhysicalDisk > > The problem of UNMAP that it could have unpredictable slowdown at any > time. So I would suggest to check results with enabled and disabled > UNMAP in a guest. > Yes. I'm aware of issues associated with TRIM/UNMAP, but that's not implemented by any hardware RAID vendor that I'm aware of. I tested with both hardware and software RAID10. The issue I'm reporting is present in both cases. I'm quite certain this has nothing to do with TRIM/UNMAP. -Matthew --------------uA3ah2p5aRXCMQ7fXwU07F9H Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 2/28/24 15:02, Vitaliy Gusev wrote:


On 28 Feb 2024, at 23:03, Matthew Grooms <mgrooms@shrew.net> wrote:

...
The virtual disks were provisioned with either a 128G disk image or a 1TB raw partition, so I don't think space was an issue.

Trim is definitely not an issue. I'm using a tiny fraction of the 32TB array have tried both heavily under-provisioned HW RAID10 and SW RAID10 using GEOM. The latter was tested after sending full trim resets to all drives individually.

It could be then TRIM/UNMAP is not used, zvol (for the instance) becomes full for the while. ZFS considers it as all blocks are used and write operations could  have troubles. I believe it was recently fixed.

Also look at this one:

    GuestFS->UNMAP->bhyve->Host-FS->PhysicalDisk

The problem of UNMAP that it could have unpredictable slowdown at any time. So I would suggest to check results with enabled and disabled UNMAP in a guest.

Yes. I'm aware of issues associated with TRIM/UNMAP, but that's not implemented by any hardware RAID vendor that I'm aware of. I tested with both hardware and software RAID10. The issue I'm reporting is present in both cases. I'm quite certain this has nothing to do with TRIM/UNMAP.

-Matthew

--------------uA3ah2p5aRXCMQ7fXwU07F9H--