From nobody Wed May 01 20:24:53 2024 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VV7pk59D3z5K0nX for ; Wed, 1 May 2024 20:25:02 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx1.shrew.net (mx1.shrew.net [204.27.62.57]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4VV7pj6dDQz4lD4 for ; Wed, 1 May 2024 20:25:01 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=shrew.net header.s=default header.b="k84/APl6"; dmarc=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 204.27.62.57 as permitted sender) smtp.mailfrom=mgrooms@shrew.net Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx1.shrew.net (8.17.1/8.17.1) with ESMTP id 441KOsPm043904 for ; Wed, 1 May 2024 15:24:54 -0500 (CDT) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1714595094; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jfjdBsXmJkF8YaXxk9WmJeugZf+RdVZ/jxvWicSZvPk=; b=k84/APl6d4vE9SnO5FOGGA9ZEFdRF2jh7Bk8dtCyQRLYdoLWEpHXyeDp3PyN8tDZSSl242 7kSmnQLUE2FVMhsaWSOlfaPEqCtzeU0esWB2AN42Bi6IXj7dgoFl88cM3qAwCfqCjaQfBK +iRj6yA4VNW8hCT6DXFgCzMRFE1QnQ4EnUA2xfedE1cDjYBBEKt6PRmERnNsD7s58N+MhY 2/fGwOS3yNiM4RJELGBR/D2BRMRsAmFX476kq3nIVkZE3gXJ1a0Q53WONtG9X/xuJQPHlJ FPlkeyj8Sk0NCC5Dlh4PZ9AWEf+9CaMnHiE1nFNf6NzlH/ad/nH/11HXdmEpSA== Received: from [10.22.200.32] (unknown [136.62.156.42]) by mail.shrew.net (Postfix) with ESMTPSA id 7416C3AB37 for ; Wed, 1 May 2024 15:24:54 -0500 (CDT) Message-ID: <67721332-fa1d-4b3c-aa57-64594ad5d77a@shrew.net> Date: Wed, 1 May 2024 15:24:53 -0500 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: how to tell if TRIM is working To: stable@freebsd.org References: <5e1b5097-c1c0-4740-a491-63c709d01c25@sentex.net> Content-Language: en-US From: Matthew Grooms In-Reply-To: <5e1b5097-c1c0-4740-a491-63c709d01c25@sentex.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[shrew.net:s=default]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCPT_COUNT_ONE(0.00)[1]; DKIM_TRACE(0.00)[shrew.net:+]; DMARC_NA(0.00)[shrew.net]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US]; TO_DN_NONE(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MIME_TRACE(0.00)[0:+] X-Rspamd-Queue-Id: 4VV7pj6dDQz4lD4 On 5/1/24 14:38, mike tancsa wrote: > Kind of struggling to check if TRIM is actually working or not with my > SSDs on RELENG_14 in ZFS. > > On a pool that has almost no files on it (capacity at 0% out of 3TB), > should not > > zpool -w trim be almost instant after a couple of runs ? > Instead it seems to always take about 10min to complete. > > Looking at the stats, > > kstat.zfs.tortank1.misc.iostats.trim_bytes_failed: 0 > kstat.zfs.tortank1.misc.iostats.trim_extents_failed: 0 > kstat.zfs.tortank1.misc.iostats.trim_bytes_skipped: 2743435264 > kstat.zfs.tortank1.misc.iostats.trim_extents_skipped: 253898 > kstat.zfs.tortank1.misc.iostats.trim_bytes_written: 14835526799360 > kstat.zfs.tortank1.misc.iostats.trim_extents_written: 1169158 > > what and why are bytes being skipped ? > > One of the drives for example > >  sysctl -a kern.cam.ada.0 > kern.cam.ada.0.trim_ticks: 0 > kern.cam.ada.0.trim_goal: 0 > kern.cam.ada.0.sort_io_queue: 0 > kern.cam.ada.0.rotating: 0 > kern.cam.ada.0.unmapped_io: 1 > kern.cam.ada.0.flags: > 0x1be3bde > kern.cam.ada.0.max_seq_zones: 0 > kern.cam.ada.0.optimal_nonseq_zones: 0 > kern.cam.ada.0.optimal_seq_zones: 0 > kern.cam.ada.0.zone_support: None > kern.cam.ada.0.zone_mode: Not Zoned > kern.cam.ada.0.write_cache: -1 > kern.cam.ada.0.read_ahead: -1 > kern.cam.ada.0.trim_lbas: 7771432624 > kern.cam.ada.0.trim_ranges: 371381 > kern.cam.ada.0.trim_count: 310842 > kern.cam.ada.0.delete_method: DSM_TRIM > > If I take one of the disks out of the pool and replace it with a > spare, and do a manual trim it seems to work > I had a hard time seeing evidence of this at the disk level while fiddling with TRIM recently. It appeared that at least some counters are driver and operation specific. For example, the da driver appears to update counters in some paths but not others. I assume that ada is different. There is a bug report for da, but haven't seen any feedback ... https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277673 You could try to run gstat with the -d flag during the time period when the delete operations are expected to occur. That should give you an idea of what's happening at the disk level in real time but may not offer more info than you're already seeing. > e.g. here was one disk in the pool that was taking a long time for > each zpool trim > > # time trim -f /dev/ada1 > trim /dev/ada1 offset 0 length 1000204886016 > 0.000u 0.057s 1:29.33 0.0%      5+184k 0+0io 0pf+0w > and then if I re-run it > #  time trim -f /dev/ada1 > trim /dev/ada1 offset 0 length 1000204886016 > 0.000u 0.052s 0:04.15 1.2%      1+52k 0+0io 0pf+0w > > 90 seconds and then 4 seconds after that. > -Matthew