how to tell if TRIM is working

From: mike tancsa <mike_at_sentex.net>
Date: Wed, 01 May 2024 19:38:24 UTC 
Kind of struggling to check if TRIM is actually working or not with my 
SSDs on RELENG_14 in ZFS.

On a pool that has almost no files on it (capacity at 0% out of 3TB), 
should not

zpool -w trim <pool> be almost instant after a couple of runs ? Instead 
it seems to always take about 10min to complete.

Looking at the stats,

kstat.zfs.tortank1.misc.iostats.trim_bytes_failed: 0
kstat.zfs.tortank1.misc.iostats.trim_extents_failed: 0
kstat.zfs.tortank1.misc.iostats.trim_bytes_skipped: 2743435264
kstat.zfs.tortank1.misc.iostats.trim_extents_skipped: 253898
kstat.zfs.tortank1.misc.iostats.trim_bytes_written: 14835526799360
kstat.zfs.tortank1.misc.iostats.trim_extents_written: 1169158

what and why are bytes being skipped ?

One of the drives for example

  sysctl -a kern.cam.ada.0
kern.cam.ada.0.trim_ticks: 0
kern.cam.ada.0.trim_goal: 0
kern.cam.ada.0.sort_io_queue: 0
kern.cam.ada.0.rotating: 0
kern.cam.ada.0.unmapped_io: 1
kern.cam.ada.0.flags: 
0x1be3bde<CAN_48BIT,CAN_FLUSHCACHE,CAN_NCQ,CAN_DMA,WAS_OTAG,CAN_TRIM,OPEN,SCTX_INIT,CAN_POWERMGT,CAN_DMA48,CAN_LOG,CAN_WCACHE,CAN_RAHEAD,PROBED,ANNOUNCED,DIRTY,PIM_ATA_EXT,UNMAPPEDIO>
kern.cam.ada.0.max_seq_zones: 0
kern.cam.ada.0.optimal_nonseq_zones: 0
kern.cam.ada.0.optimal_seq_zones: 0
kern.cam.ada.0.zone_support: None
kern.cam.ada.0.zone_mode: Not Zoned
kern.cam.ada.0.write_cache: -1
kern.cam.ada.0.read_ahead: -1
kern.cam.ada.0.trim_lbas: 7771432624
kern.cam.ada.0.trim_ranges: 371381
kern.cam.ada.0.trim_count: 310842
kern.cam.ada.0.delete_method: DSM_TRIM

If I take one of the disks out of the pool and replace it with a spare, 
and do a manual trim it seems to work

e.g. here was one disk in the pool that was taking a long time for each 
zpool trim

# time trim -f /dev/ada1
trim /dev/ada1 offset 0 length 1000204886016
0.000u 0.057s 1:29.33 0.0%      5+184k 0+0io 0pf+0w
and then if I re-run it
#  time trim -f /dev/ada1
trim /dev/ada1 offset 0 length 1000204886016
0.000u 0.052s 0:04.15 1.2%      1+52k 0+0io 0pf+0w

90 seconds and then 4 seconds after that.


     ---Mike