Re: how to tell if TRIM is working

On 5/1/24 14:38, mike tancsa wrote:
> Kind of struggling to check if TRIM is actually working or not with my 
> SSDs on RELENG_14 in ZFS.
>
> On a pool that has almost no files on it (capacity at 0% out of 3TB), 
> should not
>
> zpool -w trim <pool> be almost instant after a couple of runs ? 
> Instead it seems to always take about 10min to complete.
>
> Looking at the stats,
>
> kstat.zfs.tortank1.misc.iostats.trim_bytes_failed: 0
> kstat.zfs.tortank1.misc.iostats.trim_extents_failed: 0
> kstat.zfs.tortank1.misc.iostats.trim_bytes_skipped: 2743435264
> kstat.zfs.tortank1.misc.iostats.trim_extents_skipped: 253898
> kstat.zfs.tortank1.misc.iostats.trim_bytes_written: 14835526799360
> kstat.zfs.tortank1.misc.iostats.trim_extents_written: 1169158
>
> what and why are bytes being skipped ?
>
> One of the drives for example
>
>  sysctl -a kern.cam.ada.0
> kern.cam.ada.0.trim_ticks: 0
> kern.cam.ada.0.trim_goal: 0
> kern.cam.ada.0.sort_io_queue: 0
> kern.cam.ada.0.rotating: 0
> kern.cam.ada.0.unmapped_io: 1
> kern.cam.ada.0.flags: 
> 0x1be3bde<CAN_48BIT,CAN_FLUSHCACHE,CAN_NCQ,CAN_DMA,WAS_OTAG,CAN_TRIM,OPEN,SCTX_INIT,CAN_POWERMGT,CAN_DMA48,CAN_LOG,CAN_WCACHE,CAN_RAHEAD,PROBED,ANNOUNCED,DIRTY,PIM_ATA_EXT,UNMAPPEDIO>
> kern.cam.ada.0.max_seq_zones: 0
> kern.cam.ada.0.optimal_nonseq_zones: 0
> kern.cam.ada.0.optimal_seq_zones: 0
> kern.cam.ada.0.zone_support: None
> kern.cam.ada.0.zone_mode: Not Zoned
> kern.cam.ada.0.write_cache: -1
> kern.cam.ada.0.read_ahead: -1
> kern.cam.ada.0.trim_lbas: 7771432624
> kern.cam.ada.0.trim_ranges: 371381
> kern.cam.ada.0.trim_count: 310842
> kern.cam.ada.0.delete_method: DSM_TRIM
>
> If I take one of the disks out of the pool and replace it with a 
> spare, and do a manual trim it seems to work
>
I had a hard time seeing evidence of this at the disk level while 
fiddling with TRIM recently. It appeared that at least some counters are 
driver and operation specific. For example, the da driver appears to 
update counters in some paths but not others. I assume that ada is 
different. There is a bug report for da, but haven't seen any feedback ...

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277673

You could try to run gstat with the -d flag during the time period when 
the delete operations are expected to occur. That should give you an 
idea of what's happening at the disk level in real time but may not 
offer more info than you're already seeing.

> e.g. here was one disk in the pool that was taking a long time for 
> each zpool trim
>
> # time trim -f /dev/ada1
> trim /dev/ada1 offset 0 length 1000204886016
> 0.000u 0.057s 1:29.33 0.0%      5+184k 0+0io 0pf+0w
> and then if I re-run it
> #  time trim -f /dev/ada1
> trim /dev/ada1 offset 0 length 1000204886016
> 0.000u 0.052s 0:04.15 1.2%      1+52k 0+0io 0pf+0w
>
> 90 seconds and then 4 seconds after that.
>

-Matthew