suspend/resume versus OpenZFS on USB

[Graham Perrin]

I'm not sure what was going on with device numbers in my previous 
report, but this much seems to be easily and consistently reproducible.

If the pool is ONLINE and imported at suspend time – and if not actively 
writing to the device at suspend time – then at resume time:

* the pool is SUSPENDED
* the device is shown as REMOVED
* zpool clear followed by zpool scrub lead to disappearance of permanent 
metadata errors, without physically touching the device.

----

root at momh167-gjp4-8570p:~ # ls /dev/da*
/dev/da0        /dev/da0p1      /dev/da1        /dev/da1p1
root at momh167-gjp4-8570p:~ # zpool status usb
   pool: usb
  state: ONLINE
   scan: scrub repaired 0B in 00:00:08 with 0 errors on Sat Sep 12 
02:55:26 2020
config:

         NAME        STATE     READ WRITE CKSUM
         usb         ONLINE       0     0     0
           da0p1     ONLINE       0     0     0

errors: No known data errors
root at momh167-gjp4-8570p:~ # date ; uname -v ; zfs version ; acpiconf -s3
Sat Sep 12 02:59:20 BST 2020
FreeBSD 13.0-CURRENT #64 r365364: Sun Sep  6 01:38:18 BST 2020 
root at momh167-gjp4-8570p:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG
zfs-0.8.0-1
zfs-kmod-0.8.0-1
root at momh167-gjp4-8570p:~ # zpool status -v
   pool: copperbowl
  state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
         still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
         the pool may no longer be accessible by software that does not 
support
         the features. See zpool-features(5) for details.
   scan: scrub repaired 0B in 01:39:31 with 0 errors on Thu Sep  3 
01:12:21 2020
config:

         NAME          STATE     READ WRITE CKSUM
         copperbowl    ONLINE       0     0     0
           ada0p4.eli  ONLINE       0     0     0

errors: No known data errors

   pool: usb
  state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool 
clear'.
    see: https://zfsonlinux.org/msg/ZFS-8000-HC
   scan: scrub repaired 0B in 00:00:08 with 0 errors on Sat Sep 12 
02:55:26 2020
config:

         NAME        STATE     READ WRITE CKSUM
         usb         UNAVAIL      0     0     0  insufficient replicas
           da0p1     REMOVED      0     0     0

errors: List of errors unavailable: pool I/O is currently suspended
root at momh167-gjp4-8570p:~ # zpool clear usb && zpool scrub usb
root at momh167-gjp4-8570p:~ # date ; zpool status usb
Sat Sep 12 03:00:19 BST 2020
   pool: usb
  state: ONLINE
   scan: scrub repaired 0B in 00:00:01 with 0 errors on Sat Sep 12 
03:00:11 2020
config:

         NAME        STATE     READ WRITE CKSUM
         usb         ONLINE       0     0     0
           da0p1     ONLINE       0     0     0

errors: No known data errors
root at momh167-gjp4-8570p:~ #