Confused by zfs errors

[Mike Clarke]

I'm using zfs on FreeBSD 12.0-RELEASE-p4 GENERIC  amd64 on a desktop system which is shut down each night and rebooted each morning and my daily periodic scripts are reporting some filesystem errors which I am unable to fix and are somewhat confusing.

/etc/periodic/security/100.chksetuid is reporting:


---------------------------------------------------
Checking setuid files and devices:
find: /home/liz/Maildir/cur/1342434798.M711754P2579.curlew.lan,S=82312,W=83431:2,S: Unknown error: 122
find: /home/mike/Maildir/cur/1354984767.M156539P5390.curlew.lan,S=217133,W=220003:2,RS: Unknown error: 122
find: /home/mike/Maildir/cur/1387550678.M716573P2948.curlew.lan,S=99139,W=101030:2,S: Unknown error: 122
find: /home/mike/mp3/tapes/Wind Music of Holst & Vaughan Williams: Unknown error: 122
---------------------------------------------------

And /etc/periodic/daily/404.status-zfs is reporting:


---------------------------------------------------
Checking status of zfs pools:
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
ssd    119G  48.1G  70.9G        -         -    32%    40%  1.00x  ONLINE  -
sys    460G   314G   146G        -         -    48%    68%  1.00x  ONLINE  -

  pool: sys
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub repaired 0 in 0 days 01:28:12 with 0 errors on Wed Jun  5 11:01:47 2019
config:

        NAME          STATE     READ WRITE CKSUM
        sys           ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            gpt/sys2  ONLINE       0     0     0
            gpt/sys1  ONLINE       0     0     0

errors: 1 data errors, use '-v' for a list
---------------------------------------------------

I've checked the memory with Memtest86 which reports some errors in the hammer test which I imagine could be the cause of the filesystem corruption and I'm in the process of raising a ticket to replace the memory modules under warranty but in the meantime I need to try to fix the errors in the filesystem.

The first problem is that I can't fix the "Unknown error: 122" message for the mp3 directory and the 3 mail files because if I try to delete them or copy my backup copies into them I just get another 122 error.

When I run zpool status -v sys I get the following:

---------------------------------------------------
  pool: sys
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub repaired 0 in 0 days 01:28:12 with 0 errors on Wed Jun  5 11:01:47 2019
config:

        NAME          STATE     READ WRITE CKSUM
        sys           ONLINE       0     0    15
          mirror-0    ONLINE       0     0    60
            gpt/sys2  ONLINE       0     0    60
            gpt/sys1  ONLINE       0     0    60

errors: Permanent errors have been detected in the following files:

        sys/DATA/home:<0x0>



Can I resolve the sys/DATA/home:<0x0> issue without destroying the entire pool and restoring from backup? Yes I do have a full backup which is free from these errors but I'd prefer to avoid deleting everything unless I really have to.

The above zpool status from the command line is reporting CKSUM errors which the periodic script reports as all zeros. I've checked this over a number of days and the script always reports zeros while checks from the command line always give a number of CKSUM errors which vary (up and down) from day to day.

I also see that if I run zpool scrub without the -v option as a normal user it reports "errors: 2 data errors, use '-v' for a list" but when I run it as root it only reports 1 data error.

The errors first occurred before I ran zpool scrub on June 5 but scrub was not able to repair them.

-- 
Mike Clarke