ZFS mirror keeps going offline/removed

[Aaron]

So recently my ZFS mirror keeps getting both drives marked as removed which
makes my mirror go offline. I don't see anything obvious in my
/var/log/messages (other than as below), although the errors always seem to
be in metadata. The latest error seems to be in a non-metadata location,
although not a file but a zfs directory. Whenever I run zpool clear -F -n
bulk it ends up back online and after a scrub no errors, but it keeps
happening. Based on the below paste, what could I look at that would tell
me which drive, or both, or what the issue is?

I'm running an old dual Xeon x5650, ECC memory (ran memtest86+ on it before
installing, no errors).

--Aaron



/var/log/messages


May 29 19:20:33 darkserver kernel: ada2 at ahcich2 bus 0 scbus2 target 0
lun 0
May 29 19:20:33 darkserver kernel: ada2: <ST4000DM005-2DP166 0001> s/n
ZDH1CTDZ detached
May 29 19:20:33 darkserver kernel: (ada2:ahcich2:0:0:0): Periph destroyed
May 29 19:20:33 darkserver ZFS: vdev state changed,
pool_guid=9608097630476313818 vdev_guid=4999141564141954755
May 29 19:20:33 darkserver ZFS: vdev is removed,
pool_guid=9608097630476313818 vdev_guid=4999141564141954755
May 29 19:20:38 darkserver kernel: ada8 at ahcich9 bus 0 scbus9 target 0
lun 0
May 29 19:20:38 darkserver kernel: ada8: <ST4000DM005-2DP166 0001> s/n
ZDH1CS9M detached
May 29 19:20:38 darkserver kernel: (ada8:ahcich9:0:0:0): Periph destroyed
May 29 19:21:19 darkserver kernel: ada2 at ahcich2 bus 0 scbus2 target 0
lun 0
May 29 19:21:19 darkserver kernel: ada2: <ST4000DM005-2DP166 0001> ACS-3
ATA SATA 3.x device
May 29 19:21:19 darkserver kernel: ada2: Serial Number ZDH1CTDZ
May 29 19:21:19 darkserver kernel: ada2: 600.000MB/s transfers (SATA 3.x,
UDMA6, PIO 8192bytes)
May 29 19:21:19 darkserver kernel: ada2: Command Queueing enabled
May 29 19:21:19 darkserver kernel: ada2: 3815447MB (7814037168 512 byte
sectors)
May 29 19:21:19 darkserver kernel: ada2: quirks=0x1<4K>
May 29 19:21:21 darkserver kernel: ada8 at ahcich9 bus 0 scbus9 target 0
lun 0
May 29 19:21:21 darkserver kernel: ada8: <ST4000DM005-2DP166 0001> ACS-3
ATA SATA 3.x device
May 29 19:21:21 darkserver kernel: ada8: Serial Number ZDH1CS9M
May 29 19:21:21 darkserver kernel: ada8: 300.000MB/s transfers (SATA 2.x,
UDMA6, PIO 8192bytes)
May 29 19:21:21 darkserver kernel: ada8: Command Queueing enabled
May 29 19:21:21 darkserver kernel: ada8: 3815447MB (7814037168 512 byte
sectors)
May 29 19:21:21 darkserver kernel: ada8: quirks=0x1<4K>



# zpool status -x -v

  pool: bulk
 state: UNAVAIL
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool
clear'.
   see: http://illumos.org/msg/ZFS-8000-HC
  scan: scrub repaired 0 in 7h22m with 0 errors on Tue May 29 04:27:28 2018
config:

        NAME                      STATE     READ WRITE CKSUM
        bulk                      UNAVAIL      0     0     0
          mirror-0                UNAVAIL      1     0     0
            4999141564141954755   REMOVED      0     0     0  was /dev/ada2
            16676652866205005686  REMOVED      0     0     0  was /dev/ada8

errors: Permanent errors have been detected in the following files:

        <metadata>:<0x0>
        <metadata>:<0x1>
        <metadata>:<0x1b>
        <metadata>:<0x33>
        bulk/video:<0xc0ad>