out of HDD space - zfs degraded

[Dan Langille]

On 10/2/2010 10:04 PM, Dan Langille wrote:

> After a 'shutdown -p now', it was about 20 minutes before I went and
> powered it up (I was on minecraft). The box came back with the missing HDD:
>
> $ zpool status storage
> pool: storage
> state: ONLINE
> status: One or more devices has experienced an unrecoverable error. An
> attempt was made to correct the error. Applications are unaffected.
> action: Determine if the device needs to be replaced, and clear the errors
> using 'zpool clear' or replace the device with 'zpool replace'.
> see: http://www.sun.com/msg/ZFS-8000-9P
> scrub: none requested
> config:
>
> NAME STATE READ WRITE CKSUM
> storage ONLINE 0 0 0
> raidz2 ONLINE 0 0 0
> gpt/disk01-live ONLINE 0 0 0
> gpt/disk02-live ONLINE 0 0 0
> gpt/disk03-live ONLINE 0 0 0
> gpt/disk04-live ONLINE 0 0 0
> gpt/disk05-live ONLINE 0 0 0
> gpt/disk06-live ONLINE 0 0 12
> gpt/disk07-live ONLINE 0 0 0

Overnight, the following appeared in /var/log/messages:

Oct  2 21:56:46 kraken root: ZFS: checksum mismatch, zpool=storage 
path=/dev/gpt/disk06-live offset=123103157760 size=1024
Oct  2 21:56:47 kraken root: ZFS: checksum mismatch, zpool=storage 
path=/dev/gpt/disk06-live offset=123103159808 size=1024
Oct  2 21:56:47 kraken root: ZFS: checksum mismatch, zpool=storage 
path=/dev/gpt/disk06-live offset=123103164416 size=512
Oct  2 21:56:47 kraken root: ZFS: checksum mismatch, zpool=storage 
path=/dev/gpt/disk06-live offset=123103162880 size=512
Oct  2 23:00:58 kraken root: ZFS: checksum mismatch, zpool=storage 
path=/dev/gpt/disk06-live offset=1875352305152 size=1024
Oct  3 02:44:55 kraken root: ZFS: checksum mismatch, zpool=storage 
path=/dev/gpt/disk06-live offset=1914424351744 size=512
Oct  3 03:01:01 kraken root: ZFS: checksum mismatch, zpool=storage 
path=/dev/gpt/disk06-live offset=1875175041536 size=512
Oct  3 03:01:02 kraken root: ZFS: checksum mismatch, zpool=storage 
path=/dev/gpt/disk06-live offset=1886724290048 size=1024
Oct  3 04:05:44 kraken root: ZFS: checksum mismatch, zpool=storage 
path=/dev/gpt/disk06-live offset=1953680806912 size=512
Oct  3 04:05:44 kraken root: ZFS: checksum mismatch, zpool=storage 
path=/dev/gpt/disk06-live offset=1953680807424 size=512
Oct  3 04:05:44 kraken root: ZFS: checksum mismatch, zpool=storage 
path=/dev/gpt/disk06-live offset=1953680807936 size=512
Oct  3 04:05:44 kraken root: ZFS: checksum mismatch, zpool=storage 
path=/dev/gpt/disk06-live offset=1953680808448 size=512
Oct  3 04:59:38 kraken root: ZFS: checksum mismatch, zpool=storage 
path=/dev/gpt/disk06-live offset=98172631552 size=512
Oct  3 04:59:38 kraken root: ZFS: checksum mismatch, zpool=storage 
path=/dev/gpt/disk06-live offset=98172729856 size=512
Oct  3 04:59:38 kraken root: ZFS: checksum mismatch, zpool=storage 
path=/dev/gpt/disk06-live offset=98172730368 size=512
Oct  3 04:59:38 kraken root: ZFS: checksum mismatch, zpool=storage 
path=/dev/gpt/disk06-live offset=98172730880 size=512
Oct  3 04:59:38 kraken root: ZFS: checksum mismatch, zpool=storage 
path=/dev/gpt/disk06-live offset=98172731392 size=512

Given the outage from yesterday when ada0 was offline for several hours, 
I'm guessing that checksum mismatches on that drive are expected.  Yes, 
/dev/gpt/disk06-live == ada0.

The current zpool status is:

$ zpool status
   pool: storage
  state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
         attempt was made to correct the error.  Applications are 
unaffected.
action: Determine if the device needs to be replaced, and clear the errors
         using 'zpool clear' or replace the device with 'zpool replace'.
    see: http://www.sun.com/msg/ZFS-8000-9P
  scrub: resilver completed after 0h1m with 0 errors on Sun Oct  3 
00:01:17 2010
config:

         NAME                 STATE     READ WRITE CKSUM
         storage              ONLINE       0     0     0
           raidz2             ONLINE       0     0     0
             gpt/disk01-live  ONLINE       0     0     0
             gpt/disk02-live  ONLINE       0     0     0
             gpt/disk03-live  ONLINE       0     0     0
             gpt/disk04-live  ONLINE       0     0     0
             gpt/disk05-live  ONLINE       0     0     0
             gpt/disk06-live  ONLINE       0     0    25  778M resilvered
             gpt/disk07-live  ONLINE       0     0     0

errors: No known data errors


-- 
Dan Langille - http://langille.org/