Fwd: ZFS Crash/Pool in unhealthy state

[Andriy Gapon]

On 17/06/2019 17:32, Larry Rosenman wrote:
> Forwarding to -Current as well, since it's a -current box
> 
> -------- Original Message --------
> Subject: ZFS Crash/Pool in unhealthy state
> Date: 06/17/2019 8:30 am
> From: Larry Rosenman <ler at lerctr.org>
> To: Freebsd fs <freebsd-fs at freebsd.org>
> 
> I had a power failure today and my big server no longer boots
> 
> If I try to import the pool using the memstick image and -F -f I get a panic
> (see  link).
> 
> https://www.lerctr.org/~ler/ZFS_CRASH.png <--- Panic
> 
> https://www.lerctr.org/~ler/ZFS_IMPORT.png <--- import tries.
> 
> Ideas?  Help?

My reading of it is that somehow you got a situation where a block that ZFS
considers as free is also queued for deferred freeing, which makes no sense.
So, as soon as ZFS starts processing the deferred frees it sees the
inconsistency and panics.  Usually, I tend to suspect the software first, but in
this case I see that the data has got other inconsistencies in the last TXG(s)
before the crash.  That's why import -F is required.
So, I suspect that either the storage controller is misconfigured with respect
to caches (its own and disks) or it otherwise misbehaves in that respect.

As to the recovery.  The proper way would be to import the pool read-only and
transfer the data to a new pool.
The more risky way would be to import the pool on a system where that
consistency check is simply disabled.  One way to do that is to clear bit 6
(mask 0x40) from ZFS debug flags.  Or to use a STABLE FreeBSD image for
an import.


-- 
Andriy Gapon