Getting ZFS pools back.

[Willem Jan Withagen]

On 29/04/2018 20:21, Warner Losh wrote:
> 
> 
> On Sun, Apr 29, 2018 at 11:57 AM, Jan Knepper <jan at digitaldaemon.com 
> <mailto:jan at digitaldaemon.com>> wrote:
> 
>     On 04/29/2018 13:27, Willem Jan Withagen wrote:
> 
>         Trouble started when I installed (freebsd-update) 11.1 over a
>         running 10.4. Which is sort of scarry?
> 
>     This does sounds 'scary' as I am planning to do this in the (near)
>     future...
> 
>     Has anyone else experienced issues like this?
> 
>     Generally I do build the new system software on a running system,
>     but then go to single user mode to perform the actual install.
> 
>     I have done many upgrades like that over 18 or so years and never
>     seen or heard of an issue alike this.
> 
> 
> 11.x binaries aren't guaranteed to work with a 10.x kernel. So that's a 
> bit of a problem. freebsd-update shouldn't have let you do that either.
> 
> However, most 11.x binaries work well enough to at least bootstrap / fix 
> problems if booted on a 10.x kernel due to targeted forward 
> compatibility. You shouldn't count on it for long, but it generally 
> won't totally brick your box. In the past, and I believe this is still 
> true, they work well enough to compile and install a new kernel after 
> pulling sources. The 10.x -> 11.x syscall changes are such that you 
> should be fine. At least if you are on UFS.

I have been doing those kind of this for years and years. Even upgrading 
over NFS and stuff. Sometimes it is a bit too close to the sun and 
things burn. But never crash this bad.

> However, the ZFS ioctls and such are in the bag of 'don't specifically 
> guarantee and also they change a lot' so that may be why you can't mount 
> ZFS by UUID. I've not checked to see if there's specifically an issue 
> here or not. The ZFS ABI is somewhat more fragile than other parts of 
> the system, so you may have issues here.
> 
> If all else fails, you may be able to PXE boot an 11 kernel, or boot off 
> a USB memstick image to install a kernel.

Tried just about replace everything in both the boot-partition (First 
growing it to take > 64K gptzfsboot) and in /boot from the memstick.
But the error never went away.

Never had ZFS die on me this bad, that I could not get it back.

> Generally, while we don't guarantee forward compatibility (running newer 
> binaries on older kernels), we've generally built enough forward compat 
> so that things work well enough to complete the upgrade. That's why you 
> haven't hit an issue in 18 years of upgrading. However, the velocity of 
> syscall additions has increased, and we've gone from fairly stable 
> (stale?) ABIs for UFS to a more dynamic one for ZFS where backwards 
> compat is a bit of a crap shoot and forward compat isn't really there at 
> all. That's likely why you've hit a speed bump here.

Come to think of it, I did not do this step with freebsd-update, since I 
was not at an official release yet. I was going to 11.1-RELEASE, to be 
able to start using freebsd-update.

So I don't think I did just do that.... But I tried so much yesterday.
Normally I would installkernel, reboot, installworld, mergemaster, 
reboot for systems that are not up for freebsd-update.

--WjW