Mounting from zfs:zroot/ROOT/default failed with error 2: unknown file system after upgrade to 10.1-RELEASE

Wed Nov 19 17:18:01 UTC 2014

On Wed, 19 Nov 2014 17:10+0100, Anton Eliasson wrote:

> Den 2014-11-19 08:22, Trond Endrestøl skrev:
> > On Tue, 18 Nov 2014 23:05+0100, Anton Eliasson wrote:
> > 
> > > Hi!
> > > After upgrading a server from FreeBSD 10.0-RELEASE to 10.1-RELEASE it no
> > > longer starts up. The bootloader shows its usual menu and then proceeds to
> > > load zfs.ko, opensolaris.ko and some third module. Then kernel boots and
> > > continues all the way to mountroot where it fails with the message
> > > "Mounting
> > > from zfs:zroot/ROOT/default failed with error 2: unknown file system." The
> > > entire message:
> > > https://cloud.solid.lth.se/public.php?service=files&t=27c15a131ca63b19f619a7ee558fe8d1
> > > 
> > > Entering zfs:zroot, zfs:zroot/ROOT, zfs:zroot/ROOT/default or
> > > zfs:anythingelse
> > > results in the same error.
> > > 
> > > FreeBSD 10.0 was installed this summer. I believe I used the installer's
> > > guided procedure for installing on a mirrored ZFS root. Each drive has a
> > > GPT
> > > table with a 512k freebsd-boot, a freebsd-swap and a freebsd-zfs
> > > partition. No
> > > separate /boot.
> > > 
> > > There is no vfs.root.mountfrom directive in /boot/loader.conf and no / in
> > > /etc/fstab. If I set vfs.root.mountfrom="zfs:zroot" it fails with the same
> > > message but a slightly different path (zfs:zroot).
> > > 
> > > The motherboard is a Supermicro X9SRH-7TF. I can import and mount the root
> > > filesystem with a FreeBSD live CD without problems.
> > > 
> > > Some config files pulled from a very similar installation (same server
> > > model,
> > > installed at the same time). They should be more or less identical on the
> > > server that won't boot:
> > > 
> > > /etc/rc.conf:
> > > 
> > >      hostname="..."
> > >      keymap="swedish.iso.kbd"
> > >      # Use synchronous DHCP; pause boot until DHCP is completed.
> > >      ifconfig_ix0="SYNCDHCP"
> > >      sshd_enable="YES"
> > >      ntpd_enable="YES"
> > >      # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
> > >      dumpdev="AUTO"
> > >      zfs_enable="YES"
> > > 
> > >      # IPFW
> > >      [...]
> > > 
> > > /boot/loader.conf:
> > > 
> > >      zfs_load="YES"
> > >      geom_mirror_load="YES"
> > > 
> > > /etc/fstab:
> > > 
> > >      fdesc    /dev/fd        fdescfs        rw    0    0
> > >      /dev/mirror/swap none swap sw 0 0
> > > 
> > > Thanks!
> > According to the screenshot, the bootfs property on zroot is set to
> > zroot/ROOT/default. Does that dataset exist?
> > 
> > While at the loader prompt, what does these commands tell you?
> > 
> >    lszfs zroot
> $MOS
> $FREE
> $ORIGIN
> ROOT
> tmp
> usr
> var
> >    lszfs zroot/ROOT
> default
> 
> And `lszfs zroot/ROOT/default` shows nothing.
> > If you see some other potential dataset, try setting currdev using
> > this as a template before booting:
> > 
> > set currdev="zfs:zroot/ROOT/some-other-dataset:"
> >                 ^                             ^
> >                     These :'s are important.
> `show currdev` shows that it is already set to zfs:zroot/ROOT/default: (with a
> colon at the end)
> > 
> > Next, unload the old stuff, and try to load the new stuff:
> > 
> > unload
> > load /boot/kernel/kernel
> > load /boot/kernel/opensolaris.ko
> > load /boot/kernel/zfs.ko
> It looks like it loads the kernel, but then it fails on the following two
> commands because the kernel is not loaded:
> https://cloud.solid.lth.se/public.php?service=files&t=5028527feac7982c76206b691212d15b
> https://cloud.solid.lth.se/public.php?service=files&t=0c0d50aac583167a11a8eaa4de5f08ce
> 
> Doing boot -s after that gives the same error as before.
> 
> However, loading the old kernel with the following commands works better:
> 
> unload
> load /boot/kernel.old/kernel
> load /boot/kernel.old/opensolaris.ko
> load /boot/kernel/zfs.ko
> lsmod
> 
> Screenshots:
> https://cloud.solid.lth.se/public.php?service=files&t=9c6c1bbc96b2c816aae05d2718215e2a
> https://cloud.solid.lth.se/public.php?service=files&t=8aee8df4bacf65b9ceae02fa65b53711
> 
> Issuing boot -s after that shows some text and then suddenly reboots the
> computer. I can see "ZFS filesystem version 5" in the output so it looks like
> it is successfully loaded.

To break into the kernel debugger, enter this before booting the 
kernel:

set debug.debugger_on_panic="1"

Judging from the screenshots you've provided and from what you wrote 
in your reply, it seems vfs.root.mountfrom is overridden somehow. 
/boot/loader.conf would be a natural candidate.

If the bootfs property is indeed set to zroot/ROOT/default, then it's 
very strange that the kernel would proclaim:

Trying to mount root from zfs:zroot []...

instead of:

Trying to mount root from zfs:zroot/ROOT/default []...

And it's even stranger that nothing appears on lszfs' radar when asked 
to list zroot/ROOT/default.

> Selecting the old kernel in the boot menu and trying to boot normally fails
> because init is not found:
> https://cloud.solid.lth.se/public.php?service=files&t=81fd35bb6348ae7426553c2f59fb2a18
> 
> Could this be a mismatch in kernel and kernel module versions? How would I fix
> that?

If you can boot from a recent snapshot image or live image, and import 
the zpool read-only, you could walk around and see if 
zroot/ROOT/default/boot/loader.conf contains anything strange.

This zpool command would allow you to go spelunking inside /zroot:

  zpool import -o readonly=on -f -R /zroot zroot

You might need to mount readonly any dataset you wish to examine.

If you see the need for changing any of the files, you must export the 
zpool and re-import it readwrite, i.e. leave out the -o readonly=on 
option.

I admit I'm in unknown territory, even if I have managed multiple ZFS 
systems over the past years, so be careful.

-- 
+-------------------------------+------------------------------------+
| Vennlig hilsen,               | Best regards,                      |
| Trond Endrestøl,              | Trond Endrestøl,                   |
| IT-ansvarlig,                 | System administrator,              |
| Fagskolen Innlandet,          | Gjøvik Technical College, Norway,  |
| tlf. mob.   952 62 567,       | Cellular...: +47 952 62 567,       |
| sentralbord 61 14 54 00.      | Switchboard: +47 61 14 54 00.      |
+-------------------------------+------------------------------------+