ZFS parallel mounting gone wrong?
Trond Endrestøl
Trond.Endrestol at fagskolen.gjovik.no
Wed Apr 17 09:12:42 UTC 2019
On Mon, 15 Apr 2019 15:24+0200, Trond Endrestøl wrote:
> I upgraded a non-critical system running amd64 stable/12 to r346220.
>
> During multiuser boot not all ZFS filesystems below zroot/usr/local
> was mounted.
Some more explaining is in order:
This system has two 7 year old pools that complement each other.
/usr/local comes mostly from the zroot pool, but other children comes
from a zdata pool. The intermediary filesystems have their canmount
property set to off and mountpoints are specified at the top level
only. The same goes for other parts of the filesystem hierarchy, such
as /var/db and /var/spool.
I just upgraded to stable/12 global r346269, local r346268. During
a routine "zfs mount -av" performed in single user mode, the kernel
proceeded to mount a child filesystem (enterprise_zdata/var/db/mysql)
without the parent filesystems being mounted first.
I rebooted back to r345628 from March 28th, and this kernel has no
problems correctly mounting the ZFS filesystems in parallel. That BE
used LLVM 7.0.1 from base as its system compiler.
Rebooting into r346220 (April 15th) or r346269 (April 17th) clearly
shows problems mounting filesystems in the correct order. These BEs
was compiled using LLVM 8.0.0 from base.
Maybe the system compiler is irrelevant.
The name of the pools might also be a factor, the zdata pool preceedes
the zroot pool in alphanumerical order.
Maybe there is a bug in the code, or the code breaks when parts of the
filesystem hierarchy is being built from multiple pools.
Here's an attempt at explaining how this fits together:
zfs list -ro name,canmount,mountpoint enterprise_zroot/usr enterprise_zdata/usr enterprise_zroot/var enterprise_zdata/var
[the list has been slightly edited, moving zdata below zroot and adding an empty line]
NAME CANMOUNT MOUNTPOINT
enterprise_zroot/usr off /usr
enterprise_zroot/usr/compat on /usr/compat
enterprise_zroot/usr/local on /usr/local
enterprise_zroot/usr/local/certs on /usr/local/certs
enterprise_zroot/usr/local/etc on /usr/local/etc
enterprise_zroot/usr/local/etc/shellkonfig3 on /usr/local/etc/shellkonfig3
enterprise_zroot/usr/local/etc/shellkonfig3/head on /usr/local/etc/shellkonfig3/head
enterprise_zroot/usr/local/etc/shellkonfig3/stable-10 on /usr/local/etc/shellkonfig3/stable-10
enterprise_zroot/usr/local/etc/shellkonfig3/stable-11 on /usr/local/etc/shellkonfig3/stable-11
enterprise_zroot/usr/local/etc/shellkonfig3/stable-8 on /usr/local/etc/shellkonfig3/stable-8
enterprise_zroot/usr/local/etc/shellkonfig3/stable-9 on /usr/local/etc/shellkonfig3/stable-9
enterprise_zroot/usr/local/info on /usr/local/info
enterprise_zroot/usr/local/var on /usr/local/var
enterprise_zroot/usr/obj on /usr/obj
enterprise_zroot/usr/ports on /usr/ports
enterprise_zroot/usr/ports/distfiles on /usr/ports/distfiles
enterprise_zroot/usr/ports/local off /usr/ports/local
enterprise_zroot/usr/ports/packages on /usr/ports/packages
enterprise_zroot/usr/ports/workdirs on /usr/ports/workdirs
enterprise_zroot/usr/src on /usr/src
enterprise_zdata/usr off /usr
enterprise_zdata/usr/local off /usr/local
enterprise_zdata/usr/local/moodledata on /usr/local/moodledata
enterprise_zdata/usr/local/pgsql on /usr/local/pgsql
enterprise_zdata/usr/local/restaurering on /usr/local/restaurering
enterprise_zdata/usr/local/www on /usr/local/www
enterprise_zdata/usr/local/www/moodle on /usr/local/www/moodle
enterprise_zroot/var off /var
enterprise_zroot/var/Named on /var/Named
enterprise_zroot/var/account on /var/account
enterprise_zroot/var/audit on /var/audit
enterprise_zroot/var/cache off /var/cache
enterprise_zroot/var/cache/ccache on /var/cache/ccache
enterprise_zroot/var/cache/synth on /var/cache/synth
enterprise_zroot/var/crash on /var/crash
enterprise_zroot/var/db on /var/db
enterprise_zroot/var/db/darkstat on /var/db/darkstat
enterprise_zroot/var/db/dkim on /var/db/dkim
enterprise_zroot/var/db/etcupdate on /var/db/etcupdate
enterprise_zroot/var/db/hyperv on /var/db/hyperv
enterprise_zroot/var/db/ntp on /var/db/ntp
enterprise_zroot/var/db/pkg on /var/db/pkg
enterprise_zroot/var/db/ports on /var/db/ports
enterprise_zroot/var/db/sup on /var/db/sup
enterprise_zroot/var/empty on /var/empty
enterprise_zroot/var/log on /var/log
enterprise_zroot/var/mail on /var/mail
enterprise_zroot/var/munin on /var/munin
enterprise_zroot/var/run on /var/run
enterprise_zroot/var/spool on /var/spool
enterprise_zroot/var/spool/cvsup on /var/spool/cvsup
enterprise_zroot/var/synth on /var/synth
enterprise_zroot/var/synth/builders on /var/synth/builders
enterprise_zroot/var/synth/live_packages on /var/synth/live_packages
enterprise_zroot/var/tmp on /var/tmp
enterprise_zroot/var/unbound on /var/unbound
enterprise_zdata/var off /var
enterprise_zdata/var/db off /var/db
enterprise_zdata/var/db/mysql on /var/db/mysql
enterprise_zdata/var/db/mysql_secure on /var/db/mysql_secure
enterprise_zdata/var/db/mysql_tmpdir on /var/db/mysql_tmpdir
enterprise_zdata/var/db/postgres on /var/db/postgres
enterprise_zdata/var/db/postgres/data11 on /var/db/postgres/data11
enterprise_zdata/var/db/postgres/data11/base on /var/db/postgres/data11/base
enterprise_zdata/var/db/postgres/data11/pg_wal on /var/db/postgres/data11/pg_wal
enterprise_zdata/var/db/postgres/data96 on /var/db/postgres/data96
enterprise_zdata/var/db/postgres/data96/base on /var/db/postgres/data96/base
enterprise_zdata/var/db/postgres/data96/pg_xlog on /var/db/postgres/data96/pg_xlog
enterprise_zdata/var/db/prometheus on /var/db/prometheus
enterprise_zdata/var/db/prometheus/data on /var/db/prometheus/data
enterprise_zdata/var/db/prometheus/data/wal on /var/db/prometheus/data/wal
enterprise_zdata/var/spool off /var/spool
enterprise_zdata/var/spool/bareos on /var/spool/bareos
enterprise_zdata/var/spool/ftp on /var/spool/ftp
Using this remount script in singleuser mode, brings order to chaos:
#!/bin/sh
# To be run while in singleuser mode,
# preferably (re)booted directly to singleuser mode.
PATH="/bin:/sbin:/usr/bin:/usr/sbin:/rescue"
export PATH
killall devd
killall moused
umount /usr/compat/linux/dev/fd
umount /usr/compat/linux/dev
umount /usr/compat/linux/proc
umount /usr/compat/linux/sys
zfs unmount -a
for fs in `zfs list -Hro canmount,name enterprise_zroot | grep -v '^off' | grep -v 'enterprise_zroot$' | grep -v 'enterprise_zroot/ROOT' | grep -v 'enterprise_zroot/do-not-destroy' | awk '{print $2}'`; do
zfs mount ${fs}
done
for fs in `zfs list -Hro canmount,name enterprise_zdata | grep -v '^off' | grep -v 'enterprise_zdata$' | grep -v 'enterprise_zdata/do-not-destroy' | awk '{print $2}'`; do
zfs mount ${fs}
done
mount -al
echo "You may now attempt to exit to multiuser mode ..."
# EOF
-- Trond.
More information about the freebsd-stable
mailing list