Wiki / ZFS Booting

[Cody Ritts]

Hello all,

I have some suggestions for the wiki regarding root on zfs.

In short, I think it would be EXTREMELY helpful to consolidate all of 
the pages into two.

1.  Booting to ZFS
2.  Advanced ZFS partitioning

The long story is that I have just spent several days documenting my 
efforts to get multiple systems to boot from ZFS.  First, thank you to 
all who have contributed to the wiki, I would not have been able to get 
it done without you.

That being said these are 11 pages I have found on the wiki:

/RootOnZFS, /RootOnZFS/UFSBoot, /RootOnZFS/ZFSBootPartition, 
/RootOnZFS/ZFSBootSlice, /RootOnZFS/GPTZFSBoot, 
/RootOnZFS/GPTZFSBoot/Mirror, /RootOnZFS/GPTZFSBoot/RAIDZ1, 
/RootOnZFS/GPTZFSBoot/RAIDZ2, /RootOnZFS/GPTZFSBoot/9.0-RELEASE, 
/ZFSOnRoot, /ZFSOnRootWithZFSboot

I have read all of those, plus another half dozen on the internet. While 
the steps are largely the same, they all seem to have their own little 
differences.  I have copied and pasted the commands from more than one 
of them into virtual machines, and they would not boot.

My overall experience was extremely draining.  I expected that when I am 
copying and pasting commands from a wiki, and it does not boot, the 
problem is not with the commands.  When there are different commands 
being run on one example from the other, I have to go research which one 
is right.  I spent a lot of time chasing my tail.

I don't want to just complain that it should be better, I would like to 
share the internal documentation that I put together from all of the web 
pages, man pages, and server testing.

I am not 100% certain I am doing everything the "right" way, but I am 
100% CERTAIN that I have COPY/PASTED each and every one of these 
commands into a FreeBSD-9.1-RELEASE LiveCD shell and then successfully 
booted.

These commands cover all of the information that I needed to know that 
was spread out across the internet.  Hopefully someone can find this 
information useful who has the time, inclination, and write access to 
the wiki.

Cheers,

Cody


###### Boot to LiveCD on 9.1-RELEASE media ######
## -- I like copy/paste accuracy, so I enable sshd on the liveCD
dhclient em0
mkdir /tmp/etc
mount_unionfs /tmp/etc /etc
sed -i '' 's/#PermitRootLogin no/PermitRootLogin yes/g' /etc/ssh/sshd_config
service sshd onestart
passwd

###### GPT INSTALL ######
gpart create -s gpt ada0
gpart add -s 122 -t freebsd-boot -l boot0 ada0
gpart add -s 2048M -t freebsd-swap -l swap0 ada0
gpart add -t freebsd-zfs -l disk0 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
zpool create -R /mnt -O canmount=off -o cachefile=/tmp/zpool.cache zroot 
/dev/gpt/disk0

###### MBR INSTALL ######
gpart create -s mbr ada0
gpart add -t freebsd ada0
gpart create -s bsd ada0s1
gpart add -s 51G -t freebsd-zfs  ada0s1
gpart add -s 8G  -t freebsd-swap ada0s1
glabel label disk0 /dev/ada0s1a
glabel label swap  /dev/ada0s1b
gpart set -a active -i 1 ada0
gpart bootcode -b /boot/mbr ada0
dd if=/boot/zfsboot of=/dev/ada0s1 count=1
dd if=/boot/zfsboot of=/dev/ada0s1a skip=1 seek=1024
zpool create -R /mnt -O canmount=off -o cachefile=/tmp/zpool.cache zroot 
/dev/label/disk0

###### ZRAID/MIRROR GPT INSTALL ######
gpart create -s gpt ada0
gpart add -s 122 -t freebsd-boot -l boot0 ada0
gpart add -s 2048M -t freebsd-swap -l swap0 ada0
gpart add -t freebsd-zfs -l disk0 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gpart create -s gpt ada1
gpart add -s 122 -t freebsd-boot -l boot1 ada1
gpart add -s 2048M -t freebsd-swap -l swap1 ada1
gpart add -t freebsd-zfs -l disk1 ada1
## -- repeat for adaN
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
gmirror label -b load -F swap /dev/gpt/swap0 /dev/gpt/swap1
zpool create -R /mnt -O canmount=off -o cachefile=/tmp/zpool.cache zroot 
mirror /dev/gpt/disk0 /dev/gpt/disk1
## -- zpool <options> create zroot [mirror|raidz1|raidz2|raidz3] 
/dev/gpt/disk0 ... /dev/gpt/diskN

###### COMMON INSTALL COMMANDS #########
zfs set mountpoint=/zroot  zroot
zfs set checksum=fletcher4 zroot
zfs create -o mountpoint=legacy zroot/ROOT
mount -t zfs zroot/ROOT /mnt

tar -C /mnt -xppf /usr/freebsd-dist/base.txz
tar -C /mnt -xppf /usr/freebsd-dist/kernel.txz

zpool set bootfs=zroot/ROOT zroot
zfs set freebsd:boot-environment=1 zroot/ROOT
cp /tmp/zpool.cache /mnt/boot/zfs/

echo 'hostname="host"'  >> /mnt/etc/rc.conf
echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
echo 'zfs_load="YES"'                      >> /mnt/boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot/ROOT"' >> /mnt/boot/loader.conf

###### GPT INSTALL ######
cat << EOF > /mnt/etc/fstab
# Device                       Mountpoint              FStype  Options 
      Dump    Pass#
/dev/gpt/swap0                 none                    swap    sw  0       0
EOF

###### MBR INSTALL ######
cat << EOF > /mnt/etc/fstab
# Device                       Mountpoint              FStype  Options 
      Dump    Pass#
/dev/label/swap                none                    swap    sw  0       0
EOF

###### ZRAID/MIRROR GPT INSTALL ######
echo 'geom_mirror_load="YES"'   >> /mnt/boot/loader.conf
cat << EOF > /mnt/etc/fstab
# Device                       Mountpoint              FStype  Options 
      Dump    Pass#
/dev/mirror/swap                 none                    swap    sw 
  0       0
EOF

###### CHROOT INSTALL COMMANDS ######
chroot /mnt
passwd root
tzsetup
cd /etc/mail
make aliases
exit
reboot


######  DISCUSSION  ######

## -- notes on swap --
swap can go on ZFS filesystem but can no longer be used as a dump device.

## -- mountpoint=legacy for zroot/ROOT
release/9.1.0/tools/tools/zfsboottest/zfsboottest.sh
"Dataset's mountpoint property should be set to 'legacy'"

I would imagine this protects you against importing the
pool on another system, and forgetting the altroot command.
When you are set legacy, you MUST either have:
echo vfs.root.mountfrom="zfs:zroot/ROOT"' >> /mnt/boot/loader.conf
   OR
an entry in fstab (I have not tested this)

## -- File system layout --
There are a lot of really cool things you can do with zfs layout
Please see [link to zfs Layout Wiki page]
The only advanced thing we did here is moved the root file system
so that /tmp /usr and /var do not inherit zfs commands from /

## -- notes on zpool.cache --
in 9.1-RELEASE -- without zpool.cache in this configuration, the the
system will NOT boot.

## -- Exporting zroot --
in 9.1-RELEASE  -- If you export zroot, your system will not boot.
You MUST boot from liveCD, and re-import it to get it to boot.