Re: difficulties replacing a ZFS installer zroot pool with a new zroot pool on a new disk

On 3/30/22 15:22, Russell L. Carter wrote:
> Greetings,
> I am going to top post because all of the previous discussion
> is moot given what I have found out to get the problem solved,
> where the problem is to simply replace a ZFS system pool's drive
> with a new drive.
> 
> To begin from the start:
> 
> I installed the new NVMe SSD drive and I was able to boot the USB
> install image and install a new FreeBSD system on it.  On reboot I
> first tried keeping the old SATA drive as it was.  However the
> motherboard BIOS (CSM enabled, legacy, ASUS Prime X570-PRO) refused
> all of my efforts to set the boot drive to the new SSD.  I finally
> resorted to disconnecting the data cable of the old SATA drive, and
> the new SSD booted fine.  I then powered down the motherboard,
> reattached the old SATA data cable, and booted.  The motherboard again
> refused to boot the new NVMe SSD.  After about an hour of fighting the
> BIOS, I gave up, set the SATA drive as "hot pluggable" in the BIOS,
> and rebooted with the SATA data cable disconnected.  Once the NVMe SSD
> was booted, I reattached the SATA data cable and it showed up in the
> 'zpool import' list.  'zpool import zroot' was not a happy solution as
> it collided with the new SSD zroot pool.
> 
> I eventually worked out that I should rename the old pool zroot.old on
> import.  That was also not a happy solution as it continued to
> automatically mount itself on top of the new SSD zroot pool.  I then
> worked out that I need to specify an altroot:
> 
> zpool import -o altroot=/mnt/zroot.old zroot.old
> 
> This at first glance appeared to work.  But it did NOT.  I was left with
> the complaint I made in my original email to the freebsd-questions list:
> 
> Where are my subsidiary datasets, and especially their data?
> zfs mount -a, for instance, did nothing, very quietly.
> 
> I tried zfs-mount(8)ing several of the subsidiary datasets (eg
> zroot.old/usr/src), and that worked!  But still I was missing some
> important stuff, like /root and /usr/local.  To get /root, I made
> a wild guess, and tried:
> 
> zfs mount zroot.old/ROOT/default
> 
> And that brought all my subsidiary datasets (and the data) back.
> 
> I would submit that looking back on 30+ years of successfully
> performing this exercise, that having the old SSD zroot.orig
> automatically import and mount everything, but not automatically
> import and mount everything when set given an altroot, is confusing.
> I would add that manually performing
> 
> zfs mount zroot.old/ROOT/default
> 
> (but not, say zfs unmount zroot.old/usr; zfs mount zroot.old/src)
> 
> mount everything I was missing, is also confusing and unintuitive.
> 
> I have two more ZFS system pools to upgrade to SSDs.  I am going to
> try the following procedure:
> 
> 1) unplug the old SATA drive and install FreeBSD to the new SSD.
> 
> 2) shutdown,  reattach old drive, reboot
>     If the new drive boots (instead of the old drive), I boot
>     to single user.  I then try
> 
>     zpool import -o altroot /mnt/zrool.old  zroot zroot.old
> 
> 3) If that works, I try:
> 
>     zfs mount zpool.old/ROOT/default
> 
> 4) If it doesn't work, I'll probably have to zpool export the pool and
>     iterate.  I forget exactly how I got the old pool renamed in the
>     above.
> 
>     Anyway, onwards.
> 
> Russell


I am glad you were able to navigate those obstacles and get FreeBSD 
running on NVMe.  The performance improvement must be dramatic.  :-)


I suggest that you try cloning the two more ZFS system pools from their 
old devices to their new devices (same or larger size) using a live OS 
instance that does not automatically react to ZFS.  (I believe the 
FreeBSD installer has this property, but am not certain.)


When I was cloning ZFS FreeBSD instances between SSD's and USB flash 
drives using MBR partitioning, often the target device would not boot. 
Solutions included connecting the target device to a different port and 
using the FreeBSD installer live system/ shell to delete 
/boot/zfs/zpool.cache on the target device.


David