gptzfsboot targeting wrong vdev

Tue Jul 7 14:59:32 UTC 2020

On 2020-07-07 02:21, Andriy Gapon wrote:
> On 06/07/2020 16:10, Christian Kratzer wrote:
>> Hi,
>>
>> I have a couple of freebsd based zfs servers on 12.1-RELEASE-p6 that have zfs
>> root and separate geli procted zfs data pools.
>>
>> I have been booting these off usb sticks or sdcards with a copy of /boot
>> and the geli keys and then subsequently mounting root from the zfsroot pool.
>>
>>     root at zfs1:/home/ck # zpool list
>>     NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH 
>> ALTROOT
>>     zp1    3.62T   242G  3.39T        -         -     3%     6%  1.00x  ONLINE  -
>>     zp2    21.8T  13.2T  8.51T        -         -    13%    60%  1.00x  ONLINE  -
>>     zroot   127G   112G  15.1G        -         -    30%    88%  1.00x  ONLINE  -
>>     root at zfs1:/home/ck #
>>
>> In above setup zroot is an unencrypted two ssd vdev with just the OS on it.
>>
>> The disks for zp1 and zp2 are encrypted using geli via keys provided in
>> loader.conf.
>>
>> Above setup is working fine whilst I keep /boot on the usb stick.
>>
>> I have tried again to get rid of the usb stick and booting directly from the
>> mirror on zroot.
>>
>> I have following in my loader.conf
>>
>>     # zfs
>>     zfs_load="YES"
>>     vfs.root.mountfrom="zfs:zroot/ROOT/default"
>>     vfs.zfs.boot.primary_pool="17582686064334413803"
>>     vfs.zfs.boot.primary_vdev="9317482241260982593"
>>
>> I got the guid values using
>>
>>     root at zfs1:/usr/src # zpool get guid zroot
>>     NAME   PROPERTY  VALUE  SOURCE
>>     zroot  guid      17582686064334413803  default
>>
>>     root at zfs1:/usr/src # zfs get guid zroot/ROOT/default
>>     NAME                PROPERTY  VALUE  SOURCE
>>     zroot/ROOT/default  guid      9317482241260982593  -
>>
>>
>> I also have following setup on the devices that are part of zroot
>>
>>     root at zfs1:/home/ck # gpart show ada0
>>     =>       40  468862048  ada0  GPT  (224G)
>>          40       1024     1  freebsd-boot  (512K)
>>            1064  134217728     2  freebsd-swap  (64G)
>>       134218792   33554432     3  freebsd-zfs  (16G)
>>       167773224   33554432     4  freebsd-zfs  (16G)
>>       201327656  267534424     5  freebsd-zfs  (128G)
>>       468862080          8        - free -  (4.0K)
>>
>>     root at zfs1:/home/ck #
>>
>> When booting from ada0 I get following:
>>
>>     ZFS: i/o error - all block copies unavailable
>>     ZFS: can't read MOS of pool zp1
>>     gptzfsboot: failed to mount default pool zp1
>>
>>     FreeBSD/x86 boot
>>

So, just to be clear, at this point you have not loaded the boot loader
yet. You are in the bootstrap (gptzfsboot), and it is unable to load the
loader.

I think it just looks at the first 'freebsd-zfs' type'd partition.
However, if zp1 is GELI encrypted, it shouldn't be able to even tell the
name of the pool.

You might try changing the partition type of the boots you are not
booting from, to 'freebsd-vinum' or something other than 'freebsd-zfs'
so that gptzfsboot only sees 1 'freebsd-zfs' to boot from.

>> At least it seems that my bios is loading the zfs loader from the correct gpt
>> partition.
>>
>> But I cannot figure out why it is still trying to boot from zp1 and not zroot.
>>
>> I have not been able to break out into the loader via serial console to inspect
>> if those vfs values are really there.
>>
>> I do see them if I boot via my usb stick copy of /boot
>>
>>     root at zfs1:/home/ck # kenv | grep vfs
>>     vfs.root.mountfrom="zfs:zroot/ROOT/default"
>>     vfs.zfs.boot.primary_pool="17582686064334413803"
>>     vfs.zfs.boot.primary_vdev="9317482241260982593"
>>     root at zfs1:/home/ck #
>>
>> This should all be unrelated to geli as I only need the zroot pool mounted at
>> this point in time which is unencrypted.
>>
>> I am a bit lost at this stage and would have following questions if anybody can
>> help me:
>>
>> 1. Am I getting the correct guid values for the primary_pool and _vdev or am I
>> misundestanding the requirements ?
> 
> You are.  Your vdev guid is obviously incorrect.  I am not sure how you decided
> that a dataset guid is a vdev guid.
> Also, those variables are for internal use only.  They are a part of
> communication between an early boot block and loader.
> 
>> 2. How can I break into the loader to inspect or set kenv values ?
> 
> That's an interesting question.
> I always see a menu and simply press a button to go the loader prompt.
> Maybe you wanted to ask something else?
> Do you see anything from loader on serial console at all?
> Do you have loader serial console support enabled?
> 
>> 3. Should I perhaps try exporting zp1 and zp2 so as only to have zroot visible ?
> 
> gptzfsboot(8) describes the pool discovery algorithm.
> 
>> 4. Can I try setting the vfs variables from /config ? How would the syntax for
>> that be ?
> 
> Assuming you mean /boot.config or /boot/config -- no.  You can read boot(8)
> about all supported options.  It could be helpful with respect to serial console
> as well.
> 
>> Anything else I can do or check to get this to work ?
> 
> There are no that many things that can be controlled with respect to ZFS boot.
> BIOS boot device priority and order of partitions is what controls which pool
> should be used for boot.
> 
> You provided output of gpart show ada0, there are more than zfs partitions in
> it.  You did not explain which one is which.
> 

-- 
Allan Jude

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 834 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20200707/c2cc5b81/attachment.sig>