moutnroot failing on zpools in Azure after upgrade from 10 to 11 due to lack of waiting for da0

[Warner Losh]

On Thu, Mar 16, 2017 at 6:06 AM, Pete French <petefrench at ingresso.co.uk> wrote:
>> I don't like the delay and retry approach at all.
>
> Its not ideal, but it is what we do for UFS after all...
>
>> Imagine that you told the kernel that you want to mount your root from a ZFS
>> pool which is on a USB driver which you have already thrown out.  Should the
>> kernel just keep waiting for that pool to appear?
>
> I'm not talking about an infinite loop here, just making it honour
> the 'vfs.mountroot.timeout' setting like it does ofr UFS. So it
> should wait for the timeout I have set and then proceed as it would if
> there had been no timeout. Default behaviout is for it to behave as it
> does now, its onyl when you need the retry that you enable it.

Put another way: With UFS is keeps retrying until the timeout expires.
If the first try succeeds, the boot is immediate.

> Right now this works for UFS, but not for ZFS, which is an inconsistency
> that I dont like, and also means I am being forced down a UFS root
> path if I require this.

Yes. ZFS is special, but I don't think the assumptions behind its
specialness are quite right:

        /*
         * In case of ZFS and NFS we don't have a way to wait for
         * specific device.  Also do the wait if the user forced that
         * behaviour by setting vfs.root_mount_always_wait=1.
         */
        if (strcmp(fs, "zfs") == 0 || strstr(fs, "nfs") != NULL ||
            dev[0] == '\0' || root_mount_always_wait != 0) {
                vfs_mountroot_wait();
                return (0);
        }

So you can make it always succeed by forcing the wait, but that's lame...