Re: ZPool on iSCSI storage not available after a reboot

Reply: Dennis Clarke : "Re: ZPool on iSCSI storage not available after a reboot"
In reply to: Dennis Clarke : "ZPool on iSCSI storage not available after a reboot"
Go to: [ bottom of page ] [ top of archives ] [ this month ]
From: Alan Somers <asomers_at_freebsd.org>
Date: Tue, 12 Mar 2024 19:41:58 UTC
On Tue, Mar 12, 2024 at 1:28 PM Dennis Clarke <dclarke@blastwave.org> wrote:
>
>
> This is a somewhat odd problem and may have nothing to do with iSCSI
> config at all. Suffice it to say that I have the following in the server
> /etc/rc.conf :
>
> #
> # the iSCSI initiator
> iscsid_enable="YES"
> iscsictl_enable="YES"
> iscsictl_flags="-Aa"
> #
>
> During boot I see this on the console :
>
>
> cannot import 'proteus': no suchpid 55 (zpool) is attempting to use
> unsafe AIO requests - not logging anymore
>   pool or dataset
>          Destroy and re-create the pool from
>          a backup source.
> cachefile import failed, retrying
> no pools available to import
>
>
> Sure enough the machine brings up a 10Gbit link with jumboframes *after*
> the above messages :
>
>
> ix0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP>
> metric 0 mtu 9000
>
> options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
>          ether 8c:dc:d4:ae:18:b8
>          inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
>          media: Ethernet autoselect (10Gbase-Twinax
> <full-duplex,rxpause,txpause>)
>          status: active
>          nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>
>
> Then a little later I see iscsi doing its goodness :
>
>
> da0 at iscsi1 bus 0 scbus8 target 0 lun 0
> da0: <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
> da0: Serial Number MYSERIAL0000
> da0: 150.000MB/s transfers
> da0: Command Queueing enabled
> da0: 2097152MB (4294967296 512 byte sectors)
> add net ::0.0.0.0: gateway ::1
> Starting iscsid.
> Starting iscsictl.
>
> The storage exists just fine and iSCSI seems to be doing its thing :
>
> root@titan:~ #
> root@titan:~ # camcontrol devlist
> <ST8000NM002A-2KE102 NN03>         at scbus0 target 0 lun 0 (pass0,ada0)
> <Samsung SSD 870 QVO 1TB SVQ01B6Q>  at scbus1 target 0 lun 0 (pass1,ada1)
> <AHCI SGPIO Enclosure 2.00 0001>   at scbus2 target 0 lun 0 (ses0,pass2)
> <AHCI SGPIO Enclosure 2.00 0001>   at scbus6 target 0 lun 0 (ses1,pass3)
> <SAMSUNG MZVKW512HMJP-000L7 6L6QCXA7>  at scbus7 target 0 lun 1 (pass4,nda0)
> <FREEBSD CTLDISK 0001>             at scbus8 target 0 lun 0 (da0,pass5)
> root@titan:~ #
> root@titan:~ # gpart show da0
> =>        40  4294967216  da0  GPT  (2.0T)
>            40           8       - free -  (4.0K)
>            48  4294967200    1  freebsd-zfs  (2.0T)
>    4294967248           8       - free -  (4.0K)
>
> root@titan:~ #
>
> However the zpool therein is not seen :
>
> root@titan:~ #
> root@titan:~ # zpool list
> NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP
> HEALTH  ALTROOT
> iota  7.27T   597G  6.68T        -         -     0%     8%  1.00x
> ONLINE  -
> t0     444G  40.8G   403G        -         -     4%     9%  1.00x
> ONLINE  -
> root@titan:~ #
>
>
> Of course I can manually import it :
>
>
> root@titan:~ # zpool import
>     pool: proteus
>       id: 15277728307274839698
>    state: ONLINE
> status: Some supported features are not enabled on the pool.
>          (Note that they may be intentionally disabled if the
>          'compatibility' property is set.)
>   action: The pool can be imported using its name or numeric identifier,
> though
>          some features will not be available without an explicit 'zpool
> upgrade'.
>   config:
>
>          proteus     ONLINE
>            da0p1     ONLINE
> root@titan:~ #
>
> It seems as if there is something out of sequence and the iSCSI
> processes should be happening earlier in the boot process? I really do
> not know and am wondering why that zpool proteus on the iSCSI storage
> needs to be manually import'ed after a reboot.
>
> Any insights would be wonderful.
>
> --
> Dennis Clarke
> RISC-V/SPARC/PPC/ARM/CISC
> UNIX and Linux spoken

Yes, this looks exactly like an ordering problem.  zpools get imported
early in the boot process, under the assumption that most of them are
local.  Networking comes up later, under the assumption that
networking might require files that are mounted on ZFS.  For you, I
suggest setting proteus's cachefile to a non-default location and
importing it from /etc/rc.local, like this:

zpool set cachefile=/var/cache/iscsi-zpools.cache proteus

Then in /etc/rc.local:
zpool import -a -c /var/cache/iscsi-zpools.cache -o
cachefile=/var/cache/iscsi-zpools.cache

-Alan