From nobody Tue Mar 12 19:41:58 2024 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TvPDM47hjz5FMht for ; Tue, 12 Mar 2024 19:42:11 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-vk1-f173.google.com (mail-vk1-f173.google.com [209.85.221.173]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TvPDM2CL6z4PWK for ; Tue, 12 Mar 2024 19:42:11 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-vk1-f173.google.com with SMTP id 71dfb90a1353d-4d40ca07bf6so207745e0c.3 for ; Tue, 12 Mar 2024 12:42:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710272530; x=1710877330; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TACxOTWfbZQep58VclR1WoDJIq8AOWsZ+Xxu8CJa87s=; b=lDwusSIWL9xb16mK/FmSUi3KqP9D6dPJWhzT9xgJ1Sfwn8BgumqmaqLCNSHkfpGxGK lgETlXktf5lAf95CV187FYQK1yXJjXSWJhCz66guqr0y32tYCO5PmvNl6J6hZKbtRbC+ ERaCIkFbSN6ejZCJR4fojAuf2nJJnBzVqTZXuox02PGNGNCneKDqG5xhh8ET1d9Ykd95 VZS2WiAr+oBTBWmqJsMP62uG8NomGH/JmCbhOjRFuB+etsPA9W/uzN6IJZxFTW0+I5pr IiYa856wb4Pnbjwa+tVAehzeYeEZgVu/6iMBHBqHKkAQS0HAEiTCflNp5/atJ1miW0Uv I99Q== X-Gm-Message-State: AOJu0Yy3qJe7Uwwex+wZ698IQ7X8Dyeo6tExr7MRpKg3j7CpAoW7YTHO 3uwYzkGrjRf0YLk3Kdhpj5CLh3yJI+sBjR7c8MwtAUXs9tTa2YuSqiVnYg2Fx7eTZHo7BVA4RP4 Zw5OvcEPvQvoC8Qwt/XbT0EofuSQG3LhqTNQ= X-Google-Smtp-Source: AGHT+IGKz/CjGXTom1SC8kBpqWCR45Icx5E6TftVI3/SHYQhCMfisUBkHO8EobOgCKG3OqnbBTO6f+50AhPEbpCcZes= X-Received: by 2002:a05:6122:498d:b0:4c0:9ed8:57b3 with SMTP id ex13-20020a056122498d00b004c09ed857b3mr6181306vkb.1.1710272530442; Tue, 12 Mar 2024 12:42:10 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <8228ca0c-85a0-4436-aaf4-d2d987e0f5a4@blastwave.org> In-Reply-To: <8228ca0c-85a0-4436-aaf4-d2d987e0f5a4@blastwave.org> From: Alan Somers Date: Tue, 12 Mar 2024 13:41:58 -0600 Message-ID: Subject: Re: ZPool on iSCSI storage not available after a reboot To: Dennis Clarke Cc: current@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Rspamd-Queue-Id: 4TvPDM2CL6z4PWK On Tue, Mar 12, 2024 at 1:28=E2=80=AFPM Dennis Clarke wrote: > > > This is a somewhat odd problem and may have nothing to do with iSCSI > config at all. Suffice it to say that I have the following in the server > /etc/rc.conf : > > # > # the iSCSI initiator > iscsid_enable=3D"YES" > iscsictl_enable=3D"YES" > iscsictl_flags=3D"-Aa" > # > > During boot I see this on the console : > > > cannot import 'proteus': no suchpid 55 (zpool) is attempting to use > unsafe AIO requests - not logging anymore > pool or dataset > Destroy and re-create the pool from > a backup source. > cachefile import failed, retrying > no pools available to import > > > Sure enough the machine brings up a 10Gbit link with jumboframes *after* > the above messages : > > > ix0: flags=3D1008843 > metric 0 mtu 9000 > > options=3D4e53fbb > ether 8c:dc:d4:ae:18:b8 > inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255 > media: Ethernet autoselect (10Gbase-Twinax > ) > status: active > nd6 options=3D29 > > > Then a little later I see iscsi doing its goodness : > > > da0 at iscsi1 bus 0 scbus8 target 0 lun 0 > da0: Fixed Direct Access SPC-5 SCSI device > da0: Serial Number MYSERIAL0000 > da0: 150.000MB/s transfers > da0: Command Queueing enabled > da0: 2097152MB (4294967296 512 byte sectors) > add net ::0.0.0.0: gateway ::1 > Starting iscsid. > Starting iscsictl. > > The storage exists just fine and iSCSI seems to be doing its thing : > > root@titan:~ # > root@titan:~ # camcontrol devlist > at scbus0 target 0 lun 0 (pass0,ada0) > at scbus1 target 0 lun 0 (pass1,ada1) > at scbus2 target 0 lun 0 (ses0,pass2) > at scbus6 target 0 lun 0 (ses1,pass3) > at scbus7 target 0 lun 1 (pass4,nd= a0) > at scbus8 target 0 lun 0 (da0,pass5) > root@titan:~ # > root@titan:~ # gpart show da0 > =3D> 40 4294967216 da0 GPT (2.0T) > 40 8 - free - (4.0K) > 48 4294967200 1 freebsd-zfs (2.0T) > 4294967248 8 - free - (4.0K) > > root@titan:~ # > > However the zpool therein is not seen : > > root@titan:~ # > root@titan:~ # zpool list > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP > HEALTH ALTROOT > iota 7.27T 597G 6.68T - - 0% 8% 1.00x > ONLINE - > t0 444G 40.8G 403G - - 4% 9% 1.00x > ONLINE - > root@titan:~ # > > > Of course I can manually import it : > > > root@titan:~ # zpool import > pool: proteus > id: 15277728307274839698 > state: ONLINE > status: Some supported features are not enabled on the pool. > (Note that they may be intentionally disabled if the > 'compatibility' property is set.) > action: The pool can be imported using its name or numeric identifier, > though > some features will not be available without an explicit 'zpool > upgrade'. > config: > > proteus ONLINE > da0p1 ONLINE > root@titan:~ # > > It seems as if there is something out of sequence and the iSCSI > processes should be happening earlier in the boot process? I really do > not know and am wondering why that zpool proteus on the iSCSI storage > needs to be manually import'ed after a reboot. > > Any insights would be wonderful. > > -- > Dennis Clarke > RISC-V/SPARC/PPC/ARM/CISC > UNIX and Linux spoken Yes, this looks exactly like an ordering problem. zpools get imported early in the boot process, under the assumption that most of them are local. Networking comes up later, under the assumption that networking might require files that are mounted on ZFS. For you, I suggest setting proteus's cachefile to a non-default location and importing it from /etc/rc.local, like this: zpool set cachefile=3D/var/cache/iscsi-zpools.cache proteus Then in /etc/rc.local: zpool import -a -c /var/cache/iscsi-zpools.cache -o cachefile=3D/var/cache/iscsi-zpools.cache -Alan