From nobody Fri Aug 11 03:32:51 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RMTrv6QKwz4m6Y2 for ; Fri, 11 Aug 2023 03:33:03 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: from mail-ot1-x32d.google.com (mail-ot1-x32d.google.com [IPv6:2607:f8b0:4864:20::32d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RMTrv4sQLz4Q0w for ; Fri, 11 Aug 2023 03:33:03 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ot1-x32d.google.com with SMTP id 46e09a7af769-6bca857accbso1454667a34.0 for ; Thu, 10 Aug 2023 20:33:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kev009.com; s=google; t=1691724782; x=1692329582; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=YkE9yQjZhDpfgVc2U517Q296S99Fx0L+uB+Kj1GJKT4=; b=q0eNGSiG3lBYqsjhnW+3hkCpoEqKA6B5/nY5GfPhq4EJYPAC/Ivp4N+sqIm8pe7brE 3XX/s2DvAFp1DK82entUcX+hzryX6n4y4rXbCewqgwmmc6qBhcozYVGzamq6VeFpbrJY WYBzzsD1eImGrpgOuY4+O2wH3p0uo9A5tlSec= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691724782; x=1692329582; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=YkE9yQjZhDpfgVc2U517Q296S99Fx0L+uB+Kj1GJKT4=; b=cvnJp2FntvymhwnAEBwvwGb7eCl0lE6SiYeumRFC55Hsu/84mRDqUErMmE2uxcgSLB BlsaPNNPHg/sn+wiXOwiS1uTQQQm5Ehaqbjhw/DhFB+stUtOxV7oWsvW8Gh1OPIGQjew Mamtafcf6HdwqAbneogU4QL0/6bnoiPOKNxX6HPNLZjc5HROlq5NRoPxOGBlmVMYudWw AWxiqP28WmoUx6Or4g2sHgAlV2UvLwNt+7fxSus5oFj/sYzfgzv4+WVooMTDALLSm8s1 C4qnBThPuWnCSFPnZ3dr5hCsmTIYBeoBkBtpSLQ6Px1yeY6ugLMXA3uuN+mbYufifijE ndFg== X-Gm-Message-State: AOJu0YwxsRKfgDNbFJ2lcCv438QibT7QYQ2RTVibmGaNQl1xm7/sZTnO XT+qvEkWEt/zYsL7NrPgOnh3hUAHeBZwG2tXWGnJKQ== X-Google-Smtp-Source: AGHT+IHxyyucuw3BYazqcIDHVy2wty/rL1oGJL5aTQPxA5wODFI72FrYJ8q68T7Ob8Jb9Y0xSj+RDbG/bc04M0HS10E= X-Received: by 2002:a05:6870:5b9b:b0:1bb:68ce:382c with SMTP id em27-20020a0568705b9b00b001bb68ce382cmr730924oab.8.1691724782486; Thu, 10 Aug 2023 20:33:02 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <86leeltqcb.fsf@ltc.des.no> <20230810133745.D0EC0178@slippy.cwsent.com> <20230810233312.0E10AF4@slippy.cwsent.com> In-Reply-To: <20230810233312.0E10AF4@slippy.cwsent.com> From: Kevin Bowling Date: Thu, 10 Aug 2023 20:32:51 -0700 Message-ID: Subject: Re: ZFS deadlock in 14 To: Cy Schubert Cc: =?UTF-8?Q?Dag=2DErling_Sm=C3=B8rgrav?= , current@freebsd.org Content-Type: multipart/alternative; boundary="000000000000b7db4b06029d5ea5" X-Rspamd-Queue-Id: 4RMTrv4sQLz4Q0w X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] --000000000000b7db4b06029d5ea5 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Spoke too soon still seeing zfs lockups under heavy poudriere workload after the MFVs. Regression time matches what has been reported here. On Thu, Aug 10, 2023 at 4:33 PM Cy Schubert wrote: > I haven't experienced any problems (yet) either. > > > -- > Cheers, > Cy Schubert > FreeBSD UNIX: Web: https://FreeBSD.org > NTP: Web: https://nwtime.org > > e^(i*pi)+1=3D0 > > > In message > om> > , Kevin Bowling writes: > > The two MFVs on head have improved/fixed stability with poudriere for > > me 48 core bare metal. > > > > On Thu, Aug 10, 2023 at 6:37=3DE2=3D80=3DAFAM Cy Schubert > > com> wrote: > > > > > > In message > > l.c > > > om> > > > , Kevin Bowling writes: > > > > Possibly > https://github.com/openzfs/zfs/commit/2cb992a99ccadb78d97049b4=3D > > 0bd4=3D3D > > > > 42eb4fdc549d > > > > > > > > On Tue, Aug 8, 2023 at 10:08=3D3DE2=3D3D80=3D3DAFAM Dag-Erling > Sm=3D3DC3=3D3DB8rg=3D > > rav > > > sd.org> wrote: > > > > > > > > > > At some point between 42d088299c (4 May) and f0c9703301 (26 June)= , > a > > > > > deadlock was introduced in ZFS. It is still present as of > 9c2823bae9=3D > > (4 > > > > > August) and is 100% reproducable just by starting poudriere bulk > in a > > > > > 16-core VM and waiting a few hours until deadlkres kicks in. In > the > > > > > latest instance, deadlkres complained about a bash process: > > > > > > > > > > #0 sched_switch (td=3D3D3Dtd@entry=3D3D3D0xfffffe02fb1d8000, > flags=3D > > =3D3D3Dflags@e=3D3D > > > > ntry=3D3D3D259) at /usr/src/sys/kern/sched_ule.c:2299 > > > > > #1 0xffffffff80b5a0a3 in mi_switch (flags=3D3D3Dflags@entry > =3D3D3D25=3D > > 9) at /u=3D3D > > > > sr/src/sys/kern/kern_synch.c:550 > > > > > #2 0xffffffff80babcb4 in sleepq_switch > (wchan=3D3D3D0xfffff818543a=3D > > 9e70, =3D3D > > > > pri=3D3D3D64) at /usr/src/sys/kern/subr_sleepqueue.c:609 > > > > > #3 0xffffffff80babb8c in sleepq_wait > (wchan=3D3D3D, p=3D > > ri=3D3D3D<=3D3D > > > > unavailable>) at /usr/src/sys/kern/subr_sleepqueue.c:660 > > > > > #4 0xffffffff80b1c1b0 in sleeplk (lk=3D3D3Dlk@entry > =3D3D3D0xfffff818=3D > > 543a9e70=3D3D > > > > , flags=3D3D3Dflags@entry=3D3D3D2121728, ilk=3D3D3Dilk@entry=3D3D3D= 0x0, > wmesg=3D > > =3D3D3Dwmesg@entry=3D3D > > > > =3D3D3D0xffffffff8222a054 "zfs", pri=3D3D3D, pri@ent= ry > =3D3D3D6=3D > > 4, timo=3D3D3D=3D3D > > > > timo@entry=3D3D3D6, queue=3D3D3D1) at /usr/src/sys/kern/kern_lock.c= :310 > > > > > #5 0xffffffff80b1a23f in lockmgr_slock_hard > (lk=3D3D3D0xfffff81854=3D > > 3a9e70=3D3D > > > > , flags=3D3D3D2121728, ilk=3D3D3D, > file=3D3D3D0xffffffff812544=3D > > fb "/usr/s=3D3D > > > > rc/sys/kern/vfs_subr.c", line=3D3D3D3057, lwa=3D3D3D0x0) at > /usr/src/sys/ke=3D > > rn/kern_=3D3D > > > > lock.c:705 > > > > > #6 0xffffffff80c59ec3 in VOP_LOCK1 > (vp=3D3D3D0xfffff818543a9e00, f=3D > > lags=3D3D > > > > =3D3D3D2105344, file=3D3D3D0xffffffff812544fb > "/usr/src/sys/kern/vfs_subr.c=3D > > ", line=3D3D > > > > =3D3D3D3057) at ./vnode_if.h:1120 > > > > > #7 _vn_lock (vp=3D3D3Dvp@entry=3D3D3D0xfffff818543a9e00, > flags=3D3D3D2=3D > > 105344, fi=3D3D > > > > le=3D3D3D, line=3D3D3D, line@entry=3D3D3D= 3057) > at /=3D > > usr/src/sy=3D3D > > > > s/kern/vfs_vnops.c:1815 > > > > > #8 0xffffffff80c4173d in vget_finish > (vp=3D3D3D0xfffff818543a9e00,=3D > > flags=3D3D > > > > =3D3D3D, vs=3D3D3Dvs@entry=3D3D3DVGET_USECOUNT) at > /usr/src/sys/=3D > > kern/vfs_s=3D3D > > > > ubr.c:3057 > > > > > #9 0xffffffff80c1c9b7 in cache_lookup (dvp=3D3D3Ddvp@entry > =3D3D3D0xf=3D > > ffff802c=3D3D > > > > d02ac40, vpp=3D3D3Dvpp@entry=3D3D3D0xfffffe046b20ac30, cnp=3D3D3Dcn= p@entry > =3D3D=3D > > 3D0xfffffe04=3D3D > > > > 6b20ac58, tsp=3D3D3Dtsp@entry=3D3D3D0x0, ticksp=3D3D3Dticksp@entry= =3D3D3D0x0) > a=3D > > t /usr/src/s=3D3D > > > > ys/kern/vfs_cache.c:2086 > > > > > #10 0xffffffff80c2150c in vfs_cache_lookup (ap=3D3D3D out=3D > > >) at =3D3D > > > > /usr/src/sys/kern/vfs_cache.c:3068 > > > > > #11 0xffffffff80c32c37 in VOP_LOOKUP > (dvp=3D3D3D0xfffff802cd02ac40,=3D > > vpp=3D3D > > > > =3D3D3D0xfffffe046b20ac30, cnp=3D3D3D0xfffffe046b20ac58) at > ./vnode_if.h:69 > > > > > #12 vfs_lookup (ndp=3D3D3Dndp@entry=3D3D3D0xfffffe046b20abd8)= at > /usr=3D > > /src/sys=3D3D > > > > /kern/vfs_lookup.c:1266 > > > > > #13 0xffffffff80c31ce1 in namei (ndp=3D3D3Dndp@entry > =3D3D3D0xfffffe04=3D > > 6b20abd8=3D3D > > > > ) at /usr/src/sys/kern/vfs_lookup.c:689 > > > > > #14 0xffffffff80c52090 in kern_statat > (td=3D3D3D0xfffffe02fb1d8000,=3D > > flag=3D3D > > > > =3D3D3D, fd=3D3D3D-100, path=3D3D3D0xa75b480e070 Canno=3D > > t access m=3D3D > > > > emory at address 0xa75b480e070>, pathseg=3D3D3Dpathseg@entry > =3D3D3DUIO_USER=3D > > SPACE, s=3D3D > > > > bp=3D3D3Dsbp@entry=3D3D3D0xfffffe046b20ad18) > > > > > at /usr/src/sys/kern/vfs_syscalls.c:2441 > > > > > #15 0xffffffff80c52797 in sys_fstatat (td=3D3D3D= , > uap=3D > > =3D3D3D0xff=3D3D > > > > fffe02fb1d8400) at /usr/src/sys/kern/vfs_syscalls.c:2419 > > > > > #16 0xffffffff81049398 in syscallenter (td=3D3D3D out>) a=3D > > t /usr=3D3D > > > > /src/sys/amd64/amd64/../../kern/subr_syscall.c:190 > > > > > #17 amd64_syscall (td=3D3D3D0xfffffe02fb1d8000, traced=3D3D3D= 0) at > /u=3D > > sr/src/s=3D3D > > > > ys/amd64/amd64/trap.c:1199 > > > > > #18 > > > > > > > > > > The lock it is trying to acquire in frame 5 belongs to another ba= sh > > > > > process which is in the process of creating a fifo: > > > > > > > > > > #0 sched_switch (td=3D3D3Dtd@entry=3D3D3D0xfffffe046acd8e40, > flags=3D > > =3D3D3Dflags@e=3D3D > > > > ntry=3D3D3D259) at /usr/src/sys/kern/sched_ule.c:2299 > > > > > #1 0xffffffff80b5a0a3 in mi_switch (flags=3D3D3Dflags@entry > =3D3D3D25=3D > > 9) at /u=3D3D > > > > sr/src/sys/kern/kern_synch.c:550 > > > > > #2 0xffffffff80babcb4 in sleepq_switch > (wchan=3D3D3D0xfffff8018acb=3D > > f154, =3D3D > > > > pri=3D3D3D87) at /usr/src/sys/kern/subr_sleepqueue.c:609 > > > > > #3 0xffffffff80babb8c in sleepq_wait > (wchan=3D3D3D, p=3D > > ri=3D3D3D<=3D3D > > > > unavailable>) at /usr/src/sys/kern/subr_sleepqueue.c:660 > > > > > #4 0xffffffff80b59606 in _sleep (ident=3D3D3Dident@entry > =3D3D3D0xfff=3D > > ff8018ac=3D3D > > > > bf154, lock=3D3D3Dlock@entry=3D3D3D0xfffff8018acbf120, > priority=3D3D3Dpriorit=3D > > y@entry=3D3D3D=3D3D > > > > 87, wmesg=3D3D3D0xffffffff8223af0e "zfs teardown inactive", > sbt=3D3D3Dsbt@e=3D > > ntry=3D3D3D0=3D3D > > > > , pr=3D3D3Dpr@entry=3D3D3D0, flags=3D3D3D256) > > > > > at /usr/src/sys/kern/kern_synch.c:225 > > > > > #5 0xffffffff80b45dc0 in rms_rlock_fallback > (rms=3D3D3D0xfffff8018=3D > > acbf12=3D3D > > > > 0) at /usr/src/sys/kern/kern_rmlock.c:1015 > > > > > #6 0xffffffff80b45c93 in rms_rlock (rms=3D3D3D, > rms@e=3D > > ntry=3D3D > > > > =3D3D3D0xfffff8018acbf120) at /usr/src/sys/kern/kern_rmlock.c:1036 > > > > > #7 0xffffffff81fb147b in zfs_freebsd_reclaim > (ap=3D3D3D > out>) =3D3D > > > > at > /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:51=3D > > 64 > > > > > #8 0xffffffff8111d245 in VOP_RECLAIM_APV > (vop=3D3D3D0xffffffff822e=3D > > 71a0 <=3D3D > > > > zfs_vnodeops>, a=3D3D3Da@entry=3D3D3D0xfffffe0410f1c9c8) at > vnode_if.c:2180 > > > > > #9 0xffffffff80c43569 in VOP_RECLAIM > (vp=3D3D3D0xfffff802cdbaca80)=3D > > at ./=3D3D > > > > vnode_if.h:1084 > > > > > #10 vgonel (vp=3D3D3Dvp@entry=3D3D3D0xfffff802cdbaca80) at > /usr/src/s=3D > > ys/kern/=3D3D > > > > vfs_subr.c:4143 > > > > > #11 0xffffffff80c3ef61 in vtryrecycle > (vp=3D3D3D0xfffff802cdbaca80)=3D > > at /u=3D3D > > > > sr/src/sys/kern/vfs_subr.c:1693 > > > > > #12 vnlru_free_impl (count=3D3D3Dcount@entry=3D3D3D1, > mnt_op=3D3D3Dmnt_=3D > > op@entry=3D3D > > > > =3D3D3D0x0, mvp=3D3D3D0xfffff8010864da00) at > /usr/src/sys/kern/vfs_subr.c:1=3D > > 344 > > > > > #13 0xfffff =C3=A3=C2=B5=C2=B6=E2=80=93, fff80c49553 in vn= lru_free_locked > (count=3D3D3D1) at /usr=3D > > /src/s=3D3D > > > > ys/kern/vfs_subr.c:1357 > > > > > #14 vn_alloc_hard (mp=3D3D3Dmp@entry=3D3D3D0x0) at > /usr/src/sys/kern/=3D > > vfs_subr=3D3D > > > > .c:1744 > > > > > #15 0xffffffff80c3f6f0 in vn_alloc (mp=3D3D3D0x0) at > /usr/src/sys/a=3D > > md64/i=3D3D > > > > nclude/atomic.h:375 > > > > > #16 getnewvnode_reserve () at /usr/src/sys/kern/vfs_subr.c:18= 88 > > > > > #17 0xffffffff81faa072 in zfs_create > (dzp=3D3D3D0xfffff812200261d0,=3D > > name=3D3D > > > > =3D3D3D0xfffff8011b8ac805 "sh-np.yPbxoo", vap=3D3D3D0xfffffe0410f1c= c20, > exc=3D > > l=3D3D3D > > > imized out>, mode=3D3D3D, zpp=3D3D3Dzpp@entry > =3D3D3D0xfffffe04=3D > > 10f1cbc8, =3D3D > > > > cr=3D3D3D0xfffff80140fb1100, flag=3D3D3D, vsecp=3D3D= 3D0x0, > mnt=3D > > _ns=3D3D3D0x0) > > > > > at > /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vno=3D > > ps_o=3D3D > > > > s.c:1146 > > > > > #18 0xffffffff81faea57 in zfs_freebsd_create > (ap=3D3D3D0xfffffe0410=3D > > f1cda0=3D3D > > > > ) at > /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:=3D > > 4618 > > > > > #19 0xffffffff8111aa9a in VOP_MKNOD_APV > (vop=3D3D3D0xffffffff822e71=3D > > a0 > > > s_vnodeops>, a=3D3D3Da@entry=3D3D3D0xfffffe0410f1cda0) at vnode_if.= c:372 > > > > > #20 0xffffffff80c50207 in VOP_MKNOD (dvp=3D3D3D, > cnp=3D > > =3D3D3D0xfff=3D3D > > > > ffe0410f1cd50, vap=3D3D3D0xfffffe0410f1cc20, vpp=3D3D3D) > at =3D > > ./vnode_=3D3D > > > > if.h:188 > > > > > #21 kern_mkfifoat (td=3D3D3D0xfffffe046acd8e40, fd=3D3D3D-100= , > path=3D > > =3D3D3D0x12772=3D3D > > > > f073500 , > pathse=3D > > g=3D3D3D=3D3D > > > > UIO_USERSPACE, mode=3D3D3D) at > /usr/src/sys/kern/vfs_sysca=3D > > lls.c:=3D3D > > > > 1492 > > > > > #22 0xffffffff81049398 in syscallenter (td=3D3D3D out>) a=3D > > t /usr=3D3D > > > > /src/sys/amd64/amd64/../../kern/subr_syscall.c:190 > > > > > #23 amd64_ =3DE6=3D90=3DAC=3DEE=3D8A=3D80 syscall > (td=3D3D3D0xfffffe046acd8e=3D > > 40, traced=3D3D3D0) at /usr/src/s=3D3D > > > > ys/amd64/amd64/trap.c:1199 > > > > > #24 > > > > > > > > > > Frame 7 is trying to acquire the ZFS teardown inactive lock, whic= h > is > > > > > held by a process which is performing a ZFS rollback and is > waiting f=3D > > or > > > > > the transaction to sync: > > > > > > > > > > #0 sched_switch (td=3D3D3Dtd@entry=3D3D3D0xfffffe0422ef8560, > flags=3D > > =3D3D3Dflags@e=3D3D > > > > ntry=3D3D3D259) at /usr/src/sys/kern/sched_ule.c:2299 > > > > > #1 0xffffffff80b5a0a3 in mi_switch (flags=3D3D3Dflags@entry > =3D3D3D25=3D > > 9) at /u=3D3D > > > > sr/src/sys/kern/kern_synch.c:550 > > > > > #2 0xffffffff80babcb4 in sleepq_switch > (wchan=3D3D3D0xfffff8011b83=3D > > d540, =3D3D > > > > pri=3D3D3D0) at /usr/src/sys/kern/subr_sleepqueue.c:609 > > > > > #3 0xffffffff80babb8c in sleepq_wait > (wchan=3D3D3D, w=3D > > chan@e=3D3D > > > > ntry=3D3D3D0xfffff8011b83d540, pri=3D3D3D, pri@entry= =3D3D3D0) > at=3D > > /usr/src/=3D3D > > > > sys/kern/subr_sleepqueue.c:660 > > > > > #4 0xffffffff80ad7f75 in _cv_wait (cvp=3D3D3Dcvp@entry > =3D3D3D0xfffff=3D > > 8011b83d=3D3D > > > > 540, lock=3D3D3Dlock@entry=3D3D3D0xfffff8011b83d4d0) at > /usr/src/sys/kern/k=3D > > ern_cond=3D3D > > > > var.c:146 > > > > > #5 0xffffffff820b42fb in txg_wait_synced_impl (dp=3D3D3Ddp@e= ntry > =3D > > =3D3D3D0xfff=3D3D > > > > ff8011b83d000, txg=3D3D3D8585097, wait_sig=3D3D3Dwait_sig@entry=3D3= D3D0) > at /=3D > > usr/src/sy=3D3D > > > > s/contrib/openzfs/module/zfs/txg.c:726 > > > > > #6 0xffffffff820b3cab in txg_wait_synced > (dp=3D3D3D, =3D > > dp@ent=3D3D > > > > ry=3D3D3D0xfffff8011b83d000, txg=3D3D3D) at > /usr/src/sys/contr=3D > > ib/openz=3D3D > > > > fs/module/zfs/txg.c:736 > > > > > #7 0xffffffff8206d5b5 in dsl_sync_task_common > (pool=3D3D3Dpool@ent=3D > > ry=3D3D3D0=3D3D > > > > xfffffe0401d15000 "zroot/poudriere/jails/13amd64-default-ref/15", > check=3D > > func=3D3D > > > > =3D3D3D, syncfunc=3D3D3D0xffffffff8203fbc0 > > back_syn=3D3D > > > > c>, sigfunc=3D3D3Dsigfunc@entry=3D3D3D0x0, arg=3D3D3Darg@entry > =3D3D3D0xfffffe02=3D > > fb827a90, > > > > > blocks_modified=3D3D3Dblocks_modified@entry=3D3D3D1, > space_check=3D > > =3D3D3DZFS_SP=3D3D > > > > ACE_CHECK_RESERVED, early=3D3D3D0) at > /usr/src/sys/contrib/openzfs/module=3D > > /zfs/d=3D3D > > > > sl_synctask.c:93 > > > > > #8 0xffffffff8206d3c7 in dsl_sync_task > (pool=3D3D3D, =3D > > pool@e=3D3D > > > > ntry=3D3D3D0xfffffe0401d15000 > "zroot/poudriere/jails/13amd64-default-ref/=3D > > 15", c=3D3D > > > > heckfunc=3D3D3D, syncfunc=3D3D3D, > arg=3D3D3D > lable>, ar=3D3D > > > > g@entry=3D3D3D0xfffffe02fb827a90, blocks_modified=3D3D3D, > > > > > blocks_modified@entry=3D3D3D1, > space_check=3D3D3D, s=3D > > pace_che=3D3D > > > > ck@entry=3D3D3DZFS_SPACE_CHECK_RESERVED) at > /usr/src/sys/contrib/openzfs/=3D > > module=3D3D > > > > /zfs/dsl_synctask.c:132 > > > > > #9 0xffffffff8204075b in dsl_dataset_rollback > (fsname=3D3D3D > ilable=3D3D > > > > >, fsname@entry=3D3D3D0xfffffe0401d15000 > "zroot/poudriere/jails/13amd64-d=3D > > efault=3D3D > > > > -ref/15", tosnap=3D3D3D, owner=3D3D3D= , > result=3D > > =3D3D3Dresul=3D3D > > > > t@entry=3D3D3D0xfffff81c826a9ea0) > > > > > at > /usr/src/sys/contrib/openzfs/module/zfs/dsl_dataset.c:3261 > > > > > #10 0xffffffff82168dd9 in zfs_ioc_rollback > (fsname=3D3D3D0xfffffe04=3D > > 01d150=3D3D > > > > 00 "zroot/poudriere/jails/13amd64-default-ref/15", fsname@entry > =3D3D3D > ror re=3D3D > > > > ading variable: value is not available>, innvl=3D3D3D, > innvl=3D > > @entry=3D3D > > > > =3D3D3D, > > > > > outnvl=3D3D3D0xfffff81c826a9ea0, outnvl@entry=3D3D3D readin=3D > > g variab=3D3D > > > > le: value is not available>) at > /usr/src/sys/contrib/openzfs/module/zfs=3D > > /zfs=3D3D > > > > _ioctl.c:4405 > > > > > #11 0xffffffff82164522 in zfsdev_ioctl_common > (vecnum=3D3D3Dvecnum@=3D > > entry=3D3D > > > > =3D3D3D25, zc=3D3D3Dzc@entry=3D3D3D0xfffffe0401d15000, flag=3D3D3Df= lag@entry > =3D3D=3D > > 3D0) at /usr/s=3D3D > > > > rc/sys/contrib/openzfs/module/zfs/zfs_ioctl.c:7798 > > > > > #12 0xffffffff81f97fca in zfsdev_ioctl (dev=3D3D3D out>, =3D > > zcmd=3D3D > > > > =3D3D3D, zcmd@entry=3D3D3D is no=3D > > t availa=3D3D > > > > ble>, arg=3D3D3D0xfffffe02fb827d50 "\017", arg@entry=3D3D3D reading v=3D > > ariable:=3D3D > > > > value is not available>, flag=3D3D3D, > td=3D3D3D > ut>) > > > > > at > /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/kmod_co=3D > > re.c=3D3D > > > > :168 > > > > > #13 0xffffffff809d6212 in devfs_ioctl > (ap=3D3D3D0xfffffe02fb827c50)=3D > > at /u=3D3D > > > > sr/src/sys/fs/devfs/devfs_vnops.c:935 > > > > > #14 0xffffffff80c585f2 in vn_ioctl (fp=3D3D3D0xfffff8052cdd80= f0, > co=3D > > m=3D3D3D > > > ptimized out>, data=3D3D3D0xfffffe02fb827d50, > active_cred=3D3D3D0xfffff8012=3D > > 2ab1e00,=3D3D > > > > td=3D3D3D) at /usr/src/sys/kern/vfs_vnops.c:1704 > > > > > #15 0xffffffff809d68ee in devfs_ioctl_f (fp=3D3D3D, > fp=3D > > @entry=3D3D > > > > =3D3D3D, > com=3D3D3D > able>, c=3D3D > > > > om@entry=3D3D3D, > data=3D3D3=3D > > D > > > lable>, data@entry=3D3D3D available>=3D > > , > > > > > cred=3D3D3D, cred@entry=3D3D3D variabl=3D > > e: value=3D3D > > > > is not available>, td=3D3D3D, td@entry=3D3D3D reading v=3D > > ariable:=3D3D > > > > value is not available>) at /usr/src/sys/fs/devfs/devfs_vnops.c:86= 6 > > > > > #16 0xffffffff80bc57e6 in fo_ioctl (fp=3D3D3D0xfffff8052cdd80= f0, > co=3D > > m=3D3D3D32=3D3D > > > > 22821401, data=3D3D3D, active_cred=3D3D3D= , > td=3D3D3=3D > > D0xfffffe0=3D3D > > > > 422ef8560) at /usr/src/sys/sys/file.h:367 > > > > > #17 kern_ioctl (td=3D3D3Dtd@entry=3D3D3D0xfffffe0422ef8560, > fd=3D3D3D4,=3D > > com=3D3D3Dcom=3D3D > > > > @entry=3D3D3D3222821401, data=3D3D3D, data@entry > =3D3D3D0xfffffe0=3D > > 2fb827d50 =3D3D > > > > "\017") at /usr/src/sys/kern/sys_generic.c:807 > > > > > #18 0xffffffff80bc54f2 in sys_ioctl > (td=3D3D3D0xfffffe0422ef8560, u=3D > > ap=3D3D3D0=3D3D > > > > xfffffe0422ef8960) at /usr/src/sys/kern/sys_generic.c:715 > > > > > #19 0xffffffff81049398 in syscallenter (td=3D3D3D out>) a=3D > > t /usr=3D3D > > > > /src/sys/amd64/amd64/../../kern/subr_syscall.c:190 > > > > > #20 amd64_syscall (td=3D3D3D0xfffffe0422ef8560, traced=3D3D3D= 0) at > /u=3D > > sr/src/s=3D3D > > > > ys/amd64/amd64/trap.c:1199 > > > [...] > > > > > > The backtrace looks different though it certainly smells like > PR/271945. > > > > > > I've had similar to PR/271945 panics on an amd64 with a mirrored zpoo= l > wi=3D > > th > > > four vdevs running poudriere with AMD64 jails. My other amd64 with a > > > mirrored zpool with two vdevs using i386 jails has no such issue. All > oth=3D > > er > > > workloads are unaffected. > > > > > > On the affected machine running poudriere bulk with -J N:1 circumvent= s > th=3D > > e > > > issue. So far. There were two openzfs cherry-picks this morning. I > intend > > > to try them against a full bulk build later today. > > > > > > > > > -- > > > Cheers, > > > Cy Schubert > > > FreeBSD UNIX: Web: https://FreeBSD.org > > > NTP: Web: https://nwtime.org > > > > > > e^(i*pi)+1=3D3D0 > > > > > > > > > --000000000000b7db4b06029d5ea5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Spoke too soon still seeing zfs lockups under heavy poudr= iere workload after the MFVs.=C2=A0 Regression time matches what has been r= eported here.

On Thu, Aug 10, 2023 at 4:33 PM Cy Schubert <Cy.Schubert@cschubert.com> wro= te:
I haven't experienced any problems (yet) = either.


--
Cheers,
Cy Schubert <Cy.Schubert@cschubert.com>
FreeBSD UNIX:=C2=A0 <cy@FreeBSD.org>=C2=A0 =C2=A0Web:=C2=A0 https://FreeB= SD.org
NTP:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<cy@nwtime.org>=C2=A0 =C2=A0 Web:=C2=A0 https://nwt= ime.org

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 e^(i*pi)+1=3D0


In message <CAK7dMtDJQtaai3_6VjEkwVwW5JN6e8v=3DkKTOPffp371xb=3DORUg@mail= .gmail.c
om>
, Kevin Bowling writes:
> The two MFVs on head have improved/fixed stability with poudriere for<= br> > me 48 core bare metal.
>
> On Thu, Aug 10, 2023 at 6:37=3DE2=3D80=3DAFAM Cy Schubert <Cy.Schub= ert@cschubert.=3D
> com> wrote:
> >
> > In message <CAK7dMtDJeuf8rjWbsNEZABUfeqpjUyCHzuOL9AAhKk93sy+PK= g@mail.gmai=3D
> l.c
> > om>
> > , Kevin Bowling writes:
> > > Possibly https://git= hub.com/openzfs/zfs/commit/2cb992a99ccadb78d97049b4=3D
> 0bd4=3D3D
> > > 42eb4fdc549d
> > >
> > > On Tue, Aug 8, 2023 at 10:08=3D3DE2=3D3D80=3D3DAFAM Dag-Erli= ng Sm=3D3DC3=3D3DB8rg=3D
> rav <des@freeb=3D3D
> > > sd.org> wrote:
> > > >
> > > > At some point between 42d088299c (4 May) and f0c9703301= (26 June), a
> > > > deadlock was introduced in ZFS.=C2=A0 It is still prese= nt as of 9c2823bae9=3D
>=C2=A0 (4
> > > > August) and is 100% reproducable just by starting poudr= iere bulk in a
> > > > 16-core VM and waiting a few hours until deadlkres kick= s in.=C2=A0 In the
> > > > latest instance, deadlkres complained about a bash proc= ess:
> > > >
> > > >=C2=A0 =C2=A0 =C2=A0#0=C2=A0 sched_switch (td=3D3D3Dtd@e= ntry=3D3D3D0xfffffe02fb1d8000, flags=3D
> =3D3D3Dflags@e=3D3D
> > > ntry=3D3D3D259) at /usr/src/sys/kern/sched_ule.c:2299
> > > >=C2=A0 =C2=A0 =C2=A0#1=C2=A0 0xffffffff80b5a0a3 in mi_sw= itch (flags=3D3D3Dflags@entry=3D3D3D25=3D
> 9) at /u=3D3D
> > > sr/src/sys/kern/kern_synch.c:550
> > > >=C2=A0 =C2=A0 =C2=A0#2=C2=A0 0xffffffff80babcb4 in sleep= q_switch (wchan=3D3D3D0xfffff818543a=3D
> 9e70, =3D3D
> > > pri=3D3D3D64) at /usr/src/sys/kern/subr_sleepqueue.c:609
> > > >=C2=A0 =C2=A0 =C2=A0#3=C2=A0 0xffffffff80babb8c in sleep= q_wait (wchan=3D3D3D<unavailable>, p=3D
> ri=3D3D3D<=3D3D
> > > unavailable>) at /usr/src/sys/kern/subr_sleepqueue.c:660<= br> > > > >=C2=A0 =C2=A0 =C2=A0#4=C2=A0 0xffffffff80b1c1b0 in sleep= lk (lk=3D3D3Dlk@entry=3D3D3D0xfffff818=3D
> 543a9e70=3D3D
> > > , flags=3D3D3Dflags@entry=3D3D3D2121728, ilk=3D3D3Dilk@entry= =3D3D3D0x0, wmesg=3D
> =3D3D3Dwmesg@entry=3D3D
> > > =3D3D3D0xffffffff8222a054 "zfs", pri=3D3D3D<opt= imized out>, pri@entry=3D3D3D6=3D
> 4, timo=3D3D3D=3D3D
> > > timo@entry=3D3D3D6, queue=3D3D3D1) at /usr/src/sys/kern/kern= _lock.c:310
> > > >=C2=A0 =C2=A0 =C2=A0#5=C2=A0 0xffffffff80b1a23f in lockm= gr_slock_hard (lk=3D3D3D0xfffff81854=3D
> 3a9e70=3D3D
> > > , flags=3D3D3D2121728, ilk=3D3D3D<optimized out>, file= =3D3D3D0xffffffff812544=3D
> fb "/usr/s=3D3D
> > > rc/sys/kern/vfs_subr.c", line=3D3D3D3057, lwa=3D3D3D0x0= ) at /usr/src/sys/ke=3D
> rn/kern_=3D3D
> > > lock.c:705
> > > >=C2=A0 =C2=A0 =C2=A0#6=C2=A0 0xffffffff80c59ec3 in VOP_L= OCK1 (vp=3D3D3D0xfffff818543a9e00, f=3D
> lags=3D3D
> > > =3D3D3D2105344, file=3D3D3D0xffffffff812544fb "/usr/src= /sys/kern/vfs_subr.c=3D
> ", line=3D3D
> > > =3D3D3D3057) at ./vnode_if.h:1120
> > > >=C2=A0 =C2=A0 =C2=A0#7=C2=A0 _vn_lock (vp=3D3D3Dvp@entry= =3D3D3D0xfffff818543a9e00, flags=3D3D3D2=3D
> 105344, fi=3D3D
> > > le=3D3D3D<unavailable>, line=3D3D3D<unavailable>= , line@entry=3D3D3D3057) at /=3D
> usr/src/sy=3D3D
> > > s/kern/vfs_vnops.c:1815
> > > >=C2=A0 =C2=A0 =C2=A0#8=C2=A0 0xffffffff80c4173d in vget_= finish (vp=3D3D3D0xfffff818543a9e00,=3D
>=C2=A0 flags=3D3D
> > > =3D3D3D<unavailable>, vs=3D3D3Dvs@entry=3D3D3DVGET_USE= COUNT) at /usr/src/sys/=3D
> kern/vfs_s=3D3D
> > > ubr.c:3057
> > > >=C2=A0 =C2=A0 =C2=A0#9=C2=A0 0xffffffff80c1c9b7 in cache= _lookup (dvp=3D3D3Ddvp@entry=3D3D3D0xf=3D
> ffff802c=3D3D
> > > d02ac40, vpp=3D3D3Dvpp@entry=3D3D3D0xfffffe046b20ac30, cnp= =3D3D3Dcnp@entry=3D3D=3D
> 3D0xfffffe04=3D3D
> > > 6b20ac58, tsp=3D3D3Dtsp@entry=3D3D3D0x0, ticksp=3D3D3Dticksp= @entry=3D3D3D0x0) a=3D
> t /usr/src/s=3D3D
> > > ys/kern/vfs_cache.c:2086
> > > >=C2=A0 =C2=A0 =C2=A0#10 0xffffffff80c2150c in vfs_cache_= lookup (ap=3D3D3D<optimized out=3D
> >) at =3D3D
> > > /usr/src/sys/kern/vfs_cache.c:3068
> > > >=C2=A0 =C2=A0 =C2=A0#11 0xffffffff80c32c37 in VOP_LOOKUP= (dvp=3D3D3D0xfffff802cd02ac40,=3D
>=C2=A0 vpp=3D3D
> > > =3D3D3D0xfffffe046b20ac30, cnp=3D3D3D0xfffffe046b20ac58) at = ./vnode_if.h:69
> > > >=C2=A0 =C2=A0 =C2=A0#12 vfs_lookup (ndp=3D3D3Dndp@entry= =3D3D3D0xfffffe046b20abd8) at /usr=3D
> /src/sys=3D3D
> > > /kern/vfs_lookup.c:1266
> > > >=C2=A0 =C2=A0 =C2=A0#13 0xffffffff80c31ce1 in namei (ndp= =3D3D3Dndp@entry=3D3D3D0xfffffe04=3D
> 6b20abd8=3D3D
> > > ) at /usr/src/sys/kern/vfs_lookup.c:689
> > > >=C2=A0 =C2=A0 =C2=A0#14 0xffffffff80c52090 in kern_stata= t (td=3D3D3D0xfffffe02fb1d8000,=3D
>=C2=A0 flag=3D3D
> > > =3D3D3D<optimized out>, fd=3D3D3D-100, path=3D3D3D0xa7= 5b480e070 <error: Canno=3D
> t access m=3D3D
> > > emory at address 0xa75b480e070>, pathseg=3D3D3Dpathseg@en= try=3D3D3DUIO_USER=3D
> SPACE, s=3D3D
> > > bp=3D3D3Dsbp@entry=3D3D3D0xfffffe046b20ad18)
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at /usr/src/sys/kern/v= fs_syscalls.c:2441
> > > >=C2=A0 =C2=A0 =C2=A0#15 0xffffffff80c52797 in sys_fstata= t (td=3D3D3D<unavailable>, uap=3D
> =3D3D3D0xff=3D3D
> > > fffe02fb1d8400) at /usr/src/sys/kern/vfs_syscalls.c:2419
> > > >=C2=A0 =C2=A0 =C2=A0#16 0xffffffff81049398 in syscallent= er (td=3D3D3D<optimized out>) a=3D
> t /usr=3D3D
> > > /src/sys/amd64/amd64/../../kern/subr_syscall.c:190
> > > >=C2=A0 =C2=A0 =C2=A0#17 amd64_syscall (td=3D3D3D0xfffffe= 02fb1d8000, traced=3D3D3D0) at /u=3D
> sr/src/s=3D3D
> > > ys/amd64/amd64/trap.c:1199
> > > >=C2=A0 =C2=A0 =C2=A0#18 <signal handler called> > > > >
> > > > The lock it is trying to acquire in frame 5 belongs to = another bash
> > > > process which is in the process of creating a fifo:
> > > >
> > > >=C2=A0 =C2=A0 =C2=A0#0=C2=A0 sched_switch (td=3D3D3Dtd@e= ntry=3D3D3D0xfffffe046acd8e40, flags=3D
> =3D3D3Dflags@e=3D3D
> > > ntry=3D3D3D259) at /usr/src/sys/kern/sched_ule.c:2299
> > > >=C2=A0 =C2=A0 =C2=A0#1=C2=A0 0xffffffff80b5a0a3 in mi_sw= itch (flags=3D3D3Dflags@entry=3D3D3D25=3D
> 9) at /u=3D3D
> > > sr/src/sys/kern/kern_synch.c:550
> > > >=C2=A0 =C2=A0 =C2=A0#2=C2=A0 0xffffffff80babcb4 in sleep= q_switch (wchan=3D3D3D0xfffff8018acb=3D
> f154, =3D3D
> > > pri=3D3D3D87) at /usr/src/sys/kern/subr_sleepqueue.c:609
> > > >=C2=A0 =C2=A0 =C2=A0#3=C2=A0 0xffffffff80babb8c in sleep= q_wait (wchan=3D3D3D<unavailable>, p=3D
> ri=3D3D3D<=3D3D
> > > unavailable>) at /usr/src/sys/kern/subr_sleepqueue.c:660<= br> > > > >=C2=A0 =C2=A0 =C2=A0#4=C2=A0 0xffffffff80b59606 in _slee= p (ident=3D3D3Dident@entry=3D3D3D0xfff=3D
> ff8018ac=3D3D
> > > bf154, lock=3D3D3Dlock@entry=3D3D3D0xfffff8018acbf120, prior= ity=3D3D3Dpriorit=3D
> y@entry=3D3D3D=3D3D
> > > 87, wmesg=3D3D3D0xffffffff8223af0e "zfs teardown inacti= ve", sbt=3D3D3Dsbt@e=3D
> ntry=3D3D3D0=3D3D
> > > , pr=3D3D3Dpr@entry=3D3D3D0, flags=3D3D3D256)
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at /usr/src/sys/kern/k= ern_synch.c:225
> > > >=C2=A0 =C2=A0 =C2=A0#5=C2=A0 0xffffffff80b45dc0 in rms_r= lock_fallback (rms=3D3D3D0xfffff8018=3D
> acbf12=3D3D
> > > 0) at /usr/src/sys/kern/kern_rmlock.c:1015
> > > >=C2=A0 =C2=A0 =C2=A0#6=C2=A0 0xffffffff80b45c93 in rms_r= lock (rms=3D3D3D<unavailable>, rms@e=3D
> ntry=3D3D
> > > =3D3D3D0xfffff8018acbf120) at /usr/src/sys/kern/kern_rmlock.= c:1036
> > > >=C2=A0 =C2=A0 =C2=A0#7=C2=A0 0xffffffff81fb147b in zfs_f= reebsd_reclaim (ap=3D3D3D<optimized =3D
> out>) =3D3D
> > > at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vn= ops_os.c:51=3D
> 64
> > > >=C2=A0 =C2=A0 =C2=A0#8=C2=A0 0xffffffff8111d245 in VOP_R= ECLAIM_APV (vop=3D3D3D0xffffffff822e=3D
> 71a0 <=3D3D
> > > zfs_vnodeops>, a=3D3D3Da@entry=3D3D3D0xfffffe0410f1c9c8) = at vnode_if.c:2180
> > > >=C2=A0 =C2=A0 =C2=A0#9=C2=A0 0xffffffff80c43569 in VOP_R= ECLAIM (vp=3D3D3D0xfffff802cdbaca80)=3D
>=C2=A0 at ./=3D3D
> > > vnode_if.h:1084
> > > >=C2=A0 =C2=A0 =C2=A0#10 vgonel (vp=3D3D3Dvp@entry=3D3D3D= 0xfffff802cdbaca80) at /usr/src/s=3D
> ys/kern/=3D3D
> > > vfs_subr.c:4143
> > > >=C2=A0 =C2=A0 =C2=A0#11 0xffffffff80c3ef61 in vtryrecycl= e (vp=3D3D3D0xfffff802cdbaca80)=3D
>=C2=A0 at /u=3D3D
> > > sr/src/sys/kern/vfs_subr.c:1693
> > > >=C2=A0 =C2=A0 =C2=A0#12 vnlru_free_impl (count=3D3D3Dcou= nt@entry=3D3D3D1, mnt_op=3D3D3Dmnt_=3D
> op@entry=3D3D
> > > =3D3D3D0x0, mvp=3D3D3D0xfffff8010864da00) at /usr/src/sys/ke= rn/vfs_subr.c:1=3D
> 344
> > > >=C2=A0 =C2=A0 =C2=A0#13 0xfffff =C3=A3=C2=B5=C2=B6=E2=80= =93,=C2=A0 =C2=A0 fff80c49553 in vnlru_free_locked (count=3D3D3D1) at /usr= =3D
> /src/s=3D3D
> > > ys/kern/vfs_subr.c:1357
> > > >=C2=A0 =C2=A0 =C2=A0#14 vn_alloc_hard (mp=3D3D3Dmp@entry= =3D3D3D0x0) at /usr/src/sys/kern/=3D
> vfs_subr=3D3D
> > > .c:1744
> > > >=C2=A0 =C2=A0 =C2=A0#15 0xffffffff80c3f6f0 in vn_alloc (= mp=3D3D3D0x0) at /usr/src/sys/a=3D
> md64/i=3D3D
> > > nclude/atomic.h:375
> > > >=C2=A0 =C2=A0 =C2=A0#16 getnewvnode_reserve () at /usr/s= rc/sys/kern/vfs_subr.c:1888
> > > >=C2=A0 =C2=A0 =C2=A0#17 0xffffffff81faa072 in zfs_create= (dzp=3D3D3D0xfffff812200261d0,=3D
>=C2=A0 name=3D3D
> > > =3D3D3D0xfffff8011b8ac805 "sh-np.yPbxoo", vap=3D3D= 3D0xfffffe0410f1cc20, exc=3D
> l=3D3D3D<opt=3D3D
> > > imized out>, mode=3D3D3D<optimized out>, zpp=3D3D3D= zpp@entry=3D3D3D0xfffffe04=3D
> 10f1cbc8, =3D3D
> > > cr=3D3D3D0xfffff80140fb1100, flag=3D3D3D<optimized out>= ;, vsecp=3D3D3D0x0, mnt=3D
> _ns=3D3D3D0x0)
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at /usr/src/sys/contri= b/openzfs/module/os/freebsd/zfs/zfs_vno=3D
> ps_o=3D3D
> > > s.c:1146
> > > >=C2=A0 =C2=A0 =C2=A0#18 0xffffffff81faea57 in zfs_freebs= d_create (ap=3D3D3D0xfffffe0410=3D
> f1cda0=3D3D
> > > ) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_= vnops_os.c:=3D
> 4618
> > > >=C2=A0 =C2=A0 =C2=A0#19 0xffffffff8111aa9a in VOP_MKNOD_= APV (vop=3D3D3D0xffffffff822e71=3D
> a0 <zf=3D3D
> > > s_vnodeops>, a=3D3D3Da@entry=3D3D3D0xfffffe0410f1cda0) at= vnode_if.c:372
> > > >=C2=A0 =C2=A0 =C2=A0#20 0xffffffff80c50207 in VOP_MKNOD = (dvp=3D3D3D<unavailable>, cnp=3D
> =3D3D3D0xfff=3D3D
> > > ffe0410f1cd50, vap=3D3D3D0xfffffe0410f1cc20, vpp=3D3D3D<o= ptimized out>) at =3D
> ./vnode_=3D3D
> > > if.h:188
> > > >=C2=A0 =C2=A0 =C2=A0#21 kern_mkfifoat (td=3D3D3D0xfffffe= 046acd8e40, fd=3D3D3D-100, path=3D
> =3D3D3D0x12772=3D3D
> > > f073500 <error: Cannot access memory at address 0x12772f0= 73500>, pathse=3D
> g=3D3D3D=3D3D
> > > UIO_USERSPACE, mode=3D3D3D<optimized out>) at /usr/src= /sys/kern/vfs_sysca=3D
> lls.c:=3D3D
> > > 1492
> > > >=C2=A0 =C2=A0 =C2=A0#22 0xffffffff81049398 in syscallent= er (td=3D3D3D<optimized out>) a=3D
> t /usr=3D3D
> > > /src/sys/amd64/amd64/../../kern/subr_syscall.c:190
> > > >=C2=A0 =C2=A0 =C2=A0#23 amd64_=C2=A0 =C2=A0 =3DE6=3D90= =3DAC=3DEE=3D8A=3D80 syscall (td=3D3D3D0xfffffe046acd8e=3D
> 40, traced=3D3D3D0) at /usr/src/s=3D3D
> > > ys/amd64/amd64/trap.c:1199
> > > >=C2=A0 =C2=A0 =C2=A0#24 <signal handler called> > > > >
> > > > Frame 7 is trying to acquire the ZFS teardown inactive = lock, which is
> > > > held by a process which is performing a ZFS rollback an= d is waiting f=3D
> or
> > > > the transaction to sync:
> > > >
> > > >=C2=A0 =C2=A0 =C2=A0#0=C2=A0 sched_switch (td=3D3D3Dtd@e= ntry=3D3D3D0xfffffe0422ef8560, flags=3D
> =3D3D3Dflags@e=3D3D
> > > ntry=3D3D3D259) at /usr/src/sys/kern/sched_ule.c:2299
> > > >=C2=A0 =C2=A0 =C2=A0#1=C2=A0 0xffffffff80b5a0a3 in mi_sw= itch (flags=3D3D3Dflags@entry=3D3D3D25=3D
> 9) at /u=3D3D
> > > sr/src/sys/kern/kern_synch.c:550
> > > >=C2=A0 =C2=A0 =C2=A0#2=C2=A0 0xffffffff80babcb4 in sleep= q_switch (wchan=3D3D3D0xfffff8011b83=3D
> d540, =3D3D
> > > pri=3D3D3D0) at /usr/src/sys/kern/subr_sleepqueue.c:609
> > > >=C2=A0 =C2=A0 =C2=A0#3=C2=A0 0xffffffff80babb8c in sleep= q_wait (wchan=3D3D3D<unavailable>, w=3D
> chan@e=3D3D
> > > ntry=3D3D3D0xfffff8011b83d540, pri=3D3D3D<unavailable>= , pri@entry=3D3D3D0) at=3D
>=C2=A0 /usr/src/=3D3D
> > > sys/kern/subr_sleepqueue.c:660
> > > >=C2=A0 =C2=A0 =C2=A0#4=C2=A0 0xffffffff80ad7f75 in _cv_w= ait (cvp=3D3D3Dcvp@entry=3D3D3D0xfffff=3D
> 8011b83d=3D3D
> > > 540, lock=3D3D3Dlock@entry=3D3D3D0xfffff8011b83d4d0) at /usr= /src/sys/kern/k=3D
> ern_cond=3D3D
> > > var.c:146
> > > >=C2=A0 =C2=A0 =C2=A0#5=C2=A0 0xffffffff820b42fb in txg_w= ait_synced_impl (dp=3D3D3Ddp@entry=3D
> =3D3D3D0xfff=3D3D
> > > ff8011b83d000, txg=3D3D3D8585097, wait_sig=3D3D3Dwait_sig@en= try=3D3D3D0) at /=3D
> usr/src/sy=3D3D
> > > s/contrib/openzfs/module/zfs/txg.c:726
> > > >=C2=A0 =C2=A0 =C2=A0#6=C2=A0 0xffffffff820b3cab in txg_w= ait_synced (dp=3D3D3D<unavailable>, =3D
> dp@ent=3D3D
> > > ry=3D3D3D0xfffff8011b83d000, txg=3D3D3D<unavailable>) = at /usr/src/sys/contr=3D
> ib/openz=3D3D
> > > fs/module/zfs/txg.c:736
> > > >=C2=A0 =C2=A0 =C2=A0#7=C2=A0 0xffffffff8206d5b5 in dsl_s= ync_task_common (pool=3D3D3Dpool@ent=3D
> ry=3D3D3D0=3D3D
> > > xfffffe0401d15000 "zroot/poudriere/jails/13amd64-defaul= t-ref/15", check=3D
> func=3D3D
> > > =3D3D3D<optimized out>, syncfunc=3D3D3D0xffffffff8203f= bc0 <dsl_dataset_roll=3D
> back_syn=3D3D
> > > c>, sigfunc=3D3D3Dsigfunc@entry=3D3D3D0x0, arg=3D3D3Darg@= entry=3D3D3D0xfffffe02=3D
> fb827a90,
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0blocks_modified=3D3D3D= blocks_modified@entry=3D3D3D1, space_check=3D
> =3D3D3DZFS_SP=3D3D
> > > ACE_CHECK_RESERVED, early=3D3D3D0) at /usr/src/sys/contrib/o= penzfs/module=3D
> /zfs/d=3D3D
> > > sl_synctask.c:93
> > > >=C2=A0 =C2=A0 =C2=A0#8=C2=A0 0xffffffff8206d3c7 in dsl_s= ync_task (pool=3D3D3D<unavailable>, =3D
> pool@e=3D3D
> > > ntry=3D3D3D0xfffffe0401d15000 "zroot/poudriere/jails/13= amd64-default-ref/=3D
> 15", c=3D3D
> > > heckfunc=3D3D3D<unavailable>, syncfunc=3D3D3D<unava= ilable>, arg=3D3D3D<unavai=3D
> lable>, ar=3D3D
> > > g@entry=3D3D3D0xfffffe02fb827a90, blocks_modified=3D3D3D<= unavailable>,
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0blocks_modified@entry= =3D3D3D1, space_check=3D3D3D<unavailable>, s=3D
> pace_che=3D3D
> > > ck@entry=3D3D3DZFS_SPACE_CHECK_RESERVED) at /usr/src/sys/con= trib/openzfs/=3D
> module=3D3D
> > > /zfs/dsl_synctask.c:132
> > > >=C2=A0 =C2=A0 =C2=A0#9=C2=A0 0xffffffff8204075b in dsl_d= ataset_rollback (fsname=3D3D3D<unava=3D
> ilable=3D3D
> > > >, fsname@entry=3D3D3D0xfffffe0401d15000 "zroot/poud= riere/jails/13amd64-d=3D
> efault=3D3D
> > > -ref/15", tosnap=3D3D3D<optimized out>, owner=3D3= D3D<optimized out>, result=3D
> =3D3D3Dresul=3D3D
> > > t@entry=3D3D3D0xfffff81c826a9ea0)
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at /usr/src/sys/contri= b/openzfs/module/zfs/dsl_dataset.c:3261
> > > >=C2=A0 =C2=A0 =C2=A0#10 0xffffffff82168dd9 in zfs_ioc_ro= llback (fsname=3D3D3D0xfffffe04=3D
> 01d150=3D3D
> > > 00 "zroot/poudriere/jails/13amd64-default-ref/15",= fsname@entry=3D3D3D<er=3D
> ror re=3D3D
> > > ading variable: value is not available>, innvl=3D3D3D<= unavailable>, innvl=3D
> @entry=3D3D
> > > =3D3D3D<error reading variable: value is not available>= ;,
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0outnvl=3D3D3D0xfffff81= c826a9ea0, outnvl@entry=3D3D3D<error readin=3D
> g variab=3D3D
> > > le: value is not available>) at /usr/src/sys/contrib/open= zfs/module/zfs=3D
> /zfs=3D3D
> > > _ioctl.c:4405
> > > >=C2=A0 =C2=A0 =C2=A0#11 0xffffffff82164522 in zfsdev_ioc= tl_common (vecnum=3D3D3Dvecnum@=3D
> entry=3D3D
> > > =3D3D3D25, zc=3D3D3Dzc@entry=3D3D3D0xfffffe0401d15000, flag= =3D3D3Dflag@entry=3D3D=3D
> 3D0) at /usr/s=3D3D
> > > rc/sys/contrib/openzfs/module/zfs/zfs_ioctl.c:7798
> > > >=C2=A0 =C2=A0 =C2=A0#12 0xffffffff81f97fca in zfsdev_ioc= tl (dev=3D3D3D<optimized out>, =3D
> zcmd=3D3D
> > > =3D3D3D<unavailable>, zcmd@entry=3D3D3D<error readi= ng variable: value is no=3D
> t availa=3D3D
> > > ble>, arg=3D3D3D0xfffffe02fb827d50 "\017", arg@= entry=3D3D3D<error reading v=3D
> ariable:=3D3D
> > >=C2=A0 value is not available>, flag=3D3D3D<optimized o= ut>, td=3D3D3D<optimized o=3D
> ut>)
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at /usr/src/sys/contri= b/openzfs/module/os/freebsd/zfs/kmod_co=3D
> re.c=3D3D
> > > :168
> > > >=C2=A0 =C2=A0 =C2=A0#13 0xffffffff809d6212 in devfs_ioct= l (ap=3D3D3D0xfffffe02fb827c50)=3D
>=C2=A0 at /u=3D3D
> > > sr/src/sys/fs/devfs/devfs_vnops.c:935
> > > >=C2=A0 =C2=A0 =C2=A0#14 0xffffffff80c585f2 in vn_ioctl (= fp=3D3D3D0xfffff8052cdd80f0, co=3D
> m=3D3D3D<o=3D3D
> > > ptimized out>, data=3D3D3D0xfffffe02fb827d50, active_cred= =3D3D3D0xfffff8012=3D
> 2ab1e00,=3D3D
> > >=C2=A0 td=3D3D3D<unavailable>) at /usr/src/sys/kern/vfs= _vnops.c:1704
> > > >=C2=A0 =C2=A0 =C2=A0#15 0xffffffff809d68ee in devfs_ioct= l_f (fp=3D3D3D<unavailable>, fp=3D
> @entry=3D3D
> > > =3D3D3D<error reading variable: value is not available>= ;, com=3D3D3D<unavail=3D
> able>, c=3D3D
> > > om@entry=3D3D3D<error reading variable: value is not avai= lable>, data=3D3D3=3D
> D<unavai=3D3D
> > > lable>, data@entry=3D3D3D<error reading variable: valu= e is not available>=3D
> ,
> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0cred=3D3D3D<unavail= able>, cred@entry=3D3D3D<error reading variabl=3D
> e: value=3D3D
> > >=C2=A0 is not available>, td=3D3D3D<unavailable>, td= @entry=3D3D3D<error reading v=3D
> ariable:=3D3D
> > >=C2=A0 value is not available>) at /usr/src/sys/fs/devfs/d= evfs_vnops.c:866
> > > >=C2=A0 =C2=A0 =C2=A0#16 0xffffffff80bc57e6 in fo_ioctl (= fp=3D3D3D0xfffff8052cdd80f0, co=3D
> m=3D3D3D32=3D3D
> > > 22821401, data=3D3D3D<unavailable>, active_cred=3D3D3D= <unavailable>, td=3D3D3=3D
> D0xfffffe0=3D3D
> > > 422ef8560) at /usr/src/sys/sys/file.h:367
> > > >=C2=A0 =C2=A0 =C2=A0#17 kern_ioctl (td=3D3D3Dtd@entry=3D= 3D3D0xfffffe0422ef8560, fd=3D3D3D4,=3D
>=C2=A0 com=3D3D3Dcom=3D3D
> > > @entry=3D3D3D3222821401, data=3D3D3D<unavailable>, dat= a@entry=3D3D3D0xfffffe0=3D
> 2fb827d50 =3D3D
> > > "\017") at /usr/src/sys/kern/sys_generic.c:807
> > > >=C2=A0 =C2=A0 =C2=A0#18 0xffffffff80bc54f2 in sys_ioctl = (td=3D3D3D0xfffffe0422ef8560, u=3D
> ap=3D3D3D0=3D3D
> > > xfffffe0422ef8960) at /usr/src/sys/kern/sys_generic.c:715 > > > >=C2=A0 =C2=A0 =C2=A0#19 0xffffffff81049398 in syscallent= er (td=3D3D3D<optimized out>) a=3D
> t /usr=3D3D
> > > /src/sys/amd64/amd64/../../kern/subr_syscall.c:190
> > > >=C2=A0 =C2=A0 =C2=A0#20 amd64_syscall (td=3D3D3D0xfffffe= 0422ef8560, traced=3D3D3D0) at /u=3D
> sr/src/s=3D3D
> > > ys/amd64/amd64/trap.c:1199
> > [...]
> >
> > The backtrace looks different though it certainly smells like PR/= 271945.
> >
> > I've had similar to PR/271945 panics on an amd64 with a mirro= red zpool wi=3D
> th
> > four vdevs running poudriere with AMD64 jails. My other amd64 wit= h a
> > mirrored zpool with two vdevs using i386 jails has no such issue.= All oth=3D
> er
> > workloads are unaffected.
> >
> > On the affected machine running poudriere bulk with -J N:1 circum= vents th=3D
> e
> > issue. So far. There were two openzfs cherry-picks this morning. = I intend
> > to try them against a full bulk build later today.
> >
> >
> > --
> > Cheers,
> > Cy Schubert <Cy.Schubert@cschubert.com>
> > FreeBSD UNIX:=C2=A0 <cy@FreeBSD.org>=C2=A0 =C2=A0Web:=C2=A0= https= ://FreeBSD.org
> > NTP:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<cy@nwtime.org>=C2=A0 =C2=A0 Web:= =C2=A0 = https://nwtime.org
> >
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0e^(i*pi)+1=3D3D0
> >
> >


--000000000000b7db4b06029d5ea5--