From nobody Thu Oct 28 09:43:19 2021 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 9E2E31834D78 for ; Thu, 28 Oct 2021 09:43:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hg0x33jgMz3DY9 for ; Thu, 28 Oct 2021 09:43:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 60AAF21A17 for ; Thu, 28 Oct 2021 09:43:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 19S9hJAc022142 for ; Thu, 28 Oct 2021 09:43:19 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 19S9hJr3022141 for fs@FreeBSD.org; Thu, 28 Oct 2021 09:43:19 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 258208] [zfs] locks up when using rollback or destroy on both 13.0-RELEASE & sysutils/openzfs port Date: Thu, 28 Oct 2021 09:43:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258208 --- Comment #19 from Andriy Gapon --- (In reply to Mark Johnston from comment #18) Now that you found this I also belatedly recalled that I ran into the same = kind of a problem... I think that getting the teardown lock before the vnode locks would help wi= th the problem like that. Unfortunately, that's not possible (?) to arrange v= ia VOP_LOCK because of the interlock which is not sleepable. I was leaning towards the idea that ZFS should somehow hook the teardown lo= ck into vn_start_write() or something like that. But for ZFS we would also need vn_start_read() as well. And it's not easy = to sprinkle such calls in all places where they are needed... Just in case, here is the 3-way deadlock that I saw: (kgdb) tid 102522 (kgdb) bt #0 sched_switch (td=3D0xfffff802abd53580, newtd=3D0xfffff80008d16580, flags=3D) at /usr/src/sys/kern/sched_ule.c:2005 #1 0xffffffff806a5091 in mi_switch (flags=3D260, newtd=3D0x0) at /usr/src/sys/kern/kern_synch.c:438 #2 0xffffffff806e252c in sleepq_switch (wchan=3D0xfffffe000ec025c8, pri=3D= 0) at /usr/src/sys/kern/subr_sleepqueue.c:611 #3 0xffffffff806e23d2 in sleepq_wait (wchan=3D0xfffffe000ec025c8, pri=3D0)= at /usr/src/sys/kern/subr_sleepqueue.c:690 #4 0xffffffff8063ffa3 in _cv_wait (cvp=3D0xfffffe000ec025c8, lock=3D) at /usr/src/sys/kern/kern_condvar.c:144 #5 0xffffffff8035bc6b in rrw_enter_read_impl (rrl=3D0xfffffe000ec025a8, pr= io=3D0, tag=3D0xfffff8065044cb10, try=3D0) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/rrwlock.c:201 #6 0xffffffff8035bbb3 in rrw_enter_read (rrl=3D, tag=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/rrwlock.c:223 #7 0xffffffff8035c2ff in rrm_enter_read (rrl=3D, tag=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/rrwlock.c:429 #8 0xffffffff8035c2b1 in rrm_enter (rrl=3D, rw=3D, tag=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/rrwlock.c:405 #9 0xffffffff803b81db in zfs_freebsd_lock (ap=3D0xfffffe090e81d4f8) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:6010 #10 0xffffffff8095608d in VOP_LOCK1_APV (vop=3D, a=3D0xfffffe090e81d4f8) at vnode_if.c:2087 #11 0xffffffff8075f8ab in VOP_LOCK1 (vp=3D, flags=3D, file=3D, line=3D) at ./vnode_if.h:859 #12 0xffffffff8075e755 in _vn_lock (vp=3D, flags=3D524544, file=3D0xffffffff80acfa01 "/usr/src/sys/kern/vfs_subr.c", line=3D2768) at /usr/src/sys/kern/vfs_vnops.c:1555 #13 0xffffffff8074ef5f in vputx (vp=3D0xfffff8065044cb10, func=3D1) at /usr/src/sys/kern/vfs_subr.c:2768 #14 0xffffffff8074edbe in vrele (vp=3D) at /usr/src/sys/kern/vfs_subr.c:2804 #15 0xffffffff8073a901 in vn_vptocnp (vp=3D0xfffffe090e81d628, cred=3D0xfffff800234fe000, buf=3D, buflen=3D)= at /usr/src/sys/kern/vfs_cache.c:2232 #16 0xffffffff8073a384 in vn_fullpath1 (td=3D0xfffff802abd53580, vp=3D0xfffff806cd2063b0, rdir=3D0xfffff8001bd8bb10, buf=3D0xfffff801dec4f800 "\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\2= 55\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\= 300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336= \336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\25= 5\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\3= 00\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\= 336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255= \336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\30= 0\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\3= 36\300\255\336\336\300\255\336\336\300\255\336\336\300\255\336\336\300\255\= 336\336\300\255\336\336\300\255\336\336\300\255\336"..., retbuf=3D0xfffffe090e81d770, buflen=3D1007) at /usr/src/sys/kern/vfs_cache.= c:2335 #17 0xffffffff8073a62f in vn_fullpath (td=3D0xfffff802abd53580, vn=3D0xfffff8060b24cce8, retbuf=3D0xfffffe090e81d770, freebuf=3D0xfffffe090= e81d778) at /usr/src/sys/kern/vfs_cache.c:2164 #18 0xffffffff80760af4 in vn_fill_kinfo_vnode (vp=3D0xfffff8060b24cce8, kif=3D0xfffff80232b49018) at /usr/src/sys/kern/vfs_vnops.c:2353 #19 0xffffffff8064f27c in export_vnode_to_kinfo (vp=3D0xfffff8060b24cce8, f= d=3D-1, fflags=3D1, kif=3D0xfffff80232b49018, flags=3D1) at /usr/src/sys/kern/kern_descrip.c:3408 #20 0xffffffff8064e9b4 in export_vnode_to_sb (vp=3D0xfffff8060b24cce8, fd= =3D-1, fflags=3D1, efbuf=3D0xfffff80232b49000) at /usr/src/sys/kern/kern_descrip.c= :3474 #21 0xffffffff8064e84c in kern_proc_filedesc_out (p=3D, sb=3D0xfffffe090e81d888, maxlen=3D-1, flags=3D1) at /usr/src/sys/kern/kern_descrip.c:3537 #22 0xffffffff8064f87d in sysctl_kern_proc_filedesc (oidp=3D, arg1=3D, arg2=3D, req=3D) at /usr/src/sys/kern/kern_descrip.c:3597 #23 0xffffffff806a7d6f in sysctl_root_handler_locked (oid=3D0xffffffff80d93= 760 , arg1=3D0xfffffe090e81da9c, arg2=3D1, req=3D0xfffffe090e81d9d0, tracker=3D0xfffffe090e81d940) at /usr/src/sys/kern/kern_sysctl.c:165 #24 0xffffffff806a7545 in sysctl_root (oidp=3D, arg1=3D, arg2=3D1, req=3D) at /usr/src/sys/kern/kern_sysctl.c:2027 #25 0xffffffff806a7a38 in userland_sysctl (td=3D, name=3D0xfffffe090e81da90, namelen=3D4, old=3D0x0, oldlenp=3D, inkernel=3D, new=3D, newlen=3D, retval=3D, flags=3D0) at /usr/src/sys/kern/kern_sysctl.c:2122 #26 0xffffffff806a78bf in sys___sysctl (td=3D0xfffff802abd53580, uap=3D0xfffff802abd53930) at /usr/src/sys/kern/kern_sysctl.c:2057 #27 0xffffffff808fdecb in syscallenter (td=3D0xfffff802abd53580) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:132 #28 0xffffffff808fda87 in amd64_syscall (td=3D0xfffff802abd53580, traced=3D= 0) at /usr/src/sys/amd64/amd64/trap.c:915 (kgdb) p rrl->rr_writer->td_tid $2 =3D 100351 (kgdb) tid 100351 (kgdb) bt #0 sched_switch (td=3D0xfffff80025288000, newtd=3D0xfffff806ff236580, flags=3D) at /usr/src/sys/kern/sched_ule.c:2005 #1 0xffffffff806a5091 in mi_switch (flags=3D260, newtd=3D0x0) at /usr/src/sys/kern/kern_synch.c:438 #2 0xffffffff806e252c in sleepq_switch (wchan=3D0xfffffe000ec032d8, pri=3D= 0) at /usr/src/sys/kern/subr_sleepqueue.c:611 #3 0xffffffff806e23d2 in sleepq_wait (wchan=3D0xfffffe000ec032d8, pri=3D0)= at /usr/src/sys/kern/subr_sleepqueue.c:690 #4 0xffffffff8063ffa3 in _cv_wait (cvp=3D0xfffffe000ec032d8, lock=3D) at /usr/src/sys/kern/kern_condvar.c:144 #5 0xffffffff8035bde5 in rrw_enter_write (rrl=3D0xfffffe000ec032b8) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/rrwlock.c:253 #6 0xffffffff8035c329 in rrm_enter_write (rrl=3D0xfffffe000ec020e8) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/rrwlock.c:438 #7 0xffffffff8035c2b8 in rrm_enter (rrl=3D, rw=3D, tag=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/rrwlock.c:407 #8 0xffffffff803b364b in zfs_mount (vfsp=3D0xfffff80043895000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:1715 #9 0xffffffff80747c99 in vfs_domount_update (td=3D, vp=3D0xfffff800438b3588, fsflags=3D268505368, optlist=3D0xfffffe090e404a80)= at /usr/src/sys/kern/vfs_mount.c:976 #10 0xffffffff807454fb in vfs_domount (td=3D0xfffff80025288000, fstype=3D, fspath=3D0xfffff804fe9e41a0 "/usr/obj", fsflags=3D, optlist=3D0xfffffe090e404a80) at /usr/src/sys/kern/vfs_mount.c:1132 #11 0xffffffff80744c52 in vfs_donmount (td=3D0xfffff80025288000, fsflags=3D, fsoptions=3D0xfffff8059de54900) at /usr/src/sys/kern/vfs_mount.c:687 #12 0xffffffff80744608 in sys_nmount (td=3D0xfffff80025288000, uap=3D) at /usr/src/sys/kern/vfs_mount.c:421 #13 0xffffffff808fdecb in syscallenter (td=3D0xfffff80025288000) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:132 #14 0xffffffff808fda87 in amd64_syscall (td=3D0xfffff80025288000, traced=3D= 0) at /usr/src/sys/amd64/amd64/trap.c:915 (kgdb) tid 102044 (kgdb) bt #0 sched_switch (td=3D0xfffff806f0cb0580, newtd=3D0xfffff80008d17000, flags=3D) at /usr/src/sys/kern/sched_ule.c:2005 #1 0xffffffff806a5091 in mi_switch (flags=3D260, newtd=3D0x0) at /usr/src/sys/kern/kern_synch.c:438 #2 0xffffffff806e252c in sleepq_switch (wchan=3D0xfffff8065044cb78, pri=3D= 96) at /usr/src/sys/kern/subr_sleepqueue.c:611 #3 0xffffffff806e23d2 in sleepq_wait (wchan=3D0xfffff8065044cb78, pri=3D96= ) at /usr/src/sys/kern/subr_sleepqueue.c:690 #4 0xffffffff8067311f in sleeplk (lk=3D0xfffff8065044cb78, flags=3D2121728, ilk=3D, wmesg=3D0xffffffff80aae927 "zfs", pri=3D96, timo=3D5= 1, queue=3D) at /usr/src/sys/kern/kern_lock.c:288 #5 0xffffffff80671c05 in __lockmgr_args (lk=3D0xfffff8065044cb78, flags=3D= 2121728, ilk=3D0xfffff8065044cba8, wmesg=3D, pri=3D, timo=3D, file=3D, line=3D) at /usr/src/sys/kern/kern_lock.c:873 #6 0xffffffff806711c5 in lockmgr_lock_fast_path (lk=3D0xfffff8065044cb78, flags=3D2121728, ilk=3D0xfffff8065044cba8, file=3D0xffffffff80acfa01 "/usr/src/sys/kern/vfs_subr.c", line=3D2606) at /usr/src/sys/kern/kern_lock= .c:605 #7 0xffffffff8073d939 in vop_stdlock (ap=3D) at /usr/src/sys/kern/vfs_default.c:517 #8 0xffffffff803b81e8 in zfs_freebsd_lock (ap=3D0xfffffe090e8a9588) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:5987 #9 0xffffffff8095608d in VOP_LOCK1_APV (vop=3D, a=3D0xfffffe090e8a9588) at vnode_if.c:2087 #10 0xffffffff8075f8ab in VOP_LOCK1 (vp=3D, flags=3D, file=3D, line=3D) at ./vnode_if.h:859 #11 0xffffffff8075e755 in _vn_lock (vp=3D, flags=3D2121728, file=3D0xffffffff80acfa01 "/usr/src/sys/kern/vfs_subr.c", line=3D2606) at /usr/src/sys/kern/vfs_vnops.c:1555 #12 0xffffffff8074e5af in vget (vp=3D0xfffff8065044cb10, flags=3D, td=3D0xfffff806f0cb0580) at /usr/src/sys/kern/vfs_subr.c:2606 #13 0xffffffff8073719c in cache_lookup (dvp=3D0xfffff806cd2063b0, vpp=3D0xfffffe090e8a9a18, cnp=3D0xfffffe090e8a9a40, tsp=3D0x0, ticksp=3D) at /usr/src/sys/kern/vfs_cache.c:1321 #14 0xffffffff8073a0a4 in vfs_cache_lookup (ap=3D) at /usr/src/sys/kern/vfs_cache.c:2067 #15 0xffffffff803b8391 in zfs_cache_lookup (ap=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4956 #16 0xffffffff8095275a in VOP_LOOKUP_APV (vop=3D, a=3D0xfffffe090e8a9720) at vnode_if.c:127 #17 0xffffffff80743779 in VOP_LOOKUP (dvp=3D, vpp=3D, cnp=3D) at ./vnode_if.h:54 #18 0xffffffff80742bce in lookup (ndp=3D0xfffffe090e8a99b8) at /usr/src/sys/kern/vfs_lookup.c:891 #19 0xffffffff807424a7 in namei (ndp=3D0xfffffe090e8a99b8) at /usr/src/sys/kern/vfs_lookup.c:453 #20 0xffffffff8075e046 in vn_open_cred (ndp=3D0xfffffe090e8a99b8, flagp=3D0xfffffe090e8a9ac4, cmode=3D0, vn_open_flags=3D0, cred=3D0xfffff800= 234fe000, fp=3D0xfffff8072ca41b90) at /usr/src/sys/kern/vfs_vnops.c:277 #21 0xffffffff8075de4f in vn_open (ndp=3D, flagp=3D, cmode=3D, fp=3D) at /usr/src/sys/kern/vfs_vnops.c= :180 #22 0xffffffff80757b2f in kern_openat (td=3D0xfffff806f0cb0580, fd=3D-100, path=3D0x800de1a80 , pathseg=3DUIO_USERSPACE, flags=3D1, mode=3D) at /usr/src/sys/kern/vfs_syscalls.c:1082 #23 0xffffffff80757d2b in sys_openat (td=3D0xfffff806f0cb0580, uap=3D0xfffff806f0cb0930) at /usr/src/sys/kern/vfs_syscalls.c:1030 #24 0xffffffff808fdecb in syscallenter (td=3D0xfffff806f0cb0580) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:132 #25 0xffffffff808fda87 in amd64_syscall (td=3D0xfffff806f0cb0580, traced=3D= 0) at /usr/src/sys/amd64/amd64/trap.c:915 Summary: 1. Thread 102044 is doing a lookup, dvp must be already locked and thus the filesystem is read-locked The thread is blocked on the child vp lock. 2. Thread 100351 is doing mount -u and wants to write-lock the filesystem, blocked by thread 102044. 3. Thread 102522 does vrele on the child vp and needs to lock it. The thread has already got the vnode lock is blocked by thread 100351. --=20 You are receiving this mail because: You are the assignee for the bug.=