panic: lockmgr on FreeBSD 7.0-RELEASE-p4 amd64

[Jeff Wheelhouse]

On Sep 24, 2008, at 12:34 PM, John Baldwin wrote:

> On Wednesday 24 September 2008 12:17:56 pm Jeff Wheelhouse wrote:
>> panic: lockmgr: thread 0xffffff0050858350, not exclusive lock holder
>> 0xffffff00074959f0 unlocking
>> cpuid = 0
>> KDB: stack backtrace:
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
>> panic() at panic+0x17a
>> _lockmgr() at _lockmgr+0x872
>> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0x46
>> null_unlock() at null_unlock+0xff
>> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0x46
>> nullfs_mount() at nullfs_mount+0x244
>> vfs_donmount() at vfs_donmount+0xe4d
>> nmount() at nmount+0xa5
>> syscall() at syscall+0x254
>> Xfast_syscall() at Xfast_syscall+0xab
>> --- syscall (378, FreeBSD ELF64, nmount), rip = 0x206845ac, rsp =
>> 0x7fffffffdfc8, rbp = 0x7fffffffdfd0 ---
>
> Can you use gdb or the like to get the souce file/line for the
> nullfs_mount+0x244 frame?

Got it again, this time with the full debug kernel, and I'm getting  
the same weird results from gdb, so I'll go ahead and post it:

panic: lockmgr: thread 0xffffff0003e499f0, not exclusive lock holder  
0xffffff000a5e16a0 unlocking
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
panic() at panic+0x17a
_lockmgr() at _lockmgr+0x872
VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0x46
null_unlock() at null_unlock+0xff
VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0x46
nullfs_mount() at nullfs_mount+0x244
vfs_donmount() at vfs_donmount+0xe4d
nmount() at nmount+0xa5
syscall() at syscall+0x254
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (378, FreeBSD ELF64, nmount), rip = 0x206845ac, rsp =  
0x7fffffffe1c8, rbp = 0x7fffffffe1d0 ---

$ gdb /boot/kernel/nullfs.ko
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and  
you are
welcome to change it and/or distribute copies of it under certain  
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for  
details.
This GDB was configured as "amd64-marcel-freebsd"...
(gdb) l *nullfs_mount+0x244
0x9c4 is in nullfs_mount (namei.h:163).
158		struct thread *td)
159	{
160		ndp->ni_cnd.cn_nameiop = op;
161		ndp->ni_cnd.cn_flags = flags;
162		ndp->ni_segflg = segflg;
163		ndp->ni_dirp = namep;
164		ndp->ni_cnd.cn_thread = td;
165	}
166	
167	#define NDF_NO_DVP_RELE		0x00000001
(gdb)

(That's NDINIT(), but line 163 doesn't look like it belongs in the  
middle of a call stack.  There's a VOP_UNLOCK a few lines above  
NDINIT() in mount_nullfs(), and another one some ways farther on in  
the function.)

The good news is we took this particular machine out of production and  
came up with a synthetic test based on our in-house code that can  
probably reliably reproduce this within a few minutes.  As you might  
expect, the test involves hammering the same nullfs mount point with  
mounts and umounts from multiple processes without any external  
synchronization.

Thanks,
Jeff