stable/10 r258317 - UFS lock problem

[Özkan KIRIK]

Hi,

I compiled GENERIC kernel with;

makeoptions DEBUG=-g
options DDB
options INVARIANTS
options INVARIANT_SUPPORT
options WITNESS
options WITNESS_SKIPSPIN
options DEBUG_LOCKS
options DEBUG_VFS_LOCKS
options DIAGNOSTIC


When I run portsnap fetch extract, I catched these situations on console:

lock order reversal:
 1st Oxfffffe0080531888 bufwait (bufwait) • ,usr/src/sys/kern/vfs
bio.c:3050
 2nd Oxfffff8000d48a400 dirhash (dirhash)
/usr/src/sys/ufs/ufs/ufs_dirhash.c:284
KDB: stack backtrace:
kdb_trace_self_wrapper() at db_trace_self_wrapper.0x2b/frame
Oxfffffe009aaa3700
db_backtrace() at kdb backtrace.0x39/frame Oxfffffe009aaa37b0
witness_checkorder() at witness_checkorder.Oxd23/frame Oxfffffe009aaa3840
_sx_xlock() at _sx xlock.0x75/frame Oxfffffe009aaa3880
ufsdirhash_remove() at ufsdirhash remove.0x47/frame Oxfffffe009aaa38b0
ufs_dirremove() at ufs_dirremove.0x11b/frame Oxfffffe009aaa3900
ufs_rmdir() at ufs_rmdir.Oxe3/frame Oxfffffe009aaa3940
VOP_RMDIR_APV() at VOP RHDIR APV.0x12d/frame Oxfffffe009aaa3970
kern_rmdirat() at kern rmdirat.0x1b8/frame Oxfffffe009aaa3ae0
amd64_syscall() at amd64_syscal1.0x282/frame Oxfffffe009aaa3bf0
Xfast_syscall() at Xfast_syscall.Oxfb/frame Oxfffffe009aaa3bf0
--- syscall (137. FreeBSD ELF64. sys_rmdir). rip = Ox80088alfa. rsp =
Ox7fffffffd998, rbp = Ox7fffffffda60 ---


"ps" output over ssh is below:

# ps ax | grep D+
29730  0  D+    0:00.00 tar -xz --numeric-owner -f
/var/db/portsnap/files/cafbda170898ff0143598e1363c08fc3ac82ba2f4d43334fc53b251dbcd28063.gz
-C /usr/ports/games/highmoon/ (bsdtar)





On Sun, Dec 8, 2013 at 9:38 PM, Özkan KIRIK <ozkan.kirik at gmail.com> wrote:

> Hi,
>
> You're right Peter. Now i am compiling with DDB option.
> But
> http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html page,
> dont tell about the "DDB" option.
>
> Thank you,
>
>
> On Sat, Dec 7, 2013 at 8:30 PM, Peter Holm <peter at holm.cc> wrote:
>
>> On Sat, Dec 07, 2013 at 08:24:32AM +0200, Özkan KIRIK wrote:
>> > Hi,
>> >
>> > I'm trying to compile a debug kernel with these options :
>> >
>> > makeoptions DEBUG=-g
>> > options INVARIANTS
>> >  options INVARIANT_SUPPORT
>> > options WITNESS
>> >  options DEBUG_LOCKS
>> > options DEBUG_VFS_LOCKS
>> >  options DIAGNOSTIC
>> >
>> > While compiling; compilations fails due to this error :
>> > /usr/src/sys/kern/subr_vmem.c:1357:1: error: unused function 'vmem_dump'
>> >     [-Werror, -Wunused-function]
>> >
>>
>> Did you forget "options DDB" ?
>>
>> - Peter
>>
>> >
>> > I'll comment out this function and recompiled kernel.
>> > Now, while booting, kernel panics. Error line is below;  KDB stack
>> > backtrace is attached as PNG screenshot.
>> >
>> > "panic: mtx_lock_spin: recursed on non-recursive mutex cnput_mtx @
>> > /usr/src/sys/kern/kern_cons.c:500"
>> >
>> > I think, i hit an other bug.
>> >
>> > Best wishes,
>> >
>> > Ozkan KIRIK
>> > ePati Information Technologies
>> > http://www.epati.com.tr/
>> >
>> >
>> > On Fri, Dec 6, 2013 at 10:13 PM, Konstantin Belousov <
>> kostikbel at gmail.com>wrote:
>> >
>> > > On Fri, Dec 06, 2013 at 08:10:55PM +0200, ?zkan KIRIK wrote:
>> > > > Hi,
>> > > >
>> > > > I am using FreeBSD 10.0-BETA3 r258317 GENERIC kernel.
>> > > >
>> > > > I run "portsnap fetch extract".
>> > > > While extracting files, extracting operation hangs, and i cant kill
>> > > process.
>> > > >
>> > > > Output of : "ps ax | grep D+"  ( uninterruptable process list )
>> > > > % ps ax | grep D+
>> > > > 28933  0  D+    0:00.01 tar -xz --numeric-owner -f
>> > > >
>> > >
>> /var/db/portsnap/files/77d41f10d2832f8450e4e02a4db5c0a6131c97d15076ed0c76e761c9ce58338d.gz
>> > > > -C /usr/ports/emulators/qemu/ (bsdtar)
>> > > > 29051  1  S+    0:00.00 grep D+
>> > > > %
>> > > >
>> > > >
>> > > > Because of D flag, process cannot be interrupted even by SIGKILL.
>> > > > I cannot remove usr/ports folder while this process on D+ state.
>> > > > If I run rm -r /usr/ports, rm process is flagged as D+ also.
>> > > >
>> > > > I tested this situation on both vmware workstation and different
>> real
>> > > > hardware installations. ( Sun Fire X4150, HP DL380G4, Sun X3
>> servers. )
>> > > >
>> > > > Problem is repeatable. If you install FreeBSD on ZFS root, problem
>> > > doesn't
>> > > > occur.
>> > > >
>> > > >
>> > > > # mount
>> > > > /dev/da0p2 on / (ufs, local, journaled soft-updates)
>> > > > devfs on /dev (devfs, local, multilabel)
>> > >
>> > > See
>> > >
>> > >
>> http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html
>> > >
>> > _______________________________________________
>>
>
>