witness performance improvements

[Lars Engels]

On Fri, Jul 18, 2008 at 04:41:58PM -1000, Jeff Roberson wrote:
> Hello,
> 
> I have a patch that improves witness performance available at:
> 
> http://people.freebsd.org/~jeff/witness.diff
> 
> This improvement comes at the cost of some significant space overhead.
> It changes the witness graph from a linked tree to a matrix based
> approach. Relationships can be quickly resolved with a table lookup.
> The table size is WITNESS_COUNT^2, or 1MB with the current count of
> 1024.
> 
> This patch also makes struct witness objects persistent even after the
> last lock using this name has been removed.  This is helpful for short
> lived objects which may be created frequently.
> 
> To reduce lock contention on SMP witness_checkorder() now runs without
> the w_mtx when there are no lock violations.  I also cache a
> lock_list_entry in each thread as allocating these requires the w_mtx.
> The entry is disposed of at thread_exit().
> 
> There is also a new sysctl that produces dot output which graphs lock
> order relationships with the graphviz program.
> 
> Most of this work was done by Ilya Maykov while he was at Isilon
> systems. The locking work and some cleanup/porting/refinement was done
> by me on behalf of Nokia.
> 
> The performance improvement can be significant.  It is only on the
> order of 10-20% for buildkernel but on a packet forwarding test at
> nokia it sped things up by 5x putting a witness enabled kernel within
> about 50% of the performance of a kernel without.  I believe
> buildworld isn't helped as much because forking and exiting a lot
> would then contend on the witness lock.
> 
> I'm mostly interested in hearing what people have to say about the
> space bloat.  I believe it is in a commit ready state.
> 
> Thanks,
> Jeff

The speed improvement is significant here (Core Duo Machine). The kernel
build time went from ~8:30 min to 5:30 min. 
But when I run sysctl -a the kernel panics.
kgdb output:

Unread portion of the kernel message buffer:
panic: Assertion mcount == fcount failed at
/usr/src/sys/kern/subr_witness.c:2882
cpuid = 1
KDB: enter: panic
panic: from debugger
cpuid = 1
KDB: stack backtrace:
Physical memory: 1002 MB
Dumping 68 MB: 53 37 21 5

(kgdb) bt
#0  doadump () at pcpu.h:196
#1  0xc0476c69 in db_fncall (dummy1=-1061959744, dummy2=0, dummy3=3,
dummy4=0xe54ce8ec "") at /usr/src/sys/ddb/db_command.c:516
#2  0xc0477218 in db_command (last_cmdp=0xc099a9b0, cmd_table=0x0,
dopager=1) at /usr/src/sys/ddb/db_command.c:413
#3  0xc047734a in db_command_loop () at
/usr/src/sys/ddb/db_command.c:466
#4  0xc0478b3d in db_trap (type=3, code=0) at
/usr/src/sys/ddb/db_main.c:228
#5  0xc0618d56 in kdb_trap (type=3, code=0, tf=0xe54cea94) at
/usr/src/sys/kern/subr_kdb.c:534
#6  0xc087ec36 in trap (frame=0xe54cea94) at
/usr/src/sys/i386/i386/trap.c:683
#7  0xc08634bb in calltrap () at /usr/src/sys/i386/i386/exception.s:165
#8  0xc0618eda in kdb_enter (why=0xc08dc49c "panic", msg=0xc08dc49c
"panic") at cpufunc.h:60
#9  0xc05ebe9c in panic (fmt=0xc08d72ed "Assertion %s failed at %s:%d")
at /usr/src/sys/kern/kern_shutdown.c:556
#10 0xc062bf62 in sysctl_debug_witness_cyclegraph (oidp=0xc096bec0,
arg1=0x0, arg2=0, req=0xe54ceba4) at
/usr/src/sys/kern/subr_witness.c:2882
#11 0xc05f59d7 in sysctl_root (oidp=Variable "oidp" is not available.
) at /usr/src/sys/kern/kern_sysctl.c:1325
#12 0xc05f5b05 in userland_sysctl (td=0xc49c08c0, name=0xe54cec10,
namelen=3, old=0x0, oldlenp=0xbfbfd9b0, inkernel=0, new=0x0, newlen=0,
retval=0xe54cec70, flags=0)
    at /usr/src/sys/kern/kern_sysctl.c:1420
#13 0xc05f5f4c in __sysctl (td=0xc49c08c0, uap=0xe54cecf8) at
/usr/src/sys/kern/kern_sysctl.c:1355
#14 0xc087e3b3 in syscall (frame=0xe54ced38) at
/usr/src/sys/i386/i386/trap.c:1081
#15 0xc0863520 in Xint0x80_syscall () at
/usr/src/sys/i386/i386/exception.s:261
#16 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)

The kernel sources are one week old.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-arch/attachments/20080719/9cb5d78a/attachment.pgp