ZFS gets stuck

[Stefan Bethke]

For the past week, I've been stress testing two new boxes by running  
make -j4 universe. /usr/src is on ufs, /usr/obj on zfs, backed by a  
single disk pool. Every so often (about once every one or two days),  
processes start getting wedged on accessing the zfs file systems. Just  
this morning I found:

output from make universe:
 >> sparc64 started on Sun Dec  7 01:34:41 UTC 2008
 >> amd64 completed on Sun Dec  7 02:18:39 UTC 2008
 >> pc98 buildworld completed on Sun Dec  7 02:24:35 UTC 2008
 >> sun4v started on Sun Dec  7 02:30:45 UTC 2008

ps shows me these processes that apparently are waiting on some zfs  
operation:
     0 81836 81830   0  47  0  6732  4056 zio->i D     ??    0:13.11  
find -sx / /tank /tank/ports /usr/obj /dev/null -type f ( -perm -u+x - 
or -perm -g+x -or -perm -o+x ) ( -perm -u+s -or -perm -g+s ) -exec ls - 
liTd {} +
     0 86345 86344   0  76  0 11392 10300 zio->i D+     0    0:00.53  
make all DIRPRFX=kerberos5/lib/libasn1/
     0 91920 91420   0  76  0  7144  1792 zfsvfs D+     0    0:00.00  
sh -ev
     0 91923 91902   0  76  0  2180   400 zfsvfs D+     0    0:00.00  
[cc]
     0 91924 91900   0  76  0 10456  7572 zfsvfs D+     0    0:00.02 / 
usr/obj/pc98/usr/src/tmp/usr/libexec/cc1 -E -quiet -nostdinc -I. -I@ - 
I@/contrib/altq -I/usr/obj/pc98/usr/src/sys/MAC -M -D_LONGLONG -DPC98 - 
D_KERNEL -DKLD_MODULE -DHAVE_KERNEL_OPTION_HEADERS /usr/src/sys/ 
modules/smbfs/../../netsmb/smb_dev.c

df still works, ls /tank blocks.

FreeBSD lokschuppen.lassitu.de 8.0-CURRENT FreeBSD 8.0-CURRENT #1: Wed  
Dec  3 07:05:03 UTC 2008     root at lokschuppen.lassitu.de:/usr/obj/usr/ 
src/sys/EISENBOOT  amd64

So far, I've had this in loader.conf:
vfs.zfs.arc_max="512M"
vfs.zfs.prefetch_disable="1"


I'm now adding
vfs.zfs.zil_disable="1"

to see if that makes a difference.

Is there anything in particular people would want me to check out?  
Kernel is GENERIC minus a number of devices, and without INVARIANTS  
and WITNESS.


Stefan

-- 
Stefan Bethke <stb at lassitu.de>   Fon +49 170 346 0140