Kernel panic after excessive access to NFSv4 as client (FreeBSD-9.1)

Sun Dec 9 18:40:54 UTC 2012

Hi,

My system has been encountering kernel panic every time it accesses 
large number of files over NFSv4.
The backtrace left after reboot (I haven't saved the core dump):
Dec  4 17:35:56 freebsd syslogd: kernel boot file is 
/boot/kernel/kernel
Dec  4 17:35:56 freebsd kernel: panic: stack overflow detected; 
backtrace may be corrupted
Dec  4 17:35:56 freebsd kernel: cpuid = 1
Dec  4 17:35:56 freebsd kernel: KDB: stack backtrace:
Dec  4 17:35:56 freebsd kernel: #0 0xffffffff809274a6 at 
kdb_backtrace+0x66
Dec  4 17:35:56 freebsd kernel: #1 0xffffffff808f13fe at panic+0x1ce
Dec  4 17:35:56 freebsd kernel: #2 0xffffffff8091a452 at 
__stack_chk_fail+0x12
Dec  4 17:35:56 freebsd kernel: #3 0xffffffff81613ee7 at 
krb5_unwrap_old+0x407
Dec  4 17:35:56 freebsd kernel: #4 0xffffffff8161413c at 
krb5_unwrap+0x12c
Dec  4 17:35:56 freebsd kernel: #5 0xffffffff816446e4 at 
xdr_rpc_gss_unwrap_data+0x164
Dec  4 17:35:56 freebsd kernel: #6 0xffffffff81642a9d at 
rpc_gss_validate+0x1bd
Dec  4 17:35:56 freebsd kernel: #7 0xffffffff80ad09be at 
clnt_vc_call+0x8de
Dec  4 17:35:56 freebsd kernel: #8 0xffffffff80acec0b at 
clnt_reconnect_call+0xfb
Dec  4 17:35:56 freebsd kernel: #9 0xffffffff807f07e5 at 
newnfs_request+0x595
Dec  4 17:35:56 freebsd kernel: #10 0xffffffff80826c42 at 
nfscl_request+0x72
Dec  4 17:35:56 freebsd kernel: #11 0xffffffff808117dd at 
nfsrpc_lookup+0x1bd
Dec  4 17:35:56 freebsd kernel: #12 0xffffffff80821982 at 
nfs_lookup+0x292
Dec  4 17:35:56 freebsd kernel: #13 0xffffffff80c6ee60 at 
VOP_LOOKUP_APV+0x40
Dec  4 17:35:56 freebsd kernel: #14 0xffffffff8097abaf at lookup+0x41f
Dec  4 17:35:56 freebsd kernel: #15 0xffffffff8097bcab at namei+0x4db
Dec  4 17:35:56 freebsd kernel: #16 0xffffffff8098d203 at 
kern_statat_vnhook+0xb3
Dec  4 17:35:56 freebsd kernel: #17 0xffffffff8098d3c5 at 
kern_statat+0x15
Dec  4 17:35:56 freebsd kernel: Uptime: 21h22m34s
Dec  4 17:35:56 freebsd kernel: Automatic reboot in 15 seconds - press 
a key on the console to abort
Dec  4 17:35:56 freebsd kernel: Rebooting...

Here's another backtrace:
Dec  2 00:15:33 freebsd syslogd: kernel boot file is 
/boot/kernel/kernel
Dec  2 00:15:33 freebsd kernel: panic: stack overflow detected; 
backtrace may be corrupted
Dec  2 00:15:33 freebsd kernel: cpuid = 3
Dec  2 00:15:33 freebsd kernel: KDB: stack backtrace:
Dec  2 00:15:33 freebsd kernel: #0 0xffffffff809274a6 at 
kdb_backtrace+0x66
Dec  2 00:15:33 freebsd kernel: #1 0xffffffff808f13fe at panic+0x1ce
Dec  2 00:15:33 freebsd kernel: #2 0xffffffff8091a452 at 
__stack_chk_fail+0x12
Dec  2 00:15:33 freebsd kernel: #3 0xffffffff81613ee7 at 
krb5_unwrap_old+0x407
Dec  2 00:15:33 freebsd kernel: #4 0xffffffff8161413c at 
krb5_unwrap+0x12c
Dec  2 00:15:33 freebsd kernel: #5 0xffffffff816446e4 at 
xdr_rpc_gss_unwrap_data+0x164
Dec  2 00:15:33 freebsd kernel: #6 0xffffffff81642a9d at 
rpc_gss_validate+0x1bd
Dec  2 00:15:33 freebsd kernel: #7 0xffffffff80ad09be at 
clnt_vc_call+0x8de
Dec  2 00:15:33 freebsd kernel: #8 0xffffffff80acec0b at 
clnt_reconnect_call+0xfb
Dec  2 00:15:33 freebsd kernel: #9 0xffffffff807f07e5 at 
newnfs_request+0x595
Dec  2 00:15:33 freebsd kernel: #10 0xffffffff80826c42 at 
nfscl_request+0x72
Dec  2 00:15:33 freebsd kernel: #11 0xffffffff8080fcd4 at 
nfsrpc_write+0x4e4
Dec  2 00:15:33 freebsd kernel: #12 0xffffffff8081ea72 at 
ncl_writerpc+0x62
Dec  2 00:15:33 freebsd kernel: #13 0xffffffff80829f26 at 
ncl_doio+0x196
Dec  2 00:15:33 freebsd kernel: #14 0xffffffff8082e002 at 
nfssvc_iod+0xc2
Dec  2 00:15:33 freebsd kernel: #15 0xffffffff808c1f0f at 
fork_exit+0x11f
Dec  2 00:15:33 freebsd kernel: #16 0xffffffff80bc8c3e at 
fork_trampoline+0xe
Dec  2 00:15:33 freebsd kernel: Uptime: 2h6m33s
Dec  2 00:15:33 freebsd kernel: Automatic reboot in 15 seconds - press 
a key on the console to abort
Dec  2 00:15:33 freebsd kernel: Rebooting...

# uname -a
FreeBSD freebsd.nj-k.org 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #5 
r243328: Sat Nov 24 03:01:20 JST 2012     
root@:/usr/obj/usr/src/sys/GENERIC  amd64

Disabling rpcbind stopped the kernel panic for a while.
# cat /etc/rc.conf (truncated lines for apache and etc.)
hostname="freebsd.nj-k.org"
keymap="us.iso.kbd"
ifconfig_re0="DHCP"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
nscd_enable="YES"
##rpcbind_enable="YES"
gssd_enable="YES"
nfsuserd_enable="YES"
nfsuserd_flags="-domain nj-k.org"
nfscbd_enable="YES"
nfs_client_enable="YES"
sendmail_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"

# cat /etc/fstab
# Device	Mountpoint	FStype	Options	Dump	Pass#
/dev/vtbd0p2	/		ufs	rw	1	1
/dev/vtbd0p3	none		swap	sw	0	0
nfs.nj-k.org:/	/mnt/nfs	nfs	rw,nfsv4,sec=krb5p,rsize=32768,wsize=32768,intr	0	0

# klist
Credentials cache: FILE:/tmp/krb5cc_0
         Principal: host/freebsd.nj-k.org at NJ-K.ORG

   Issued           Expires          Principal
Dec 10 00:45:00  Dec 10 10:45:00  krbtgt/NJ-K.ORG at NJ-K.ORG
Dec 10 00:46:00  Dec 10 10:45:00  ldap/debian.nj-k.org at NJ-K.ORG

I previously had the same problem with 9.0-RELEASE, and thought 
installing 9-STABLE might solve the problem.
After deinstalling security/krb5, I have rebuilt world, kernel and 
ports.
I have following lines in /etc/krb5.conf under [libdefaults]:
         forwardable = true
         dns_lookup_kdc = true
         dns_lookup_realm = true
         allow_weak_crypto = true
         default_etypes = des-cbc-crc
         default_etypes_des = des-cbc-crc
         fcc-mit-ticketflags = true

Lines after allow_weak_crypto were added because I thought the problem 
was related to the encryption type.
The user of the service which accesses the nfs is www which is in 
/etc/passwd and ldap
# id www (truncated some groups)
uid=80(www) gid=80(www) groups=80(www),110(ssl-cert)
# su www -c 'KRB5CCNAME=FILE:/tmp/krb5cc_80 klist -v'
Credentials cache: FILE:/tmp/krb5cc_80
         Principal: www at NJ-K.ORG
     Cache version: 4

Server: krbtgt/NJ-K.ORG at NJ-K.ORG
Client: www at NJ-K.ORG
Ticket etype: des-cbc-crc, kvno 1
Ticket length: 204
Auth time:  Dec 10 00:55:00 2012
End time:   Dec 10 10:55:00 2012
Ticket flags: forwardable, initial, pre-authenticated
Addresses: addressless

I am new to FreeBSD (I installed my first FreeBSD system 6 months ago), 
but willing to get adapted to it.
This system is on kvm so I can freely create another instance of the 
system for test.
Any help or advice is appreciated!!!
Please tell me if this mail was not a well-formed question (this is my 
first mail to any mailing lists).

Best regards,
Tak