Re: FreeBSD panics possibly caused by nfs clients
Date: Mon, 19 Feb 2024 15:43:43 UTC
Hi all,
So I finally induced a panic on a "pure" ufs system - root and exported
filesystem were both ufs. So, I think this definitively rules out zfs as
a source of the issue.
This panic was on 14.0p5 without debugging options, so the core may not
be helpful. The panic and backtrace are below in case they're
interesting to anyone.
Next, I'm going to try a CURRENT kernel without debugging options
enabled just to see if I can finally induce a panic here. My other two
VMs running CURRENT with full debugging are still clanking along.
-Matt
[218716] Fatal trap 12: page fault while in kernel mode
[218716] cpuid = 4; apic id = 08
[218716] fault virtual address = 0x100000017
[218716] fault code = supervisor read data, page not present
[218716] instruction pointer = 0x20:0xffffffff80e9165d
[218716] stack pointer = 0x28:0xfffffe010b5aa3b0
[218716] frame pointer = 0x28:0xfffffe010b5aa400
[218716] code segment = base 0x0, limit 0xfffff, type 0x1b
[218716] = DPL 0, pres 1, long 1, def32 0, gran 1
[218716] processor eflags = interrupt enabled, resume, IOPL = 0
[218716] current process = 49575 (nfsd: service)
[218716] rdi: 00000000ffffffff rsi: fffff800038ec900 rdx: fffffe00d9326000
[218716] rcx: 0000000000030eb0 r8: 0000000000000000 r9: fffffe010b5aa410
[218716] rax: 00000000008f0eb0 rbx: fffff8038ac4cd00 rbp: fffffe010b5aa400
[218716] r10: 0000000000000000 r11: 0000000000000000 r12: 0000000000000000
[218716] r13: fffff80003647c00 r14: fffff802f9dced00 r15: fffff800038ec900
[218716] trap number = 12
[218716] panic: page fault
[218716] cpuid = 4
[218716] time = 1708319487
[218716] KDB: stack backtrace:
[218716] #0 0xffffffff80b9309d at kdb_backtrace+0x5d
[218716] #1 0xffffffff80b461a2 at vpanic+0x132
[218716] #2 0xffffffff80b46063 at panic+0x43
[218716] #3 0xffffffff8101d85c at trap_fatal+0x40c
[218716] #4 0xffffffff8101d8af at trap_pfault+0x4f
[218716] #5 0xffffffff80ff3fe8 at calltrap+0x8
[218716] #6 0xffffffff80e8716e at newdirrem+0x8be
[218716] #7 0xffffffff80e866fa at softdep_setup_remove+0x1a
[218716] #8 0xffffffff80ea71af at ufs_dirremove+0x21f
[218716] #9 0xffffffff80ead4f4 at ufs_remove+0xb4
[218716] #10 0xffffffff810f1428 at VOP_REMOVE_APV+0x28
[218716] #11 0xffffffff80a60db4 at nfsvno_removesub+0xc4
[218716] #12 0xffffffff80a52699 at nfsrvd_remove+0x1b9
[218716] #13 0xffffffff80a374d4 at nfsrvd_dorpc+0x1854
[218716] #14 0xffffffff80a4e76f at nfssvc_program+0x82f
[218716] #15 0xffffffff80e34080 at svc_run_internal+0xb50
[218716] #16 0xffffffff80e3475b at svc_thread_start+0xb
[218716] #17 0xffffffff80b00b7f at fork_exit+0x7f
[218716] Uptime: 2d12h45m16s
[218716] Dumping 985 out of 16350
MB:..2%..12%..22%..31%..41%..51%..61%..72%..82%..91%
#0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
#1 doadump (textdump=<optimized out>) at
/usr/src/sys/kern/kern_shutdown.c:405
#2 0xffffffff80b45d37 in kern_reboot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:526
#3 0xffffffff80b4620f in vpanic (fmt=0xffffffff81147c9c "%s",
ap=ap@entry=0xfffffe010b5aa200) at
/usr/src/sys/kern/kern_shutdown.c:970
#4 0xffffffff80b46063 in panic (fmt=<unavailable>)
at /usr/src/sys/kern/kern_shutdown.c:894
#5 0xffffffff8101d85c in trap_fatal (frame=0xfffffe010b5aa2f0,
eva=4294967319)
at /usr/src/sys/amd64/amd64/trap.c:952
#6 0xffffffff8101d8af in trap_pfault (frame=0xfffffe010b5aa2f0,
usermode=false, signo=<optimized out>, ucode=<optimized out>)
at /usr/src/sys/amd64/amd64/trap.c:760
#7 <signal handler called>
#8 cancel_diradd (dap=0xfffff8038ac4cd00,
dirrem=dirrem@entry=0xfffff800038ec900,
jremref=jremref@entry=0xfffff802f9dced00,
dotremref=dotremref@entry=0x0,
dotdotremref=dotdotremref@entry=0x0)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:9028
#9 0xffffffff80e8716e in newdirrem (bp=<optimized out>,
dp=dp@entry=0xfffff800037fea80, ip=ip@entry=0xfffff8006b3b9300,
isrmdir=isrmdir@entry=0,
prevdirremp=prevdirremp@entry=0xfffffe010b5aa4b0)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:9480
#10 0xffffffff80e866fa in softdep_setup_remove (bp=0xffffffff,
dp=0xfffff800038ec900, dp@entry=0xfffff800037fea80,
ip=0xfffffe00d9326000,
ip@entry=0xfffff8006b3b9300, isrmdir=200368, isrmdir@entry=0)
at /usr/src/sys/ufs/ffs/ffs_softdep.c:9176
#11 0xffffffff80ea71af in ufs_dirremove (dvp=dvp@entry=0xfffff801f764be00,
ip=ip@entry=0xfffff8006b3b9300, flags=<optimized out>,
isrmdir=isrmdir@entry=0) at /usr/src/sys/ufs/ufs/ufs_lookup.c:1198
#12 0xffffffff80ead4f4 in ufs_remove (ap=0xfffffe010b5aa5d8)
at /usr/src/sys/ufs/ufs/ufs_vnops.c:1054
#13 0xffffffff810f1428 in VOP_REMOVE_APV (
vop=0xffffffff8172f2d0 <ffs_vnodeops2>, a=a@entry=0xfffffe010b5aa5d8)
at vnode_if.c:1534
#14 0xffffffff80a60db4 in VOP_REMOVE (dvp=0x8f0eb0, vp=0xfffff800539b7380,
cnp=0x30eb0) at ./vnode_if.h:789
#15 nfsvno_removesub (ndp=0xfffffe010b5aa858, is_v4=<optimized out>,
cred=<optimized out>, p=p@entry=0xfffffe010ae803a0,
exp=exp@entry=0xfffffe010b5aaa88)
at /usr/src/sys/fs/nfsserver/nfs_nfsdport.c:1466
#16 0xffffffff80a52699 in nfsrvd_remove (nd=0xfffffe010b5aabf8,
isdgram=<optimized out>, dp=<optimized out>, exp=0xfffffe010b5aaa88)
at /usr/src/sys/fs/nfsserver/nfs_nfsdserv.c:1583
#17 0xffffffff80a374d4 in nfsrvd_compound (nd=0xfffffe010b5aabf8, isdgram=0,
tag=<optimized out>, taglen=0, minorvers=<optimized out>)
at /usr/src/sys/fs/nfsserver/nfs_nfsdsocket.c:1322
#18 nfsrvd_dorpc (nd=nd@entry=0xfffffe010b5aabf8, isdgram=isdgram@entry=0,
tag=<optimized out>, taglen=0, minorvers=<optimized out>)
at /usr/src/sys/fs/nfsserver/nfs_nfsdsocket.c:633
#19 0xffffffff80a4e76f in nfs_proc (nd=0xfffffe010b5aabf8,
xid=<optimized out>, xprt=0xfffff800037b4800, rpp=<optimized out>)
at /usr/src/sys/fs/nfsserver/nfs_nfsdkrpc.c:464
#20 nfssvc_program (rqst=0xfffff80128ac9000, xprt=0xfffff800037b4800)
at /usr/src/sys/fs/nfsserver/nfs_nfsdkrpc.c:348
#21 0xffffffff80e34080 in svc_executereq (rqstp=0xfffff80128ac9000)
at /usr/src/sys/rpc/svc.c:1036
#22 svc_run_internal (grp=grp@entry=0xfffff800033c2100,
ismaster=ismaster@entry=0) at /usr/src/sys/rpc/svc.c:1312
#23 0xffffffff80e3475b in svc_thread_start (arg=0xffffffff,
arg@entry=0xfffff800033c2100) at /usr/src/sys/rpc/svc.c:1340
#24 0xffffffff80b00b7f in fork_exit (
callout=0xffffffff80e34750 <svc_thread_start>, arg=0xfffff800033c2100,
frame=0xfffffe010b5aaf40) at /usr/src/sys/kern/kern_fork.c:1160
#25 <signal handler called>
#26 0x000033af7af143ea in ?? ()