Re: FreeBSD panics possibly caused by nfs clients

On Fri, Feb 9, 2024 at 10:23 AM Matthew L. Dailey
<Matthew.L.Dailey@dartmouth.edu> wrote:
>
> I had my first kernel panic with a KASAN kernel after only 01:27. This
> first panic was a "double fault," which isn't anything we've seen
> previously - usually we've seen trap 9 or trap 12, but sometimes others.
> Based on the backtrace, it definitely looks like KASAN caught something,
> but I don't have the expertise to know if this points to anything
> specific. From the backtrace, it looks like this might have originated
> in ipfw code.
I do not know anything about the KASAN kernel, but I do see it increases
the kernel stack to 6 pages and when I see a double fault, it is usually a
kernel stack being blown.

You might check what it is:
# sysctl -a | fgrep kstack_pages
If it is already 6, you could try bumping it to 8.
Just add this line to /boot/loader.conf
kern.kstack_pages=8
and reboot.

rick

>
> Please let me know what other info I can provide or what I can do to dig
> deeper.
>
> Thanks!!
>
> Panic message:
> [5674] Fatal double fault
> [5674] rip 0xffffffff812f6e32 rsp 0xfffffe014677afe0 rbp 0xfffffe014677b430
> [5674] rax 0x1fffffc028cef620 rdx 0xf2f2f2f8f2f2f2f2 rbx 0x1
> [5674] rcx 0xdffff7c000000000 rsi 0xfffffe004086a4a0 rdi 0xf8f8f8f8f2f2f2f8
> [5674] r8 0xf8f8f8f8f8f8f8f8 r9 0x162a r10 0x835003002d3a64e1
> [5674] r11 0 r12 0xfffff78028cef620 r13 0xfffffe004086a440
> [5674] r14 0xfffffe01488c0560 r15 0x26f40 rflags 0x10006
> [5674] cs 0x20 ss 0x28 ds 0x3b es 0x3b fs 0x13 gs 0x1b
> [5674] fsbase 0x95d1d81a130 gsbase 0xffffffff84a14000 kgsbase 0
> [5674] cpuid = 4; apic id = 08
> [5674] panic: double fault
> [5674] cpuid = 4
> [5674] time = 1707498420
> [5674] KDB: stack backtrace:
> [5674] Uptime: 1h34m34s
>
> Backtrace:
> #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
> #1  doadump (textdump=<optimized out>) at
> /usr/src/sys/kern/kern_shutdown.c:405
> #2  0xffffffff8128b7dc in kern_reboot (howto=howto@entry=260)
>      at /usr/src/sys/kern/kern_shutdown.c:526
> #3  0xffffffff8128c000 in vpanic (
>      fmt=fmt@entry=0xffffffff82589a00 <str> "double fault",
>      ap=ap@entry=0xfffffe0040866de0) at
> /usr/src/sys/kern/kern_shutdown.c:970
> #4  0xffffffff8128bd75 in panic (fmt=0xffffffff82589a00 <str> "double
> fault")
>      at /usr/src/sys/kern/kern_shutdown.c:894
> #5  0xffffffff81c4b335 in dblfault_handler (frame=<optimized out>)
>      at /usr/src/sys/amd64/amd64/trap.c:1012
> #6  <signal handler called>
> #7  0xffffffff812f6e32 in sched_clock (td=td@entry=0xfffffe01488c0560,
>      cnt=cnt@entry=1) at /usr/src/sys/kern/sched_ule.c:2601
> #8  0xffffffff8119e2a7 in statclock (cnt=cnt@entry=1,
>      usermode=usermode@entry=0) at /usr/src/sys/kern/kern_clock.c:760
> #9  0xffffffff8119fb67 in handleevents (now=now@entry=24371855699832,
>      fake=fake@entry=0) at /usr/src/sys/kern/kern_clocksource.c:195
> #10 0xffffffff811a10cc in timercb (et=<optimized out>, arg=<optimized out>)
>      at /usr/src/sys/kern/kern_clocksource.c:353
> #11 0xffffffff81dcd280 in lapic_handle_timer (frame=0xfffffe014677b750)
>      at /usr/src/sys/x86/x86/local_apic.c:1343
> #12 <signal handler called>
> #13 __asan_load8_noabort (addr=18446741880219689232)
>      at /usr/src/sys/kern/subr_asan.c:1113
> #14 0xffffffff851488b8 in ?? () from /boot/thayer/ipfw.ko
> #15 0xfffffe0100000000 in ?? ()
> #16 0xffffffff8134dcd5 in pcpu_find (cpuid=1238425856)
>      at /usr/src/sys/kern/subr_pcpu.c:286
> #17 0xffffffff85151f6f in ?? () from /boot/thayer/ipfw.ko
> #18 0x0000000000000000 in ?? ()