amd64/131601: 7-STABLE panic in nat_finalise

Thu Feb 12 00:23:20 PST 2009

>Number:         131601
>Category:       amd64
>Synopsis:       7-STABLE panic in nat_finalise
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-amd64
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Feb 12 08:20:01 UTC 2009
>Closed-Date:
>Last-Modified:
>Originator:     Vladimir Kurtukov
>Release:        FreeBSD 7.1-STABLE amd64
>Organization:
KuznetskBusinessBank
>Environment:
System: FreeBSD kbb2.kbb.ru 7.1-STABLE FreeBSD 7.1-STABLE #0: Fri Jan 16 12:11:42 NKZ 2009 vk at kbb2.kbb.ru:/usr/src/sys/amd64/compile/KBB2 amd64

CPU: Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz (2405.46-MHz K8-class CPU)
usable memory = 4282646528 (4084 MB)
avail memory  = 4121227264 (3930 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs

aac0: <Adaptec SATA RAID 2420SA> mem 0xfe800000-0xfe9fffff,0xfeaff000-0xfeafffff irq 17 at device 1.

mskc0: <Marvell Yukon 88E8056 Gigabit Ethernet> port 0xd800-0xd8ff mem 0xfe6fc000-0xfe6fffff irq 18
msk0: <Marvell Technology Group Ltd. Yukon EC Ultra Id 0xb4 Rev 0x03> on mskc0

mskc1: <Marvell Yukon 88E8056 Gigabit Ethernet> port 0xc800-0xc8ff mem 0xfe5fc000-0xfe5fffff irq 19
msk1: <Marvell Technology Group Ltd. Yukon EC Ultra Id 0xb4 Rev 0x03> on mskc1

em0: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0xec00-0xec3f mem 0xfebe0000-0xfebfffff,0xfeb

This machine is used as a Internet gateway with FW (ipfw) and NAT (IPF's ipnat with 
LARGE_NAT defined, because there are 4000 NAT rules)

>Description:

Sometimes (1 crash per 2 weeks or even more) machine panics with:

Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 03
fault virtual address   = 0x4
fault code              = supervisor read data, page not present
instruction pointer     = 0x8:0xffffffffb2f3a316
stack pointer           = 0x10:0xffffffffb0a28220
frame pointer           = 0x10:0xffffffffb0a28270
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 37 (mskc1 taskq)
trap number             = 12
panic: page fault
cpuid = 3
Uptime: 20d13h12m43s
Physical memory: 4084 MB
Dumping 777 MB: 762 746 730 714 698 682 666 650 634 618 602 586 570 554 538 522 506 490 474 458 442
426 410 394 378 362 346 330 314 298 282 266 250 234 218 202 186 170 154 138 122 106 90 74 58 42 26 1
0

backtrace:
#0  doadump () at ../../../kern/kern_shutdown.c:244
244             dumptid = curthread->td_tid;
(kgdb) bt
#0  doadump () at ../../../kern/kern_shutdown.c:244
#1  0xffffffff803908be in boot (howto=260) at ../../../kern/kern_shutdown.c:418
#2  0xffffffff80390e0d in panic (fmt=Could not find the frame base for "panic".
) at ../../../kern/kern_shutdown.c:574
#3  0xffffffff806d8892 in trap_fatal (frame=0xffffffffb0a28170, eva=4)
    at ../../../amd64/amd64/trap.c:764
#4  0xffffffff806d8342 in trap_pfault (frame=0xffffffffb0a28170, usermode=0)
    at ../../../amd64/amd64/trap.c:680
#5  0xffffffff806d7d20 in trap (frame=0xffffffffb0a28170) at ../../../amd64/amd64/trap.c:449
#6  0xffffffff806b73ee in calltrap () at ../../../amd64/amd64/exception.S:209
#7  0xffffffffb2f3a316 in nat_finalise (fin=0xffffffffb0a28440, nat=0xffffff002502da00,
    ni=0xffffffffb0a282b0, tcp=0x0, natsave=0x0, direction=0)
    at /usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/ip_nat.c:2577
#8  0xffffffffb2f3a11d in nat_new () from /boot/kernel/ipl.ko
#9  0xffffffffb2f3d53a in fr_checknatin (fin=0xffffffffb0a28440, passp=0xffffffffb0a2843c)
    at /usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/ip_nat.c:4122
#10 0xffffffffb2f5c822 in fr_check (ip=0xffffff004f583810, hlen=20, ifp=0xffffff0003370800,
    out=0, mp=0xffffffffb0a285c8)
    at /usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/fil.c:2572
#11 0xffffffffb2f56ec8 in fr_check_wrapper (arg=0x0, mp=0xffffffffb0a285c8,
    ifp=0xffffff0003370800, dir=1)
    at /usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/ip_fil_freebsd.c:178
#12 0xffffffff8047ae88 in pfil_run_hooks (ph=0xffffffff80928320, mp=0xffffffffb0a28608,
    ifp=0xffffff0003370800, dir=1, inp=0x0) at ../../../net/pfil.c:78
#13 0xffffffff804b0cae in ip_input (m=0xffffff00105be300) at ../../../netinet/ip_input.c:417
#14 0xffffffff8047891c in netisr_dispatch (num=2, m=0xffffff00105be300)
    at ../../../net/netisr.c:185
#15 0xffffffff8046d0b7 in ether_demux (ifp=0xffffff0003370800, m=0xffffff00105be300)
    at ../../../net/if_ethersubr.c:834
#16 0xffffffffb30f50a6 in ng_ether_rcv_upper (node=0xffffff0009f8b100, m=0xffffff00105be300)
    at /usr/src/sys/modules/netgraph/ether/../../../netgraph/ng_ether.c:664
#17 0xffffffffb30f4e02 in ng_ether_rcvdata (hook=0xffffff00097f3e00, item=0xffffff008569a690)
    at /usr/src/sys/modules/netgraph/ether/../../../netgraph/ng_ether.c:586
#18 0xffffffffb30ea8be in ng_apply_item (node=0xffffff0009f8b100, item=0xffffff008569a690, rw=0)
    at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2331
#19 0xffffffffb30ea446 in ng_snd_item (item=0xffffff008569a690, flags=0)
    at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2249
#20 0xffffffffb30f86ef in ng_tee_rcvdata (hook=0xffffff00097f4080, item=0xffffff008569a690)
    at /usr/src/sys/modules/netgraph/tee/../../../netgraph/ng_tee.c:326
#21 0xffffffffb30ea8be in ng_apply_item (node=0xffffff003b38f000, item=0xffffff008569a690, rw=0)
    at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2331
#22 0xffffffffb30ea446 in ng_snd_item (item=0xffffff008569a690, flags=0)
    at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2249
#23 0xffffffffb30f4087 in ng_ether_input () from /boot/kernel/ng_ether.ko
#24 0xffffffff8046cd7a in ether_input (ifp=0xffffff0003370800, m=0xffffff00105be300)
    at ../../../net/if_ethersubr.c:643
#25 0xffffffff802849af in msk_rxeof (sc_if=0xffffffff80c67000, status=3932416, len=60)
    at ../../../dev/msk/if_msk.c:2966
#26 0xffffffff80285934 in msk_handle_events (sc=0xffffff0003348600)
    at ../../../dev/msk/if_msk.c:3341
#27 0xffffffff802862e5 in msk_int_task (arg=0xffffff0003348600, pending=1)
    at ../../../dev/msk/if_msk.c:3523
#28 0xffffffff803daa33 in taskqueue_run (queue=0xffffff0005c09e00)
    at ../../../kern/subr_taskqueue.c:282
#29 0xffffffff803db0e1 in taskqueue_thread_loop (arg=0xffffff00033486d8)
    at ../../../kern/subr_taskqueue.c:401
#30 0xffffffff80360f72 in fork_exit (callout=0xffffffff803db0b0 <taskqueue_thread_loop>,
    arg=0xffffff00033486d8, frame=0xffffffffb0a28c80) at ../../../kern/kern_fork.c:804
#31 0xffffffff806b77be in fork_trampoline () at ../../../amd64/amd64/exception.S:455
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000001 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0x0000000000000000 in ?? ()
#39 0x0000000000000000 in ?? ()
#40 0x0000000000000000 in ?? ()
#41 0x0000000000000000 in ?? ()
#42 0x0000000000000000 in ?? ()
#43 0x0000000000000000 in ?? ()
#44 0x0000000000000000 in ?? ()
#45 0x0000000000000000 in ?? ()
#46 0x0000000000000000 in ?? ()
#47 0x0000000000000000 in ?? ()
#48 0x0000000000000000 in ?? ()
#49 0x0000000000000000 in ?? ()
#50 0x0000000000000000 in ?? ()
#51 0x0000000000000000 in ?? ()
#52 0x0000000000000000 in ?? ()
#53 0x0000000000000000 in ?? ()
#54 0x0000000000000000 in ?? ()
#55 0x0000000000000000 in ?? ()
#56 0x0000000000bcf000 in ?? ()
#57 0x0000000000000000 in ?? ()
#58 0x0000000000000000 in ?? ()
#59 0x0000000000000000 in ?? ()
#60 0xffffffff803db0b0 in taskqueue_start_threads () at ../../../kern/subr_taskqueue.c:390
(kgdb) list *0xffffffffb2f3a316
0xffffffffb2f3a316 is in nat_finalise (/usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/
ip_nat.c:2577).
2572            nat->nat_ifps[1] = np->in_ifps[1];
2573            nat->nat_ptr = np;
2574            nat->nat_p = fin->fin_p;
2575            nat->nat_mssclamp = np->in_mssclamp;
2576            if (nat->nat_p == IPPROTO_TCP)
2577                    nat->nat_seqnext[0] = ntohl(tcp->th_seq);
2578
2579            if ((np->in_apr != NULL) && ((ni->nai_flags & NAT_SLAVE) == 0))
2580                    if (appr_new(fin, nat) == -1)
2581                            return -1;

Coredump is available by request

>How-To-Repeat:

        Floating bug, can't repeat

>Fix:

	Unknown

>Release-Note:
>Audit-Trail:
>Unformatted: