amd64/157802: kernel panic in dummynet

[Alexey]

>Number:         157802
>Category:       amd64
>Synopsis:       kernel panic in dummynet
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    freebsd-amd64
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Jun 12 20:20:02 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator:     Alexey
>Release:        8.2-RELEASE-p2
>Organization:
>Environment:
FreeBSD gate22.igf.local 8.2-RELEASE-p2 FreeBSD 8.2-RELEASE-p2 #0: Fri Jun 10 04:12:07 YEKST 2011     kovalgate22.igf.local:/usr/obj/usr/src/sys/gate.vserver.x64  amd64
>Description:
kernel panic in dummynet's 'prio' scheduler on 1'st egress packet

I'm using such dummynet configuration:

>How-To-Repeat:

>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted:
 >ipfw pipe show
 00001:   2.000 Mbit/s    0 ms burst 0.
 q131073  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
  sched 65537 type FIFO flags 0x0 0 buckets 0 active
 00002: 128.000 Kbit/s    0 ms burst 10485760.
 q131074  50 sl. 0 flows (1 buckets) sched 65538 weight 1 lmax 0 pri 0 droptail
  sched 65538 type FIFO flags 0x0 0 buckets 0 active
 
 >ipfw sched show
 00001:   2.000 Mbit/s    0 ms burst 0.
  sched 1 type PRIO flags 0x0 0 buckets 0 active
    Children flowsets: 1 2 3 4 5 6 7.
 00002: 128.000 Kbit/s    0 ms burst 10485760.
  sched 2 type WF2Q+ flags 0x0 0 buckets 0 active
    Children flowsets: 10.
 
 > ipfw queue show
 q00001  50 sl. 0 flows (1 buckets) sched 1 weight 1 lmax 0 pri 0.
 <------> GRED w_q 0.001999 min_th 30 max_th 80 max_p 0.089996
 q00002  50 sl. 0 flows (1 buckets) sched 1 weight 20 lmax 0 pri 0.
 <------> GRED w_q 0.001999 min_th 30 max_th 80 max_p 0.089996
 q00003  50 sl. 0 flows (1 buckets) sched 1 weight 30 lmax 0 pri 0.
 <------> GRED w_q 0.001999 min_th 30 max_th 80 max_p 0.089996
 q00004  50 sl. 0 flows (1 buckets) sched 1 weight 40 lmax 0 pri 0.
 <------> GRED w_q 0.001999 min_th 30 max_th 80 max_p 0.089996
 q00005  50 sl. 0 flows (1 buckets) sched 1 weight 50 lmax 0 pri 0.
 <------> GRED w_q 0.001999 min_th 30 max_th 80 max_p 0.089996
 q00006  50 sl. 0 flows (1 buckets) sched 1 weight 63 lmax 0 pri 0.
 <------> GRED w_q 0.001999 min_th 30 max_th 80 max_p 0.089996
 q00007  50 sl. 0 flows (1 buckets) sched 1 weight 63 lmax 0 pri 0.
 <------> GRED w_q 0.001999 min_th 30 max_th 80 max_p 0.089996
 q00010  50 sl. 0 flows (64 buckets) sched 2 weight 10 lmax 0 pri 0.
 <------> GRED w_q 0.001999 min_th 30 max_th 80 max_p 0.089996
     mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
 
 
 
 > ipfw list | grep queue
 51100 queue 10 { tcp or udp } from any to any src-port 1024-65535 xmit lan1
 51400 queue 7 tag 1 tcp from any to any tcpflags syn
 51401 queue 7 tag 1 tcp from any to any tcpflags ack iplen 1-100 not tagged 1
 51402 queue 7 tag 1 ip from any to any iplen 1-80 not tagged 1
 51403 queue 6 tag 1 tcp from any to any dst-port 22,53 not tagged 1
 51404 queue 6 tag 1 udp from any to any dst-port 53,123 not tagged 1
 51405 queue 6 tag 1 icmp from any to any not tagged 1
 51406 queue 5 tag 1 { tcp or udp } from any 33000-33999 to any not tagged 1
 51407 queue 4 tag 1 tcp from any to any dst-port 25,110,143,465,587,993 not tagged 1
 51408 queue 4 tag 1 tcp from any 25,465,993 to any not tagged 1
 51409 queue 3 tag 1 tcp from any 80 to any not tagged 1
 51410 queue 3 tag 1 tcp from any to any dst-port 80 not tagged 1
 51411 queue 2 tag 1 tcp from any 20-21 to any not tagged 1
 51412 queue 2 tag 1 tcp from any to any dst-port 20-21 not tagged 1
 51413 queue 1 tag 1 ip from any to any not tagged 1
 
 rules 51400-51413 are on wan1 interface
 
 ping any of external address (on wan1) causes crash.
 
  kgdb /usr/obj/usr/src/sys/gate.vserver.x64/kernel.debug vmcore.0
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "amd64-marcel-freebsd"...
 
 Unread portion of the kernel message buffer:
 
 
 Fatal trap 12: page fault while in kernel mode
 cpuid = 1; apic id = 01
 fault virtual address   = 0x60
 fault code              = supervisor read data, page not present
 instruction pointer     = 0x20:0xffffffff80a297a9
 stack pointer           = 0x28:0xffffff8075921450
 frame pointer           = 0x28:0xffffff8075921460
 code segment            = base 0x0, limit 0xfffff, type 0x1b
                         = DPL 0, pres 1, long 1, def32 0, gran 1
 processor eflags        = interrupt enabled, resume, IOPL = 0
 current process         = 1438 (ping)
 trap number             = 12
 panic: page fault
 cpuid = 1
 Uptime: 17m8s
 Physical memory: 1591 MB
 Dumping 90 MB: 75 59panic: bufwrite: buffer is not busy???
 cpuid = 1
  43 27 11
 
 Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols from /boot/kernel/geom_journal.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/geom_journal.ko
 Reading symbols from /boot/kernel/alias_ftp.ko...Reading symbols from /boot/kernel/alias_ftp.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/alias_ftp.ko
 Reading symbols from /boot/kernel/dummynet.ko...Reading symbols from /boot/kernel/dummynet.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/dummynet.ko
 #0  doadump () at pcpu.h:224
 224     pcpu.h: No such file or directory.
         in pcpu.h
 (kgdb) bt
 #0  doadump () at pcpu.h:224
 #1  0xffffffff8033124d in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:419
 #2  0xffffffff80331528 in panic (fmt=Variable "fmt" is not available.
 ) at /usr/src/sys/kern/kern_shutdown.c:592
 #3  0xffffffff805350e7 in trap_fatal (frame=0xffffff80759213a0, eva=96) at /usr/src/sys/amd64/amd64/trap.c:783
 #4  0xffffffff8053536e in trap_pfault (frame=0xffffff80759213a0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:699
 #5  0xffffffff80535804 in trap (frame=0xffffff80759213a0) at /usr/src/sys/amd64/amd64/trap.c:449
 #6  0xffffffff8051d894 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224
 #7  0xffffffff80a297a9 in prio_dequeue (_si=Variable "_si" is not available.
 ) at /usr/src/sys/modules/dummynet/../../netinet/ipfw/dn_sched_prio.c:121
 #8  0xffffffff80a26cee in serve_sched (q=0xffffff80759214a0, si=0xffffff000171d400, now=101965)
     at /usr/src/sys/modules/dummynet/../../netinet/ipfw/ip_dn_io.c:457
 #9  0xffffffff80a27200 in dummynet_io (m0=0xffffff8075921718, dir=0, fwa=Variable "fwa" is not available.
 )
     at /usr/src/sys/modules/dummynet/../../netinet/ipfw/ip_dn_io.c:759
 #10 0xffffffff804139e2 in ipfw_check_hook (arg=Variable "arg" is not available.
 ) at /usr/src/sys/netinet/ipfw/ip_fw_pfil.c:192
 #11 0xffffffff803f43bc in pfil_run_hooks (ph=Variable "ph" is not available.
 ) at /usr/src/sys/net/pfil.c:82
 #12 0xffffffff8041ccab in ip_output (m=0xffffff0001b24500, opt=Variable "opt" is not available.
 ) at /usr/src/sys/netinet/ip_output.c:511
 #13 0xffffffff8041d7be in rip_output (m=0xffffff0001b24500, so=Variable "so" is not available.
 ) at /usr/src/sys/netinet/raw_ip.c:507
 #14 0xffffffff8041d882 in rip_send (so=Variable "so" is not available.
 ) at /usr/src/sys/netinet/raw_ip.c:993
 #15 0xffffffff8039b659 in sosend_generic (so=0xffffff0001941d48, addr=0xffffff00017e0160, uio=0xffffff80759219e0, top=0xffffff0001b24500,
     control=0x0, flags=0, td=0xffffff0001913460) at /usr/src/sys/kern/uipc_socket.c:1260
 #16 0xffffffff8039540f in sosend (so=Variable "so" is not available.
 ) at /usr/src/sys/kern/uipc_socket.c:1304
 #17 0xffffffff8039d11e in kern_sendit (td=0xffffff0001913460, s=3, mp=0xffffff8075921ab0, flags=0, control=0x0, segflg=UIO_USERSPACE)
     at /usr/src/sys/kern/uipc_syscalls.c:788
 #18 0xffffffff8039d33a in sendit (td=0xffffff0001913460, s=3, mp=0xffffff8075921ab0, flags=0) at /usr/src/sys/kern/uipc_syscalls.c:724
 #19 0xffffffff8039d45b in sendto (td=Variable "td" is not available.
 ) at /usr/src/sys/kern/uipc_syscalls.c:840
 #20 0xffffffff803709ee in syscallenter (td=0xffffff0001913460, sa=0xffffff8075921bb0) at /usr/src/sys/kern/subr_trap.c:315
 #21 0xffffffff805353e1 in syscall (frame=0xffffff8075921c40) at /usr/src/sys/amd64/amd64/trap.c:888
 #22 0xffffffff8051db72 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:377
 #23 0x0000000800947a7c in ?? ()
 Previous frame inner to this frame (corrupt stack?)
 (kgdb) frame 7
 #7  0xffffffff80a297a9 in prio_dequeue (_si=Variable "_si" is not available.
 ) at /usr/src/sys/modules/dummynet/../../netinet/ipfw/dn_sched_prio.c:121
 121             q = si->q_array[prio];
 (kgdb)