kern/174822: Page fault in tcp_discardcb under high traffic

[Adam Twardowski]

>Number:         174822
>Category:       kern
>Synopsis:       Page fault in tcp_discardcb under high traffic
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Dec 30 08:40:00 UTC 2012
>Closed-Date:
>Last-Modified:
>Originator:     Adam Twardowski
>Release:        9.1-RELEASE
>Organization:
>Environment:
FreeBSD ID11278.choopa.com 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec  4 09:23:10 UTC 2012     root at farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
Random kernel page fault's under high network traffic traffic.  I'm running haproxy 1.4.22 to load balance traffic to a cluster of web servers.  I'm seeing the same error on multiple machines and I already ran memtest to verify that there wasn't any issues with the memory.

[root at ID11278 /var/crash]# kgdb /boot/kernel/kernel.symbols vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x28
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80a7a951
stack pointer           = 0x28:0xffffff88c1836780
frame pointer           = 0x28:0xffffff88c1836800
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 22494 (haproxy)
trap number             = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80bd8240 at trap_fatal+0x290
#3 0xffffffff80bd857d at trap_pfault+0x1ed
#4 0xffffffff80bd8b9e at trap+0x3ce
#5 0xffffffff80bc315f at calltrap+0x8
#6 0xffffffff80a80ffa at tcp_usr_detach+0x11a
#7 0xffffffff80952f84 at sofree+0x134
#8 0xffffffff80953a81 at soclose+0x3c1
#9 0xffffffff808a9653 at _fdrop+0x23
#10 0xffffffff808aa952 at closef+0x52
#11 0xffffffff808ab192 at kern_close+0x172
#12 0xffffffff80bd7ae6 at amd64_syscall+0x546
#13 0xffffffff80bc3447 at Xfast_syscall+0xf7
Uptime: 3d2h59m9s
Dumping 1980 out of 32726 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/geom_mirror.ko
Reading symbols from /boot/modules/dscp_rewrite.ko...done.
Loaded symbols for /boot/modules/dscp_rewrite.ko
Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from /boot/kernel/ipfw.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipfw.ko
#0  doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
224     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
#1  0xffffffff808ea3a1 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:448
#2  0xffffffff808ea897 in panic (fmt=0x1 <Address 0x1 out of bounds>) at /usr/src/sys/kern/kern_shutdown.c:636
#3  0xffffffff80bd8240 in trap_fatal (frame=0xc, eva=Variable "eva" is not available.
) at /usr/src/sys/amd64/amd64/trap.c:857
#4  0xffffffff80bd857d in trap_pfault (frame=0xffffff88c18366d0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:773
#5  0xffffffff80bd8b9e in trap (frame=0xffffff88c18366d0) at /usr/src/sys/amd64/amd64/trap.c:456
#6  0xffffffff80bc315f in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228
#7  0xffffffff80a7a951 in tcp_discardcb (tp=0x0) at /usr/src/sys/netinet/tcp_subr.c:839
#8  0xffffffff80a80ffa in tcp_usr_detach (so=Variable "so" is not available.
) at /usr/src/sys/netinet/tcp_usrreq.c:205
#9  0xffffffff80952f84 in sofree (so=0xfffffe05e6dbe7f8) at /usr/src/sys/kern/uipc_socket.c:647
#10 0xffffffff80953a81 in soclose (so=0xfffffe05e6dbe7f8) at /usr/src/sys/kern/uipc_socket.c:741
#11 0xffffffff808a9653 in _fdrop (fp=0xfffffe01d5f18280, td=Variable "td" is not available.
) at file.h:316
#12 0xffffffff808aa952 in closef (fp=0xfffffe01d5f18280, td=0xfffffe000fcd28e0) at /usr/src/sys/kern/kern_descrip.c:2250
#13 0xffffffff808ab192 in kern_close (td=0xfffffe000fcd28e0, fd=Variable "fd" is not available.
) at /usr/src/sys/kern/kern_descrip.c:1236
#14 0xffffffff80bd7ae6 in amd64_syscall (td=0xfffffe000fcd28e0, traced=0) at subr_syscall.c:135
#15 0xffffffff80bc3447 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:387
#16 0x0000000800ff1cfc in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)

>How-To-Repeat:
Not easily repeatable, but it seems to happen to me every couple of days at least.
>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: