Recent problems with 6-STABLE...

[gnn at freebsd.org]

Hi,

I have two boxes running 6-STABLE, post 6.3 release, which have both
spontaneously rebooted, one under load and one not under load.  I have
attached dmesg and some traceback information, from the one trace that
looked interesting.  Any thoughts or hints would be apprecated.

To save you scanning all the dmesg first these are dual processor XEON
boxes, each processor has 4 cores.

Best,
George

Trace:

(kgdb) where
#0  doadump () at pcpu.h:172
#1  0x0000000000000004 in ?? ()
#2  0xffffffff802e38b7 in boot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:409
#3  0xffffffff802e3f51 in panic (fmt=0xffffff01b5318260 "X³-´\001ÿÿÿ°\232óz")
    at /usr/src/sys/kern/kern_shutdown.c:565
#4  0xffffffff8047263f in trap_fatal (frame=0xffffff01b5318260, 
    eva=18446742981515785048) at /usr/src/sys/amd64/amd64/trap.c:669
#5  0xffffffff80472b42 in trap (frame=
      {tf_rdi = 0, tf_rsi = -1092176739744, tf_rdx = 32491047111385957, tf_rcx = 56752, tf_r8 = -2140952488, tf_r9 = -2140952488, tf_rax = 1, tf_rbx = 4, tf_rbp = -1092193766568, tf_r10 = 34365284352, tf_r11 = -2140952488, tf_r12 = 0, tf_r13 = -1092193766568, tf_r14 = 0, tf_r15 = -1092176739744, tf_trapno = 9, tf_addr = 0, tf_flags = -1157743792, tf_err = 0, tf_rip = -2144770494, tf_cs = 8, tf_rflags = 66054, tf_rsp = -1157743920, tf_ss = 16})
    at /usr/src/sys/amd64/amd64/trap.c:470
#6  0xffffffff80459d3b in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:168
#7  0xffffffff80296642 in pfs_exit (arg=0x0, p=0xffffff01b42db358)
    at /usr/src/sys/fs/pseudofs/pseudofs_vncache.c:288
#8  0xffffffff802c355f in exit1 (td=0xffffff01b5318260, rv=0)
    at /usr/src/sys/kern/kern_exit.c:230
#9  0xffffffff802c48ee in sys_exit (td=0x0, uap=0xffffff01b5318260)
    at /usr/src/sys/kern/kern_exit.c:99
#10 0xffffffff80473531 in syscall (frame=
      {tf_rdi = 0, tf_rsi = 0, tf_rdx = 4285347, tf_rcx = 34365284352, tf_r8 = 0, tf_r9 = 0, tf_rax = 1, tf_rbx = 140737488342848, tf_rbp = 0, tf_r10 = 0, tf_r11 = 0, tf_r12 = 140737488349113, tf_r13 = 0, tf_r14 = 0, tf_r15 = 140737488343208, tf_trapno = 12, tf_addr = 5368856, tf_flags = 12, tf_err = 2, tf_rip = 34369792716, tf_cs = 43, tf_rflags = 518, tf_rsp = 140737488342376, tf_ss = 35})
    at /usr/src/sys/amd64/amd64/trap.c:807
#11 0xffffffff80459f38 in Xfast_syscall ()
    at /usr/src/sys/amd64/amd64/exception.S:287
#12 0x0000000800996acc in ?? ()
Previous frame inner to this frame (corrupt stack?)


Trap/Panic:

Fatal trap 9: general protection fault while in kernel mode
cpuid = 3; apic id = 03
instruction pointer     = 0x8:0xffffffff80296642
stack pointer           = 0x10:0xffffffffbafe3ac0
frame pointer           = 0x10:0xffffff01b42db358
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 56752 (sh)
trap number             = 9
panic: general protection fault
cpuid = 3
Uptime: 1d13h26m9s
Dumping 8190 MB (3 chunks)


Dmesg:

Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.3-STABLE #1: Wed Jan 23 19:01:23 EST 2008
    root at ntradee56:/usr/obj/usr/src/sys/LOCAL
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU           X5365  @ 3.00GHz (3000.12-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x6fb  Stepping = 11
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4e3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  Cores per package: 4
real memory  = 9395240960 (8960 MB)
avail memory = 8293945344 (7909 MB)
ACPI APIC Table: <PTLTD          APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
acpi0: <PTLTD   RSDT> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
cpu1: <ACPI CPU> on acpi0
acpi_throttle1: <ACPI CPU Throttling> on cpu1
acpi_throttle1: failed to attach P_CNT
device_attach: acpi_throttle1 attach returned 6
cpu2: <ACPI CPU> on acpi0
acpi_throttle2: <ACPI CPU Throttling> on cpu2
acpi_throttle2: failed to attach P_CNT
device_attach: acpi_throttle2 attach returned 6
cpu3: <ACPI CPU> on acpi0
acpi_throttle3: <ACPI CPU Throttling> on cpu3
acpi_throttle3: failed to attach P_CNT
device_attach: acpi_throttle3 attach returned 6
cpu4: <ACPI CPU> on acpi0
acpi_throttle4: <ACPI CPU Throttling> on cpu4
acpi_throttle4: failed to attach P_CNT
device_attach: acpi_throttle4 attach returned 6
cpu5: <ACPI CPU> on acpi0
acpi_throttle5: <ACPI CPU Throttling> on cpu5
acpi_throttle5: failed to attach P_CNT
device_attach: acpi_throttle5 attach returned 6
cpu6: <ACPI CPU> on acpi0
acpi_throttle6: <ACPI CPU Throttling> on cpu6
acpi_throttle6: failed to attach P_CNT
device_attach: acpi_throttle6 attach returned 6
cpu7: <ACPI CPU> on acpi0
acpi_throttle7: <ACPI CPU Throttling> on cpu7
acpi_throttle7: failed to attach P_CNT
device_attach: acpi_throttle7 attach returned 6
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
pci4: <ACPI PCI bus> on pcib4
em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port 0x2000-0x201f mem 0xd8020000-0xd803ffff,0xd8000000-0xd801ffff irq 18 at device 0.0 on pci4
em0: Using MSI interrupt
em0: Ethernet address: 00:30:48:62:97:1e
em1: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port 0x2020-0x203f mem 0xd8060000-0xd807ffff,0xd8040000-0xd805ffff irq 19 at device 0.1 on pci4
em1: Using MSI interrupt
em1: Ethernet address: 00:30:48:62:97:1f
pcib5: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci5: <ACPI PCI bus> on pcib5
ahd0: <Adaptec AIC7902 Ultra320 SCSI adapter> port 0x3400-0x34ff,0x3000-0x30ff mem 0xd8100000-0xd8101fff irq 30 at device 3.0 on pci5
ahd0: [GIANT-LOCKED]
aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs
ahd1: <Adaptec AIC7902 Ultra320 SCSI adapter> port 0x3c00-0x3cff,0x3800-0x38ff mem 0xd8102000-0xd8103fff irq 31 at device 3.1 on pci5
ahd1: [GIANT-LOCKED]
aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs
pcib6: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci7: <ACPI PCI bus> on pcib7
pci0: <base peripheral> at device 8.0 (no driver attached)
uhci0: <UHCI (generic) USB controller> port 0x1800-0x181f irq 17 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <UHCI (generic) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <UHCI (generic) USB controller> port 0x1820-0x183f irq 19 at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
usb1: <UHCI (generic) USB controller> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <UHCI (generic) USB controller> port 0x1840-0x185f irq 18 at device 29.2 on pci0
uhci2: [GIANT-LOCKED]
usb2: <UHCI (generic) USB controller> on uhci2
usb2: USB revision 1.0
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhci3: <UHCI (generic) USB controller> port 0x1860-0x187f irq 16 at device 29.3 on pci0
uhci3: [GIANT-LOCKED]
usb3: <UHCI (generic) USB controller> on uhci3
usb3: USB revision 1.0
uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xd8600000-0xd86003ff irq 17 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <EHCI (generic) USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub4: 8 ports with 8 removable, self powered
pcib8: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci8: <ACPI PCI bus> on pcib8
pci8: <display, VGA> at device 1.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1880-0x188f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
atapci1: <Intel AHCI controller> port 0x18c0-0x18c7,0x1894-0x1897,0x1898-0x189f,0x1890-0x1893,0x18a0-0x18bf mem 0xd8600400-0xd86007ff irq 19 at device 31.2 on pci0
atapci1: AHCI Version 01.10 controller with 6 ports detected
ata2: <ATA channel 0> on atapci1
ata3: <ATA channel 1> on atapci1
ata4: <ATA channel 2> on atapci1
ata5: <ATA channel 3> on atapci1
ata6: <ATA channel 4> on atapci1
ata7: <ATA channel 5> on atapci1
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcafff,0xcb000-0xcbfff on isa0
ppc0: cannot reserve I/O port range
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
acd0: CDRW <TSSTcorpCDW/DVD SN-M242D/SB01> at ata0-slave UDMA33
Waiting 5 seconds for SCSI devices to settle
ses0 at ahd0 bus 0 target 6 lun 0
ses0: <SUPER GEM318 0> Fixed Processor SCSI-2 device 
ses0: 3.300MB/s transfers
ses0: SAF-TE Compliant Device
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #6 Launched!
da0 at ahd0 bus 0 target 0 lun 0
da0: <SEAGATE ST3146807LC 0007> Fixed Direct Access SCSI-3 device 
da0: 320.000MB/s transfers (160.000MHz, offset 63, 16bit), Tagged Queueing Enabled
da0: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C)
Trying to mount root from ufs:/dev/da0s1a