panic in propagate_priority w/ postgresql under heavy load
John Baldwin
jhb at FreeBSD.org
Thu Sep 1 10:49:59 PDT 2005
On Thursday 01 September 2005 01:02 pm, Koen Martens wrote:
> Hi Hackers,
>
> I've had a little chat with neologism on ircnet/#freebsd about this
> already, and done as he suggested: compile a debug kernel to obtain
> a stack trace.
Can you reproduce it with a kernel that has INVARIANTS and INVARIANT_SUPPORT
on? I see that you had WITNESS on, can you check to see if there were any
witness messages about sleepign with non-sleepable locks held before the
crash?
> Anyway, what is happening is that there is a crash when running
> postgresql 8.0.3 with a very large database and doing heavy queries.
>
> Kernel is 5.4-RELEASE-p6 (5.4-RELENG i checked out tuesday a week
> ago). Kernel conf is down below.
>
> Here is the message i get on the console:
>
> ====================
> kernel trap 12 with interrupts disabled
>
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 1; apic id = 06
> fault virtual address = 0x24
> fault code = supervisor read, page not present
> instruction pointer = 0x8:0xc050cff7
> stack pointer = 0x10:0xe99c2b0c
> frame pointer = 0x10:0xe99c2b20
> code segment = base 0x0, limit 0xfffff, type 0x1b
> = DPL 0, pres 1, def32 1, gran 1
> processor eflags = resume, IOPL = 0
> current process = 4571 (postgres)
> ====================
>
> It has been a postgres process in all of the observed cases.
>
> I've looked it up with gdk on my kernel.debug, here's what i get:
>
> =====================
> yin# gdb kernel.debug
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and
> you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for
> details.
> This GDB was configured as "i386-marcel-freebsd"...
> (gdb) l *propagate_priority+0x7f
> 0xc050cff7 is in propagate_priority
> (/usr/src/sys/kern/subr_turnstile.c:245).
> 240 /*
> 241 * Pick up the lock that td is blocked on.
> 242 */
> 243 ts = td->td_blocked;
> 244 MPASS(ts != NULL);
> 245 tc = TC_LOOKUP(ts->ts_lockobj);
> 246 mtx_lock_spin(&tc->tc_lock);
> 247
> 248 /*
> 249 * This thread may not be blocked on this
> turnstile anymore
> (gdb)
> =====================
>
>
> So the next thing you'll ask for is a stack trace, but i haven't
> been able to obtain one. According to the freebsd handbook
> (http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerne
>ldebug-gdb.html) there should be a core dump in /var/crash, but there is
> none and the handbook chapter seems outdated anyway judging by it
> mentioning kgdb...
>
> Anyway, it seems the dump should've gone to the swap partition, but
> i'm into multi-user mode again so i guess i'll have to wait for
> another panic to obtain it?
>
> I'm not very knowledgeable about the freebsd kernel internals (yet),
> so i'm not sure what could be wrong here.. I hope some of you can
> provide some insight, and ideally a fix :)
>
> ======================[ kernel config:
>
> #
> # GENERIC -- Generic kernel configuration file for FreeBSD/i386
> #
> # For more information on this file, please read the handbook section on
> # Kernel Configuration Files:
> #
> #
> http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-conf
>ig.html #
> # The handbook is also available locally in /usr/share/doc/handbook
> # if you've installed the doc distribution, otherwise always see the
> # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the
> # latest information.
> #
> # An exhaustive list of options and more detailed explanations of the
> # device lines is also present in the ../../conf/NOTES and NOTES files.
> # If you are in doubt as to the purpose or necessity of a line,
> check first
> # in NOTES.
> #
> # $FreeBSD: src/sys/i386/conf/GENERIC,v 1.413.2.13 2005/04/02
> 16:37:58 scottl Exp $
>
> machine i386
> cpu I486_CPU
> cpu I586_CPU
> cpu I686_CPU
> ident YIN-YANG
>
> # debug
> options WITNESS
> options KDB
> options DDB
> #
> makeoptions DEBUG=-g
> # debug
>
>
> # To statically compile in device wiring instead of /boot/device.hints
> #hints "GENERIC.hints" # Default places to look for devices.
>
> options SCHED_4BSD # 4BSD scheduler
> options INET # InterNETworking
> options INET6 # IPv6 communications protocols
> options FFS # Berkeley Fast Filesystem
> options SOFTUPDATES # Enable FFS soft updates support
> options UFS_ACL # Support for access control lists
> options UFS_DIRHASH # Improve performance on big directories
> options MD_ROOT # MD is a potential root device
> options NFSCLIENT # Network Filesystem Client
> options NFSSERVER # Network Filesystem Server
> options NFS_ROOT # NFS usable as /, requires NFSCLIENT
> options MSDOSFS # MSDOS Filesystem
> options CD9660 # ISO 9660 Filesystem
> options PROCFS # Process filesystem (requires PSEUDOFS)
> options PSEUDOFS # Pseudo-filesystem framework
> options GEOM_GPT # GUID Partition Tables.
> options COMPAT_43 # Compatible with BSD 4.3 [KEEP THIS!]
> options COMPAT_FREEBSD4 # Compatible with FreeBSD4
> options SCSI_DELAY=15000 # Delay (in ms) before probing SCSI
> options KTRACE # ktrace(1) support
> options SYSVSHM # SYSV-style shared memory
> options SYSVMSG # SYSV-style message queues
> options SYSVSEM # SYSV-style semaphores
> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time
> extensions
> options KBD_INSTALL_CDEV # install a CDEV entry in /dev
> options AHC_REG_PRETTY_PRINT # Print register bitfields in debug
> # output. Adds ~128k to driver.
> options AHD_REG_PRETTY_PRINT # Print register bitfields in debug
> # output. Adds ~215k to driver.
> options ADAPTIVE_GIANT # Giant mutex is adaptive.
>
>
> options IPFILTER
> options IPFILTER_LOG
> options IPFILTER_DEFAULT_BLOCK
>
> options IPFILTER
> options IPFILTER_LOG
>
> options IPFIREWALL #firewall
> options IPFIREWALL_VERBOSE #enable logging to syslogd(8)
> options IPFIREWALL_FORWARD #enable transparent proxy
> support
> options IPFIREWALL_VERBOSE_LIMIT=10 #limit verbosity
> options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by
> default
> options IPV6FIREWALL #firewall for IPv6
> options IPV6FIREWALL_VERBOSE
> options IPV6FIREWALL_VERBOSE_LIMIT=10
> options IPV6FIREWALL_DEFAULT_TO_ACCEPT
>
> options SHMMAXPGS=262144
> options SHMSEG=512
> options SHMMNI=512
> options SEMMNI=512
> options SEMMNS=1024
> options SEMMNU=512
> options SEMMAP=512
> options NMBCLUSTERS=32768
>
> device apic # I/O APIC
>
> options SMP # Symmetric MultiProcessor
> Kernel
>
> # Bus support. Do not remove isa, even if you have no isa slots
> device isa
> device eisa
> device pci
>
> # Floppy drives
> device fdc
>
> # ATA and ATAPI devices
> device ata
> device atadisk # ATA disk drives
> device ataraid # ATA RAID drives
> device atapicd # ATAPI CDROM drives
> #device atapifd # ATAPI floppy drives
> #device atapist # ATAPI tape drives
> options ATA_STATIC_ID # Static device numbering
>
> # SCSI Controllers
> #device ahb # EISA AHA1742 family
> #device ahc # AHA2940 and onboard AIC7xxx devices
> #device ahd # AHA39320/29320 and onboard AIC79xx devices
> #device amd # AMD 53C974 (Tekram DC-390(T))
> #device isp # Qlogic family
> #device mpt # LSI-Logic MPT-Fusion
> ##device ncr # NCR/Symbios Logic
> #device sym # NCR/Symbios Logic (newer chipsets + those of `ncr')
> #device trm # Tekram DC395U/UW/F DC315U adapters
>
> #device adv # Advansys SCSI adapters
> #device adw # Advansys wide SCSI adapters
> #device aha # Adaptec 154x SCSI adapters
> #device aic # Adaptec 15[012]x SCSI adapters, AIC-6[23]60.
> #device bt # Buslogic/Mylex MultiMaster SCSI adapters
>
> #device ncv # NCR 53C500
> #device nsp # Workbit Ninja SCSI-3
> #device stg # TMC 18C30/18C50
>
> # SCSI peripherals
> device scbus # SCSI bus (required for SCSI)
> device ch # SCSI media changers
> device da # Direct Access (disks)
> device sa # Sequential Access (tape etc)
> device cd # CD
> device pass # Passthrough device (direct SCSI access)
> device ses # SCSI Environmental Services (and SAF-TE)
>
> # RAID controllers interfaced to the SCSI subsystem
> #device amr # AMI MegaRAID
> #device arcmsr # Areca SATA II RAID
> #device asr # DPT SmartRAID V, VI and Adaptec SCSI RAID
> #device ciss # Compaq Smart RAID 5*
> #device dpt # DPT Smartcache III, IV - See NOTES for options
> #device hptmv # Highpoint RocketRAID 182x
> #device iir # Intel Integrated RAID
> #device ips # IBM (Adaptec) ServeRAID
> #device mly # Mylex AcceleRAID/eXtremeRAID
> device twa # 3ware 9000 series PATA/SATA RAID
>
> # RAID controllers
> #device aac # Adaptec FSA RAID
> #device aacp # SCSI passthrough for aac (requires CAM)
> #device ida # Compaq Smart RAID
> #device mlx # Mylex DAC960 family
> #device pst # Promise Supertrak SX6000
> device twe # 3ware ATA RAID
>
> # atkbdc0 controls both the keyboard and the PS/2 mouse
> device atkbdc # AT keyboard controller
> device atkbd # AT keyboard
> device psm # PS/2 mouse
>
> device vga # VGA video card driver
>
> #device splash # Splash screen and screen saver support
>
> # syscons is the default console driver, resembling an SCO console
> device sc
>
> # Enable this for the pcvt (VT220 compatible) console driver
> #device vt
> #options XSERVER # support for X server on a vt console
> #options FAT_CURSOR # start with block cursor
>
> device agp # support several AGP chipsets
>
> # Floating point support - do not disable.
> device npx
>
> # Power management support (see NOTES for more options)
> #device apm
> # Add suspend/resume support for the i8254.
> device pmtimer
>
> # PCCARD (PCMCIA) support
> # PCMCIA and cardbus bridge support
> #device cbb # cardbus (yenta) bridge
> #device pccard # PC Card (16-bit) bus
> #device cardbus # CardBus (32-bit) bus
>
> # Serial (COM) ports
> device sio # 8250, 16[45]50 based serial ports
>
> # Parallel port
> device ppc
> device ppbus # Parallel port bus (required)
> device lpt # Printer
> device plip # TCP/IP over parallel
> device ppi # Parallel port interface device
> #device vpo # Requires scbus and da
>
> # If you've got a "dumb" serial or parallel PCI card that is
> # supported by the puc(4) glue driver, uncomment the following
> # line to enable it (connects to the sio and/or ppc drivers):
> #device puc
>
> # PCI Ethernet NICs.
> #device de # DEC/Intel DC21x4x (``Tulip'')
> device em # Intel PRO/1000 adapter Gigabit Ethernet Card
> #device ixgb # Intel PRO/10GbE Ethernet Card
> #device txp # 3Com 3cR990 (``Typhoon'')
> #device vx # 3Com 3c590, 3c595 (``Vortex'')
>
> # PCI Ethernet NICs that use the common MII bus controller code.
> # NOTE: Be sure to keep the 'device miibus' line in order to use
> these NICs!
> device miibus # MII bus support
> #device bfe # Broadcom BCM440x 10/100 Ethernet
> #device bge # Broadcom BCM570xx Gigabit Ethernet
> #device dc # DEC/Intel 21143 and various workalikes
> device fxp # Intel EtherExpress PRO/100B (82557, 82558)
> #device lge # Level 1 LXT1001 gigabit ethernet
> #device nge # NatSemi DP83820 gigabit ethernet
> #evice pcn # AMD Am79C97x PCI 10/100 (precedence over 'lnc')
> #device re # RealTek 8139C+/8169/8169S/8110S
> #device rl # RealTek 8129/8139
> #device sf # Adaptec AIC-6915 (``Starfire'')
> #device sis # Silicon Integrated Systems SiS 900/SiS 7016
> #device sk # SysKonnect SK-984x & SK-982x gigabit Ethernet
> #device ste # Sundance ST201 (D-Link DFE-550TX)
> #device ti # Alteon Networks Tigon I/II gigabit Ethernet
> #device tl # Texas Instruments ThunderLAN
> #device tx # SMC EtherPower II (83c170 ``EPIC'')
> #device vge # VIA VT612x gigabit ethernet
> #device vr # VIA Rhine, Rhine II
> #device wb # Winbond W89C840F
> #device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'')
>
> # ISA Ethernet NICs. pccard NICs included.
> #device cs # Crystal Semiconductor CS89x0 NIC
> # 'device ed' requires 'device miibus'
> #device ed # NE[12]000, SMC Ultra, 3c503, DS8390 cards
> #device ex # Intel EtherExpress Pro/10 and Pro/10+
> #device ep # Etherlink III based cards
> #device fe # Fujitsu MB8696x based cards
> #device ie # EtherExpress 8/16, 3C507, StarLAN 10 etc.
> #device lnc # NE2100, NE32-VL Lance Ethernet cards
> #device sn # SMC's 9000 series of Ethernet chips
> #device xe # Xircom pccard Ethernet
>
> # ISA devices that use the old ISA shims
> #device le
>
> # Wireless NIC cards
> #device wlan # 802.11 support
> #device an # Aironet 4500/4800 802.11 wireless NICs.
> #device awi # BayStack 660 and others
> #device wi # WaveLAN/Intersil/Symbol 802.11 wireless NICs.
> #device wl # Older non 802.11 Wavelan wireless NIC.
>
> # Pseudo devices.
> device loop # Network loopback
> device mem # Memory and kernel memory devices
> device io # I/O device
> device random # Entropy device
> device ether # Ethernet support
> device sl # Kernel SLIP
> device ppp # Kernel PPP
> device tun # Packet tunnel.
> device pty # Pseudo-ttys (telnet etc)
> device md # Memory "disks"
> device gif # IPv6 and IPv4 tunneling
> device faith # IPv6-to-IPv4 relaying (translation)
>
> # The `bpf' device enables the Berkeley Packet Filter.
> # Be aware of the administrative consequences of enabling this!
> # Note that 'bpf' is required for DHCP.
> device bpf # Berkeley packet filter
>
> # USB support
> #device uhci # UHCI PCI->USB interface
> #device ohci # OHCI PCI->USB interface
> #device ehci # EHCI PCI->USB interface (USB 2.0)
> #device usb # USB Bus (required)
> #device udbp # USB Double Bulk Pipe devices
> #device ugen # Generic
> #device uhid # "Human Interface Devices"
> #device ukbd # Keyboard
> #device ulpt # Printer
> #device umass # Disks/Mass storage - Requires scbus and da
> #device ums # Mouse
> #device urio # Diamond Rio 500 MP3 player
> #device uscanner # Scanners
> # USB Ethernet, requires mii
> #device aue # ADMtek USB Ethernet
> #device axe # ASIX Electronics USB Ethernet
> #device cdce # Generic USB over Ethernet
> #device cue # CATC USB Ethernet
> #device kue # Kawasaki LSI USB Ethernet
> #device rue # RealTek RTL8150 USB Ethernet
>
> # FireWire support
> #device firewire # FireWire bus code
> #device sbp # SCSI over FireWire (Requires scbus and da)
> #device fwe # Ethernet over FireWire (non-standard!)
>
>
> --
> K.F.J. Martens, Sonologic, http://www.sonologic.nl/ Networking,
> hosting, embedded systems, unix, artificial intelligence. Public PGP
> key: http://www.metro.cx/pubkey-gmc.asc Wondering about the funny
> attachment your mail program can't read? Visit http://www.openpgp.org/
--
John Baldwin <jhb at FreeBSD.org> <>< http://www.FreeBSD.org/~jhb/
"Power Users Use the Power to Serve" = http://www.FreeBSD.org
More information about the freebsd-hackers
mailing list