panic in propagate_priority w/ postgresql under heavy load

John Baldwin jhb at FreeBSD.org
Thu Sep 1 10:49:59 PDT 2005


On Thursday 01 September 2005 01:02 pm, Koen Martens wrote:
> Hi Hackers,
>
> I've had a little chat with neologism on ircnet/#freebsd about this
> already, and done as he suggested: compile a debug kernel to obtain
> a stack trace.

Can you reproduce it with a kernel that has INVARIANTS and INVARIANT_SUPPORT 
on?  I see that you had WITNESS on, can you check to see if there were any 
witness messages about sleepign with non-sleepable locks held before the 
crash?

> Anyway, what is happening is that there is a crash when running
> postgresql 8.0.3 with a very large database and doing heavy queries.
>
> Kernel is 5.4-RELEASE-p6 (5.4-RELENG i checked out tuesday a week
> ago). Kernel conf is down below.
>
> Here is the message i get on the console:
>
> ====================
> kernel trap 12 with interrupts disabled
>
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 1; apic id = 06
> fault virtual address   = 0x24
> fault code              = supervisor read, page not present
> instruction pointer     = 0x8:0xc050cff7
> stack pointer           = 0x10:0xe99c2b0c
> frame pointer           = 0x10:0xe99c2b20
> code segment            = base 0x0, limit 0xfffff, type 0x1b
>                         = DPL 0, pres 1, def32 1, gran 1
> processor eflags        = resume, IOPL = 0
> current process         = 4571 (postgres)
> ====================
>
> It has been a postgres process in all of the observed cases.
>
> I've looked it up with gdk on my kernel.debug, here's what i get:
>
> =====================
> yin# gdb kernel.debug
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and
> you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for
> details.
> This GDB was configured as "i386-marcel-freebsd"...
> (gdb) l *propagate_priority+0x7f
> 0xc050cff7 is in propagate_priority
> (/usr/src/sys/kern/subr_turnstile.c:245).
> 240                     /*
> 241                      * Pick up the lock that td is blocked on.
> 242                      */
> 243                     ts = td->td_blocked;
> 244                     MPASS(ts != NULL);
> 245                     tc = TC_LOOKUP(ts->ts_lockobj);
> 246                     mtx_lock_spin(&tc->tc_lock);
> 247
> 248                     /*
> 249                      * This thread may not be blocked on this
> turnstile anymore
> (gdb)
> =====================
>
>
> So the next thing you'll ask for is a stack trace, but i haven't
> been able to obtain one. According to the freebsd handbook
> (http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerne
>ldebug-gdb.html) there should be a core dump in /var/crash, but there is
> none and the handbook chapter seems outdated anyway judging by it
> mentioning kgdb...
>
> Anyway, it seems the dump should've gone to the swap partition, but
> i'm into multi-user mode again so i guess i'll have to wait for
> another panic to obtain it?
>
> I'm not very knowledgeable about the freebsd kernel internals (yet),
> so i'm not sure what could be wrong here.. I hope some of you can
> provide some insight, and ideally a fix :)
>
> ======================[ kernel config:
>
> #
> # GENERIC -- Generic kernel configuration file for FreeBSD/i386
> #
> # For more information on this file, please read the handbook section on
> # Kernel Configuration Files:
> #
> #
> http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-conf
>ig.html #
> # The handbook is also available locally in /usr/share/doc/handbook
> # if you've installed the doc distribution, otherwise always see the
> # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the
> # latest information.
> #
> # An exhaustive list of options and more detailed explanations of the
> # device lines is also present in the ../../conf/NOTES and NOTES files.
> # If you are in doubt as to the purpose or necessity of a line,
> check first
> # in NOTES.
> #
> # $FreeBSD: src/sys/i386/conf/GENERIC,v 1.413.2.13 2005/04/02
> 16:37:58 scottl Exp $
>
> machine		i386
> cpu		I486_CPU
> cpu		I586_CPU
> cpu		I686_CPU
> ident		YIN-YANG
>
> # debug
> options         WITNESS
> options         KDB
> options         DDB
> #
> makeoptions	DEBUG=-g
> # debug
>
>
> # To statically compile in device wiring instead of /boot/device.hints
> #hints		"GENERIC.hints"		# Default places to look for devices.
>
> options 	SCHED_4BSD		# 4BSD scheduler
> options 	INET			# InterNETworking
> options 	INET6			# IPv6 communications protocols
> options 	FFS			# Berkeley Fast Filesystem
> options 	SOFTUPDATES		# Enable FFS soft updates support
> options 	UFS_ACL			# Support for access control lists
> options 	UFS_DIRHASH		# Improve performance on big directories
> options 	MD_ROOT			# MD is a potential root device
> options 	NFSCLIENT		# Network Filesystem Client
> options 	NFSSERVER		# Network Filesystem Server
> options 	NFS_ROOT		# NFS usable as /, requires NFSCLIENT
> options 	MSDOSFS			# MSDOS Filesystem
> options 	CD9660			# ISO 9660 Filesystem
> options 	PROCFS			# Process filesystem (requires PSEUDOFS)
> options 	PSEUDOFS		# Pseudo-filesystem framework
> options 	GEOM_GPT		# GUID Partition Tables.
> options 	COMPAT_43		# Compatible with BSD 4.3 [KEEP THIS!]
> options 	COMPAT_FREEBSD4		# Compatible with FreeBSD4
> options 	SCSI_DELAY=15000	# Delay (in ms) before probing SCSI
> options 	KTRACE			# ktrace(1) support
> options 	SYSVSHM			# SYSV-style shared memory
> options 	SYSVMSG			# SYSV-style message queues
> options 	SYSVSEM			# SYSV-style semaphores
> options 	_KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time
> extensions
> options 	KBD_INSTALL_CDEV	# install a CDEV entry in /dev
> options 	AHC_REG_PRETTY_PRINT	# Print register bitfields in debug
> 					# output.  Adds ~128k to driver.
> options 	AHD_REG_PRETTY_PRINT	# Print register bitfields in debug
> 					# output.  Adds ~215k to driver.
> options 	ADAPTIVE_GIANT		# Giant mutex is adaptive.
>
>
> options         IPFILTER
> options         IPFILTER_LOG
> options         IPFILTER_DEFAULT_BLOCK
>
> options         IPFILTER
> options         IPFILTER_LOG
>
> options         IPFIREWALL              #firewall
> options         IPFIREWALL_VERBOSE      #enable logging to syslogd(8)
> options         IPFIREWALL_FORWARD      #enable transparent proxy
> support
> options         IPFIREWALL_VERBOSE_LIMIT=10    #limit verbosity
> options         IPFIREWALL_DEFAULT_TO_ACCEPT    #allow everything by
> default
> options         IPV6FIREWALL            #firewall for IPv6
> options         IPV6FIREWALL_VERBOSE
> options         IPV6FIREWALL_VERBOSE_LIMIT=10
> options         IPV6FIREWALL_DEFAULT_TO_ACCEPT
>
> options         SHMMAXPGS=262144
> options         SHMSEG=512
> options         SHMMNI=512
> options         SEMMNI=512
> options         SEMMNS=1024
> options         SEMMNU=512
> options         SEMMAP=512
> options         NMBCLUSTERS=32768
>
> device		apic			# I/O APIC
>
> options         SMP                     # Symmetric MultiProcessor
> Kernel
>
> # Bus support.  Do not remove isa, even if you have no isa slots
> device		isa
> device		eisa
> device		pci
>
> # Floppy drives
> device		fdc
>
> # ATA and ATAPI devices
> device		ata
> device		atadisk		# ATA disk drives
> device		ataraid		# ATA RAID drives
> device		atapicd		# ATAPI CDROM drives
> #device		atapifd		# ATAPI floppy drives
> #device		atapist		# ATAPI tape drives
> options 	ATA_STATIC_ID	# Static device numbering
>
> # SCSI Controllers
> #device		ahb		# EISA AHA1742 family
> #device		ahc		# AHA2940 and onboard AIC7xxx devices
> #device		ahd		# AHA39320/29320 and onboard AIC79xx devices
> #device		amd		# AMD 53C974 (Tekram DC-390(T))
> #device		isp		# Qlogic family
> #device		mpt		# LSI-Logic MPT-Fusion
> ##device		ncr		# NCR/Symbios Logic
> #device		sym		# NCR/Symbios Logic (newer chipsets + those of `ncr')
> #device		trm		# Tekram DC395U/UW/F DC315U adapters
>
> #device		adv		# Advansys SCSI adapters
> #device		adw		# Advansys wide SCSI adapters
> #device		aha		# Adaptec 154x SCSI adapters
> #device		aic		# Adaptec 15[012]x SCSI adapters, AIC-6[23]60.
> #device		bt		# Buslogic/Mylex MultiMaster SCSI adapters
>
> #device		ncv		# NCR 53C500
> #device		nsp		# Workbit Ninja SCSI-3
> #device		stg		# TMC 18C30/18C50
>
> # SCSI peripherals
> device		scbus		# SCSI bus (required for SCSI)
> device		ch		# SCSI media changers
> device		da		# Direct Access (disks)
> device		sa		# Sequential Access (tape etc)
> device		cd		# CD
> device		pass		# Passthrough device (direct SCSI access)
> device		ses		# SCSI Environmental Services (and SAF-TE)
>
> # RAID controllers interfaced to the SCSI subsystem
> #device		amr		# AMI MegaRAID
> #device		arcmsr		# Areca SATA II RAID
> #device		asr		# DPT SmartRAID V, VI and Adaptec SCSI RAID
> #device		ciss		# Compaq Smart RAID 5*
> #device		dpt		# DPT Smartcache III, IV - See NOTES for options
> #device		hptmv		# Highpoint RocketRAID 182x
> #device		iir		# Intel Integrated RAID
> #device		ips		# IBM (Adaptec) ServeRAID
> #device		mly		# Mylex AcceleRAID/eXtremeRAID
> device		twa		# 3ware 9000 series PATA/SATA RAID
>
> # RAID controllers
> #device		aac		# Adaptec FSA RAID
> #device		aacp		# SCSI passthrough for aac (requires CAM)
> #device		ida		# Compaq Smart RAID
> #device		mlx		# Mylex DAC960 family
> #device		pst		# Promise Supertrak SX6000
> device		twe		# 3ware ATA RAID
>
> # atkbdc0 controls both the keyboard and the PS/2 mouse
> device		atkbdc		# AT keyboard controller
> device		atkbd		# AT keyboard
> device		psm		# PS/2 mouse
>
> device		vga		# VGA video card driver
>
> #device		splash		# Splash screen and screen saver support
>
> # syscons is the default console driver, resembling an SCO console
> device		sc
>
> # Enable this for the pcvt (VT220 compatible) console driver
> #device		vt
> #options 	XSERVER		# support for X server on a vt console
> #options 	FAT_CURSOR	# start with block cursor
>
> device		agp		# support several AGP chipsets
>
> # Floating point support - do not disable.
> device		npx
>
> # Power management support (see NOTES for more options)
> #device		apm
> # Add suspend/resume support for the i8254.
> device		pmtimer
>
> # PCCARD (PCMCIA) support
> # PCMCIA and cardbus bridge support
> #device		cbb		# cardbus (yenta) bridge
> #device		pccard		# PC Card (16-bit) bus
> #device		cardbus		# CardBus (32-bit) bus
>
> # Serial (COM) ports
> device		sio		# 8250, 16[45]50 based serial ports
>
> # Parallel port
> device		ppc
> device		ppbus		# Parallel port bus (required)
> device		lpt		# Printer
> device		plip		# TCP/IP over parallel
> device		ppi		# Parallel port interface device
> #device		vpo		# Requires scbus and da
>
> # If you've got a "dumb" serial or parallel PCI card that is
> # supported by the puc(4) glue driver, uncomment the following
> # line to enable it (connects to the sio and/or ppc drivers):
> #device         puc
>
> # PCI Ethernet NICs.
> #device		de		# DEC/Intel DC21x4x (``Tulip'')
> device		em		# Intel PRO/1000 adapter Gigabit Ethernet Card
> #device		ixgb		# Intel PRO/10GbE Ethernet Card
> #device		txp		# 3Com 3cR990 (``Typhoon'')
> #device		vx		# 3Com 3c590, 3c595 (``Vortex'')
>
> # PCI Ethernet NICs that use the common MII bus controller code.
> # NOTE: Be sure to keep the 'device miibus' line in order to use
> these NICs!
> device		miibus		# MII bus support
> #device		bfe		# Broadcom BCM440x 10/100 Ethernet
> #device		bge		# Broadcom BCM570xx Gigabit Ethernet
> #device		dc		# DEC/Intel 21143 and various workalikes
> device		fxp		# Intel EtherExpress PRO/100B (82557, 82558)
> #device		lge		# Level 1 LXT1001 gigabit ethernet
> #device		nge		# NatSemi DP83820 gigabit ethernet
> #evice		pcn		# AMD Am79C97x PCI 10/100 (precedence over 'lnc')
> #device		re		# RealTek 8139C+/8169/8169S/8110S
> #device		rl		# RealTek 8129/8139
> #device		sf		# Adaptec AIC-6915 (``Starfire'')
> #device		sis		# Silicon Integrated Systems SiS 900/SiS 7016
> #device		sk		# SysKonnect SK-984x & SK-982x gigabit Ethernet
> #device		ste		# Sundance ST201 (D-Link DFE-550TX)
> #device		ti		# Alteon Networks Tigon I/II gigabit Ethernet
> #device		tl		# Texas Instruments ThunderLAN
> #device		tx		# SMC EtherPower II (83c170 ``EPIC'')
> #device		vge		# VIA VT612x gigabit ethernet
> #device		vr		# VIA Rhine, Rhine II
> #device		wb		# Winbond W89C840F
> #device		xl		# 3Com 3c90x (``Boomerang'', ``Cyclone'')
>
> # ISA Ethernet NICs.  pccard NICs included.
> #device		cs		# Crystal Semiconductor CS89x0 NIC
> # 'device ed' requires 'device miibus'
> #device		ed		# NE[12]000, SMC Ultra, 3c503, DS8390 cards
> #device		ex		# Intel EtherExpress Pro/10 and Pro/10+
> #device		ep		# Etherlink III based cards
> #device		fe		# Fujitsu MB8696x based cards
> #device		ie		# EtherExpress 8/16, 3C507, StarLAN 10 etc.
> #device		lnc		# NE2100, NE32-VL Lance Ethernet cards
> #device		sn		# SMC's 9000 series of Ethernet chips
> #device		xe		# Xircom pccard Ethernet
>
> # ISA devices that use the old ISA shims
> #device		le
>
> # Wireless NIC cards
> #device		wlan		# 802.11 support
> #device		an		# Aironet 4500/4800 802.11 wireless NICs.
> #device		awi		# BayStack 660 and others
> #device		wi		# WaveLAN/Intersil/Symbol 802.11 wireless NICs.
> #device		wl		# Older non 802.11 Wavelan wireless NIC.
>
> # Pseudo devices.
> device		loop		# Network loopback
> device		mem		# Memory and kernel memory devices
> device		io		# I/O device
> device		random		# Entropy device
> device		ether		# Ethernet support
> device		sl		# Kernel SLIP
> device		ppp		# Kernel PPP
> device		tun		# Packet tunnel.
> device		pty		# Pseudo-ttys (telnet etc)
> device		md		# Memory "disks"
> device		gif		# IPv6 and IPv4 tunneling
> device		faith		# IPv6-to-IPv4 relaying (translation)
>
> # The `bpf' device enables the Berkeley Packet Filter.
> # Be aware of the administrative consequences of enabling this!
> # Note that 'bpf' is required for DHCP.
> device		bpf		# Berkeley packet filter
>
> # USB support
> #device		uhci		# UHCI PCI->USB interface
> #device		ohci		# OHCI PCI->USB interface
> #device		ehci		# EHCI PCI->USB interface (USB 2.0)
> #device		usb		# USB Bus (required)
> #device		udbp		# USB Double Bulk Pipe devices
> #device		ugen		# Generic
> #device		uhid		# "Human Interface Devices"
> #device		ukbd		# Keyboard
> #device		ulpt		# Printer
> #device		umass		# Disks/Mass storage - Requires scbus and da
> #device		ums		# Mouse
> #device		urio		# Diamond Rio 500 MP3 player
> #device		uscanner	# Scanners
> # USB Ethernet, requires mii
> #device		aue		# ADMtek USB Ethernet
> #device		axe		# ASIX Electronics USB Ethernet
> #device		cdce		# Generic USB over Ethernet
> #device		cue		# CATC USB Ethernet
> #device		kue		# Kawasaki LSI USB Ethernet
> #device		rue		# RealTek RTL8150 USB Ethernet
>
> # FireWire support
> #device		firewire	# FireWire bus code
> #device		sbp		# SCSI over FireWire (Requires scbus and da)
> #device		fwe		# Ethernet over FireWire (non-standard!)
>
>
> --
> K.F.J. Martens, Sonologic, http://www.sonologic.nl/ Networking,
> hosting, embedded systems, unix, artificial intelligence. Public PGP
> key: http://www.metro.cx/pubkey-gmc.asc Wondering about the funny
> attachment your mail program can't read? Visit http://www.openpgp.org/

-- 
John Baldwin <jhb at FreeBSD.org>  <><  http://www.FreeBSD.org/~jhb/
"Power Users Use the Power to Serve"  =  http://www.FreeBSD.org


More information about the freebsd-hackers mailing list