extract panic message & debugging from vmcore.0 ?

[Mark Millard]

[Some more notes prompted by your reply.]

On 2017-Jun-29, at 7:35 PM, Mark Millard <markmi at dsl-only.net> wrote:

> On 2017-Jun-29, at 7:12 PM, Michael W. Lucas <mwlucas at michaelwlucas.com> wrote:
> 
>> 
>> # cd /usr/obj/usr/src/sys/GENERIC
>> # make gdbinit
>> # gdb kernel.debug
> 
> Did you miss a /var/crash/vmcore.0 above?

I should have noted that gdbinit is likely
for setting up to use gdb remotely against
a live system for remote kernel debugging.
(Not something that I've ever done.)

It likely does nothing useful for vmcore.*
inspection. gdb itself does not deal with
vmcore.* files to my knowledge.

So this likely was instead a false direction
for your purposes.

> Separately. . .
> 
> I'm no expert but you might need to pick
> between the default minidump vs. a full
> memory dump depending on purposes and what
> information is to be extracted:
> 
> debug.minidump: 1
> vs.
> debug.minidump: 0
> 
> in the sysctl debug.minidump output. I
> doubt that they are fully equivalent.
> 
> Of course a full memory dump implies
> needing space ready to hold that full
> copy.

I'll also note that virtual addresses
vs. physical addresses are likely an
issue for at least debug.minidump=0 .


As for the crashinfo that you mention
in your reply: On head -320482 . . .

# grep -r crashinfo /etc
/etc/rc.d/savecore:		if checkyesno crashinfo_enable; then
/etc/rc.d/savecore:			${crashinfo_program} -b -d ${dumpdir}
/etc/defaults/rc.conf:crashinfo_enable="YES"	# Automatically generate crash dump summary.
/etc/defaults/rc.conf:crashinfo_program="/usr/sbin/crashinfo"	# Script to generate crash dump summary.

Normally the boot sequence produces a crashinfo report
after creating the vmcore.* file, at least if crashinfo
is enabled.

The context for the below is a powerpc with
debug.minidump=0 (disabled) and a production
kernel for the failures. (The debug kernel
does not get the problems involved for
head -r320482 .) vmcore.*'s, core.txt.*'s,
info.*'s and *.last's were all produced
towards the end of the boot sequence before
the login prompt each time.

# ls -lT /var/crash/
total 2685292
-rw-r--r--  1 root  wheel           2 Jun 29 23:17:52 2017 bounds
-rw-------  1 root  wheel           0 Jun 29 22:47:14 2017 core.txt.5
-rw-------  1 root  wheel       26573 Jun 29 23:18:51 2017 core.txt.6
-rw-------  1 root  wheel         351 Jun 29 22:46:15 2017 info.5
-rw-------  1 root  wheel         378 Jun 29 23:17:52 2017 info.6
lrwxr-xr-x  1 root  wheel           0 Jun 29 22:47:06 2017 info.last -> 
-rw-r--r--  1 root  wheel           5 Feb 22 02:37:33 2016 minfree
-rw-------  1 root  wheel  2147487744 Jun 29 22:47:06 2017 vmcore.5
-rw-------  1 root  wheel  2147487744 Jun 29 23:18:43 2017 vmcore.6
lrwxr-xr-x  1 root  wheel           0 Jun 29 22:47:06 2017 vmcore.last -> 

(There are oddities for the system that
have made the links and core.txt.5 empty.)

# more /var/crash/info.5
Dump header from device: /dev/label/FBSDG4Sswap
  Architecture: powerpc
  Architecture Version: 1
  Dump Length: 2147487744
  Blocksize: 512
  Dumptime: Thu Jun 29 22:34:22 2017
  Hostname: FBSDG4S
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 12.0-CURRENT  r320482M
  Panic String: 
  Dump Parity: 4013015019
  Bounds: 5
  Dump Status: good

core.txt.6 attempted to have sections for:

ps -axlww
vmstat -s
vmstat -m
vmstat -z
vmstat -i
pstat -T
pstat -s
iostat
ipcs -a
ipcs -T
nfsstat
netstat -s
netstat -m
netstat -anA
netstat -aL
fstat
dmesg
kernel config
ddb capture buffer

(That does not mean that they all were
useful/accurate.)

I'll not list the text here.

Notes:

powerpc is not actually supported for kgdb
and the like and I had to change some things
to get this much working as well as it does
for me now. (It is limited but has been
useful to me.)


===
Mark Millard
markmi at dsl-only.net