Re: kernel core debugging

From: Mark Millard <marklmi_at_yahoo.com>
Date: Wed, 06 Nov 2024 15:12:44 UTC
On Nov 6, 2024, at 01:44, tuexen@freebsd.org <tuexen@FreeBSD.org> wrote:

> is debugging a kernel panic by using kgdb or lldb on a core file
> supposed to work? At least it is not right now for me...

# kgdb /boot/kernel.GENERIC-NODEBUG/kernel /var/crash/vmcore.2
GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD]
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-portbld-freebsd15.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel.GENERIC-NODEBUG/kernel...
Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/kernel.debug...

Unread portion of the kernel message buffer:
KDB: enter: manual escape to debugger

Reading symbols from /boot/kernel.GENERIC-NODEBUG/uhid.ko...
Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/uhid.ko.debug...
Reading symbols from /boot/kernel.GENERIC-NODEBUG/wmt.ko...
Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/wmt.ko.debug...
Reading symbols from /boot/kernel.GENERIC-NODEBUG/ums.ko...
Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/ums.ko.debug...
Reading symbols from /boot/kernel.GENERIC-NODEBUG/zfs.ko...
Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/zfs.ko.debug...
0xffff00000050f3f0 in doadump (textdump=0, textdump@entry=212431136) at /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c:404
warning: 404 /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c: No such file or directory
(kgdb) bt
#0  0xffff00000050f3f0 in doadump (textdump=0, textdump@entry=212431136) at /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c:404
#1  0xffff0000000ee6a8 in db_dump (dummy=<optimized out>, dummy2=<optimized out>, dummy3=<optimized out>, dummy4=<optimized out>) at /home/pkgbuild/worktrees/main/sys/ddb/db_command.c:596
#2  0xffff0000000ee478 in db_command (last_cmdp=<optimized out>, cmd_table=<optimized out>, dopager=true) at /home/pkgbuild/worktrees/main/sys/ddb/db_command.c:508
#3  0xffff0000000ee150 in db_command_loop () at /home/pkgbuild/worktrees/main/sys/ddb/db_command.c:555
#4  0xffff0000000f1ff4 in db_trap (type=<optimized out>, code=<optimized out>) at /home/pkgbuild/worktrees/main/sys/ddb/db_main.c:267
#5  0xffff000000568b0c in kdb_trap (type=60, code=0, tf=<optimized out>) at /home/pkgbuild/worktrees/main/sys/kern/subr_kdb.c:790
#6  <signal handler called>
#7  kdb_enter (why=<optimized out>, msg=<optimized out>) at /home/pkgbuild/worktrees/main/sys/kern/subr_kdb.c:556
#8  0xffff0000003625cc in vt_machine_kbdevent (vd=<optimized out>, c=<optimized out>) at /home/pkgbuild/worktrees/main/sys/dev/vt/vt_core.c:761
#9  vt_processkey (kbd=0xffffa000803caa80, vd=0xffff000000d24360 <vt_consdev>, c=-2147483514) at /home/pkgbuild/worktrees/main/sys/dev/vt/vt_core.c:903
#10 vt_kbdevent (kbd=0xffffa000803caa80, event=<optimized out>, arg=0xffff000000d24360 <vt_consdev>) at /home/pkgbuild/worktrees/main/sys/dev/vt/vt_core.c:1030
#11 0xffff0000001ea048 in kbdmux_intr (kbd=0xffffa000803caa80, arg=<optimized out>) at /home/pkgbuild/worktrees/main/sys/dev/kbdmux/kbdmux.c:565
#12 0xffff0000005839ac in taskqueue_run_locked (queue=queue@entry=0xffffa000803c9c00) at /home/pkgbuild/worktrees/main/sys/kern/subr_taskqueue.c:517
#13 0xffff000000583714 in taskqueue_run (queue=0xffffa000803c9c00) at /home/pkgbuild/worktrees/main/sys/kern/subr_taskqueue.c:532
#14 0xffff0000004bc114 in intr_event_execute_handlers (ie=0xffffa0008028ec00, p=<optimized out>) at /home/pkgbuild/worktrees/main/sys/kern/kern_intr.c:1183
#15 ithread_execute_handlers (ie=0xffffa0008028ec00, p=<optimized out>) at /home/pkgbuild/worktrees/main/sys/kern/kern_intr.c:1196
#16 ithread_loop (arg=<optimized out>, arg@entry=0xffffa000803de5a0) at /home/pkgbuild/worktrees/main/sys/kern/kern_intr.c:1289
#17 0xffff0000004b700c in fork_exit (callout=0xffff0000004bbd78 <ithread_loop>, arg=0xffffa000803de5a0, frame=0xffff00010ca97a00) at /home/pkgbuild/worktrees/main/sys/kern/kern_fork.c:1151
#18 <signal handler called>

The context here was from an official PkgBase kernel and world
installation.

I'll note that, if I reference the wrong kernel file, I get
the likes of:

# kgdb /boot/kernel/kernel /var/crash/vmcore.2
. . .
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...
/wrkdirs/usr/ports/devel/gdb/work-py311/gdb-15.1/gdb/thread.c:1423: internal-error: switch_to_thread: Assertion `thr != NULL' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
----- Backtrace -----
0x13147fb ???
0x1777763 ???
0x1777593 ???
0x1c09bbf ???
0x1733af7 ???
0x143c9d3 ???
0x1707483 ???
0x13468db ???
0x17405a3 ???
0x1505f7b ???
0x1504bd7 ???
0x150352f ???
0x1226b47 ???
0x2ee9ca33 __libc_start1
/home/pkgbuild/worktrees/main/lib/libc/csu/libc_start1.c:172
0x12261cb ???
---------------------
/wrkdirs/usr/ports/devel/gdb/work-py311/gdb-15.1/gdb/thread.c:1423: internal-error: switch_to_thread: Assertion `thr != NULL' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) y
. . .

You may have to be more explicit about the specific of the
problem(s) you are seeing.

For reference:

# uname -apKU
FreeBSD aarch64-main-PBsmallUFS 15.0-CURRENT FreeBSD 15.0-CURRENT main-n273174-8b2e7da70855 GENERIC arm64 aarch64 1500026 1500026

# ~/pkgbase-snapshot-list.sh 
Via pkg-static info -C -x '^FreeBSD-' . . .
 352 FreeBSD-*-15.snap20241023235252
   1 FreeBSD-*-15.snap20241022122410
   1 FreeBSD-*-15.snap20241020224518
   1 FreeBSD-*-15.snap20241020094723
   7 FreeBSD-*-15.snap20241015203742
   1 FreeBSD-*-15.snap20241015145839
   1 FreeBSD-*-15.snap20241015120827
   1 FreeBSD-*-15.snap20241014101436
  34 FreeBSD-*-15.snap20241011184813
 129 FreeBSD-*-15.snap20241009162208
Instead via /var/cache/pkg/*.snap*.pkg . . .
 352 FreeBSD-*-15.snap20241023235252
   1 FreeBSD-*-15.snap20241022122410
   1 FreeBSD-*-15.snap20241020224518
   1 FreeBSD-*-15.snap20241020094723
   7 FreeBSD-*-15.snap20241015203742
   1 FreeBSD-*-15.snap20241015145839
   1 FreeBSD-*-15.snap20241015120827
   1 FreeBSD-*-15.snap20241014101436
  34 FreeBSD-*-15.snap20241011184813
 129 FreeBSD-*-15.snap20241009162208

# pkg info gdb
gdb-15.1
Name           : gdb
Version        : 15.1
Installed on   : Mon Oct 14 10:30:51 2024 PDT
Origin         : devel/gdb
Architecture   : FreeBSD:15:aarch64
Prefix         : /usr/local
Categories     : devel
Licenses       : GPLv3
Maintainer     : pizzamig@FreeBSD.org
WWW            : https://www.gnu.org/software/gdb/
Comment        : GNU Project Debugger
Options        :
BUNDLED_READLINE: off
BUNDLED_ZLIB   : off
DEBUGINFOD     : off
GDB_LINK       : on
GUILE          : off
KGDB           : on
NLS            : on
PORT_ICONV     : on
PORT_READLINE  : on
PYTHON         : on
SOURCE_HIGHLIGHT: on
SYSTEM_ICONV   : off
SYSTEM_ZLIB    : on
TUI            : on
XXHASH         : on
Shared Libs required:
libzstd.so.1
libxxhash.so.0
libsource-highlight.so.4
libreadline.so.8
libpython3.11.so.1.0
libmpfr.so.6
libintl.so.8
libiconv.so.2
libgmp.so.10
libexpat.so.1
libboost_regex.so.1.85.0
Annotations    :
FreeBSD_version: 1500024
build_timestamp: 2024-10-07T19:01:59+0000
built_by       : poudriere-git-3.4.2
cpe            : cpe:2.3:a:gnu:gdb:15.1:::::freebsd15:aarch64
flavor         : py311
port_checkout_unclean: no
port_git_hash  : 82beca9e630
ports_top_checkout_unclean: no
ports_top_git_hash: 149fe86b8e7
repo_type      : binary
repository     : FreeBSD
Flat size      : 58.5MiB
Description    :
GDB is a source-level debugger for Ada, C, C++, Objective-C, Pascal and
many other languages.  GDB can target (i.e., debug programs running on)
more than a dozen different processor architectures, and GDB itself can
run on most popular GNU/Linux, Unix and Microsoft Windows variants.

It was from installing an officially built package.


===
Mark Millard
marklmi at yahoo.com