From nobody Wed Nov 06 17:28:53 2024 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XkByY5RvWz1GlCr for ; Wed, 06 Nov 2024 17:29:09 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from drew.franken.de (mail-n.franken.de [193.175.24.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "Certum Domain Validation CA SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XkByY2g8Wz4gTH for ; Wed, 6 Nov 2024 17:29:09 +0000 (UTC) (envelope-from tuexen@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (unknown [IPv6:2001:67c:370:128:c74:3c49:cab8:7b26]) (Authenticated sender: micmac) by drew.franken.de (Postfix) with ESMTPSA id 45524721BE01A; Wed, 6 Nov 2024 18:28:54 +0100 (CET) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.200.121\)) Subject: Re: kernel core debugging From: tuexen@freebsd.org In-Reply-To: Date: Wed, 6 Nov 2024 17:28:53 +0000 Cc: freebsd-arm Content-Transfer-Encoding: quoted-printable Message-Id: <92795A92-653D-4DED-ADB2-3BAAF181B813@freebsd.org> References: <34E3792D-E464-45A8-938A-AD266F2EB49F@freebsd.org> To: Mark Millard X-Mailer: Apple Mail (2.3826.200.121) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:680, ipnet:193.174.0.0/15, country:DE] X-Rspamd-Queue-Id: 4XkByY2g8Wz4gTH X-Spamd-Bar: ---- > On 6. Nov 2024, at 15:12, Mark Millard wrote: >=20 > On Nov 6, 2024, at 01:44, tuexen@freebsd.org = wrote: >=20 >> is debugging a kernel panic by using kgdb or lldb on a core file >> supposed to work? At least it is not right now for me... >=20 > # kgdb /boot/kernel.GENERIC-NODEBUG/kernel /var/crash/vmcore.2 > GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD] > Copyright (C) 2024 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later = > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. > Type "show copying" and "show warranty" for details. > This GDB was configured as "aarch64-portbld-freebsd15.0". > Type "show configuration" for configuration details. > For bug reporting instructions, please see: > . > Find the GDB manual and other documentation resources online at: > . >=20 > For help, type "help". > Type "apropos word" to search for commands related to "word"... > Reading symbols from /boot/kernel.GENERIC-NODEBUG/kernel... > Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/kernel.debug... >=20 > Unread portion of the kernel message buffer: > KDB: enter: manual escape to debugger >=20 > Reading symbols from /boot/kernel.GENERIC-NODEBUG/uhid.ko... > Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/uhid.ko.debug... > Reading symbols from /boot/kernel.GENERIC-NODEBUG/wmt.ko... > Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/wmt.ko.debug... > Reading symbols from /boot/kernel.GENERIC-NODEBUG/ums.ko... > Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/ums.ko.debug... > Reading symbols from /boot/kernel.GENERIC-NODEBUG/zfs.ko... > Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/zfs.ko.debug... > 0xffff00000050f3f0 in doadump (textdump=3D0, textdump@entry=3D212431136)= at /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c:404 > warning: 404 /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c: = No such file or directory > (kgdb) bt > #0 0xffff00000050f3f0 in doadump (textdump=3D0, = textdump@entry=3D212431136) at = /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c:404 > #1 0xffff0000000ee6a8 in db_dump (dummy=3D, = dummy2=3D, dummy3=3D, dummy4=3D) at /home/pkgbuild/worktrees/main/sys/ddb/db_command.c:596 > #2 0xffff0000000ee478 in db_command (last_cmdp=3D, = cmd_table=3D, dopager=3Dtrue) at = /home/pkgbuild/worktrees/main/sys/ddb/db_command.c:508 > #3 0xffff0000000ee150 in db_command_loop () at = /home/pkgbuild/worktrees/main/sys/ddb/db_command.c:555 > #4 0xffff0000000f1ff4 in db_trap (type=3D, = code=3D) at = /home/pkgbuild/worktrees/main/sys/ddb/db_main.c:267 > #5 0xffff000000568b0c in kdb_trap (type=3D60, code=3D0, tf=3D) at /home/pkgbuild/worktrees/main/sys/kern/subr_kdb.c:790 > #6 > #7 kdb_enter (why=3D, msg=3D) at = /home/pkgbuild/worktrees/main/sys/kern/subr_kdb.c:556 > #8 0xffff0000003625cc in vt_machine_kbdevent (vd=3D, = c=3D) at = /home/pkgbuild/worktrees/main/sys/dev/vt/vt_core.c:761 > #9 vt_processkey (kbd=3D0xffffa000803caa80, vd=3D0xffff000000d24360 = , c=3D-2147483514) at = /home/pkgbuild/worktrees/main/sys/dev/vt/vt_core.c:903 > #10 vt_kbdevent (kbd=3D0xffffa000803caa80, event=3D, = arg=3D0xffff000000d24360 ) at = /home/pkgbuild/worktrees/main/sys/dev/vt/vt_core.c:1030 > #11 0xffff0000001ea048 in kbdmux_intr (kbd=3D0xffffa000803caa80, = arg=3D) at = /home/pkgbuild/worktrees/main/sys/dev/kbdmux/kbdmux.c:565 > #12 0xffff0000005839ac in taskqueue_run_locked = (queue=3Dqueue@entry=3D0xffffa000803c9c00) at = /home/pkgbuild/worktrees/main/sys/kern/subr_taskqueue.c:517 > #13 0xffff000000583714 in taskqueue_run (queue=3D0xffffa000803c9c00) = at /home/pkgbuild/worktrees/main/sys/kern/subr_taskqueue.c:532 > #14 0xffff0000004bc114 in intr_event_execute_handlers = (ie=3D0xffffa0008028ec00, p=3D) at = /home/pkgbuild/worktrees/main/sys/kern/kern_intr.c:1183 > #15 ithread_execute_handlers (ie=3D0xffffa0008028ec00, p=3D) at /home/pkgbuild/worktrees/main/sys/kern/kern_intr.c:1196 > #16 ithread_loop (arg=3D, arg@entry=3D0xffffa000803de5a0)= at /home/pkgbuild/worktrees/main/sys/kern/kern_intr.c:1289 > #17 0xffff0000004b700c in fork_exit (callout=3D0xffff0000004bbd78 = , arg=3D0xffffa000803de5a0, frame=3D0xffff00010ca97a00) at = /home/pkgbuild/worktrees/main/sys/kern/kern_fork.c:1151 > #18 >=20 > The context here was from an official PkgBase kernel and world > installation. >=20 > I'll note that, if I reference the wrong kernel file, I get > the likes of: >=20 > # kgdb /boot/kernel/kernel /var/crash/vmcore.2 > . . . > Reading symbols from /boot/kernel/kernel... > Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug... > /wrkdirs/usr/ports/devel/gdb/work-py311/gdb-15.1/gdb/thread.c:1423: = internal-error: switch_to_thread: Assertion `thr !=3D NULL' failed. > A problem internal to GDB has been detected, > further debugging may prove unreliable. > ----- Backtrace ----- > 0x13147fb ??? > 0x1777763 ??? > 0x1777593 ??? > 0x1c09bbf ??? > 0x1733af7 ??? > 0x143c9d3 ??? > 0x1707483 ??? > 0x13468db ??? > 0x17405a3 ??? > 0x1505f7b ??? > 0x1504bd7 ??? > 0x150352f ??? > 0x1226b47 ??? > 0x2ee9ca33 __libc_start1 > /home/pkgbuild/worktrees/main/lib/libc/csu/libc_start1.c:172 > 0x12261cb ??? > --------------------- > /wrkdirs/usr/ports/devel/gdb/work-py311/gdb-15.1/gdb/thread.c:1423: = internal-error: switch_to_thread: Assertion `thr !=3D NULL' failed. > A problem internal to GDB has been detected, > further debugging may prove unreliable. > Quit this debugging session? (y or n) y > . . . >=20 > You may have to be more explicit about the specific of the > problem(s) you are seeing. OK. Here is what I am referring to: tuexen@head:~ % sudo kgdb -c /var/crash/vmcore.last /boot/kernel/kernel=20= Password: GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD] Copyright (C) 2024 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later = This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "aarch64-portbld-freebsd15.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /boot/kernel/kernel... Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug... Unread portion of the kernel message buffer: panic: tcp_do_segment: sent too much cpuid =3D 1 time =3D 1730910226 KDB: stack backtrace: db_trace_self() at db_trace_self db_trace_self_wrapper() at db_trace_self_wrapper+0x38 vpanic() at vpanic+0x1a0 panic() at panic+0x48 tcp_do_segment() at tcp_do_segment+0x2794 tcp_input_with_port() at tcp_input_with_port+0xcbc tcp_input() at tcp_input+0x10 ip_input() at ip_input+0x35c netisr_dispatch_src() at netisr_dispatch_src+0xd8 tunwrite() at tunwrite+0x2a8 devfs_write_f() at devfs_write_f+0x108 dofilewrite() at dofilewrite+0x7c kern_writev() at kern_writev+0x4c sys_writev() at sys_writev+0x40 do_el0_sync() at do_el0_sync+0x60c handle_el0_sync() at handle_el0_sync+0x4c --- exception, esr 0x56000000 KDB: enter: panic Reading symbols from /boot/kernel/tcp_rack.ko... Reading symbols from /usr/lib/debug//boot/kernel/tcp_rack.ko.debug... Reading symbols from /boot/kernel/tcphpts.ko... Reading symbols from /usr/lib/debug//boot/kernel/tcphpts.ko.debug... Reading symbols from /boot/kernel/if_bridge.ko... Reading symbols from /usr/lib/debug//boot/kernel/if_bridge.ko.debug... Reading symbols from /boot/kernel/bridgestp.ko... Reading symbols from /usr/lib/debug//boot/kernel/bridgestp.ko.debug... Reading symbols from /boot/kernel/uhid.ko... Reading symbols from /usr/lib/debug//boot/kernel/uhid.ko.debug... Reading symbols from /boot/kernel/wmt.ko... Reading symbols from /usr/lib/debug//boot/kernel/wmt.ko.debug... Reading symbols from /boot/kernel/cc_newreno.ko... Reading symbols from /usr/lib/debug//boot/kernel/cc_newreno.ko.debug... 0xffff0000004b5644 in doadump (textdump=3D0) at = /usr/home/tuexen/freebsd-src/sys/kern/kern_shutdown.c:404 404 dump_savectx(); (kgdb) up #1 0x3fdb0000000e99f0 in ?? () (kgdb) Initial frame selected; you cannot go up. (kgdb) Initial frame selected; you cannot go up. (kgdb) quit tuexen@head:~ % pkg info gdb gdb-15.1 Name : gdb Version : 15.1 Installed on : Thu Oct 24 10:32:17 2024 CEST Origin : devel/gdb Architecture : FreeBSD:15:aarch64 Prefix : /usr/local Categories : devel Licenses : GPLv3 Maintainer : pizzamig@FreeBSD.org WWW : https://www.gnu.org/software/gdb/ Comment : GNU Project Debugger Options : BUNDLED_READLINE: off BUNDLED_ZLIB : off DEBUGINFOD : off GDB_LINK : on GUILE : off KGDB : on NLS : on PORT_ICONV : on PORT_READLINE : on PYTHON : on SOURCE_HIGHLIGHT: on SYSTEM_ICONV : off SYSTEM_ZLIB : on TUI : on XXHASH : on Shared Libs required: libzstd.so.1 libxxhash.so.0 libsource-highlight.so.4 libreadline.so.8 libpython3.11.so.1.0 libmpfr.so.6 libintl.so.8 libiconv.so.2 libgmp.so.10 libexpat.so.1 libboost_regex.so.1.85.0 Annotations : FreeBSD_version: 1500025 build_timestamp: 2024-10-16T20:27:27+0000 built_by : poudriere-git-3.4.2 cpe : cpe:2.3:a:gnu:gdb:15.1:::::freebsd15:aarch64 flavor : py311 port_checkout_unclean: no port_git_hash : 82beca9e630 ports_top_checkout_unclean: no ports_top_git_hash: 94c4ac6b071 repo_type : binary repository : FreeBSD Flat size : 58.5MiB Description : GDB is a source-level debugger for Ada, C, C++, Objective-C, Pascal and many other languages. GDB can target (i.e., debug programs running on) more than a dozen different processor architectures, and GDB itself can run on most popular GNU/Linux, Unix and Microsoft Windows variants. tuexen@head:~ %=20 Using kgdb from "pkg install gdb" and locally built world and kernel. Best regards Michael >=20 > For reference: >=20 > # uname -apKU > FreeBSD aarch64-main-PBsmallUFS 15.0-CURRENT FreeBSD 15.0-CURRENT = main-n273174-8b2e7da70855 GENERIC arm64 aarch64 1500026 1500026 >=20 > # ~/pkgbase-snapshot-list.sh=20 > Via pkg-static info -C -x '^FreeBSD-' . . . > 352 FreeBSD-*-15.snap20241023235252 > 1 FreeBSD-*-15.snap20241022122410 > 1 FreeBSD-*-15.snap20241020224518 > 1 FreeBSD-*-15.snap20241020094723 > 7 FreeBSD-*-15.snap20241015203742 > 1 FreeBSD-*-15.snap20241015145839 > 1 FreeBSD-*-15.snap20241015120827 > 1 FreeBSD-*-15.snap20241014101436 > 34 FreeBSD-*-15.snap20241011184813 > 129 FreeBSD-*-15.snap20241009162208 > Instead via /var/cache/pkg/*.snap*.pkg . . . > 352 FreeBSD-*-15.snap20241023235252 > 1 FreeBSD-*-15.snap20241022122410 > 1 FreeBSD-*-15.snap20241020224518 > 1 FreeBSD-*-15.snap20241020094723 > 7 FreeBSD-*-15.snap20241015203742 > 1 FreeBSD-*-15.snap20241015145839 > 1 FreeBSD-*-15.snap20241015120827 > 1 FreeBSD-*-15.snap20241014101436 > 34 FreeBSD-*-15.snap20241011184813 > 129 FreeBSD-*-15.snap20241009162208 >=20 > # pkg info gdb > gdb-15.1 > Name : gdb > Version : 15.1 > Installed on : Mon Oct 14 10:30:51 2024 PDT > Origin : devel/gdb > Architecture : FreeBSD:15:aarch64 > Prefix : /usr/local > Categories : devel > Licenses : GPLv3 > Maintainer : pizzamig@FreeBSD.org > WWW : https://www.gnu.org/software/gdb/ > Comment : GNU Project Debugger > Options : > BUNDLED_READLINE: off > BUNDLED_ZLIB : off > DEBUGINFOD : off > GDB_LINK : on > GUILE : off > KGDB : on > NLS : on > PORT_ICONV : on > PORT_READLINE : on > PYTHON : on > SOURCE_HIGHLIGHT: on > SYSTEM_ICONV : off > SYSTEM_ZLIB : on > TUI : on > XXHASH : on > Shared Libs required: > libzstd.so.1 > libxxhash.so.0 > libsource-highlight.so.4 > libreadline.so.8 > libpython3.11.so.1.0 > libmpfr.so.6 > libintl.so.8 > libiconv.so.2 > libgmp.so.10 > libexpat.so.1 > libboost_regex.so.1.85.0 > Annotations : > FreeBSD_version: 1500024 > build_timestamp: 2024-10-07T19:01:59+0000 > built_by : poudriere-git-3.4.2 > cpe : cpe:2.3:a:gnu:gdb:15.1:::::freebsd15:aarch64 > flavor : py311 > port_checkout_unclean: no > port_git_hash : 82beca9e630 > ports_top_checkout_unclean: no > ports_top_git_hash: 149fe86b8e7 > repo_type : binary > repository : FreeBSD > Flat size : 58.5MiB > Description : > GDB is a source-level debugger for Ada, C, C++, Objective-C, Pascal = and > many other languages. GDB can target (i.e., debug programs running = on) > more than a dozen different processor architectures, and GDB itself = can > run on most popular GNU/Linux, Unix and Microsoft Windows variants. >=20 > It was from installing an officially built package. >=20 >=20 > =3D=3D=3D > Mark Millard > marklmi at yahoo.com >=20