From nobody Wed Nov 06 18:07:06 2024 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XkCpc6qV0z1GnsS for ; Wed, 06 Nov 2024 18:07:20 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from drew.franken.de (mail-n.franken.de [193.175.24.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "Certum Domain Validation CA SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XkCpc45mSz4kxf for ; Wed, 6 Nov 2024 18:07:20 +0000 (UTC) (envelope-from tuexen@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (unknown [IPv6:2001:67c:370:128:c74:3c49:cab8:7b26]) (Authenticated sender: micmac) by drew.franken.de (Postfix) with ESMTPSA id 22E7D721BE00E; Wed, 6 Nov 2024 19:07:07 +0100 (CET) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.200.121\)) Subject: Re: kernel core debugging From: tuexen@freebsd.org In-Reply-To: <078E5E0E-E90E-48D3-BED0-66C21343DBF5@yahoo.com> Date: Wed, 6 Nov 2024 18:07:06 +0000 Cc: freebsd-arm Content-Transfer-Encoding: quoted-printable Message-Id: References: <34E3792D-E464-45A8-938A-AD266F2EB49F@freebsd.org> <92795A92-653D-4DED-ADB2-3BAAF181B813@freebsd.org> <078E5E0E-E90E-48D3-BED0-66C21343DBF5@yahoo.com> To: Mark Millard X-Mailer: Apple Mail (2.3826.200.121) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:680, ipnet:193.174.0.0/15, country:DE] X-Rspamd-Queue-Id: 4XkCpc45mSz4kxf X-Spamd-Bar: ---- > On 6. Nov 2024, at 17:51, Mark Millard wrote: >=20 >=20 >=20 > On Nov 6, 2024, at 09:28, tuexen@freebsd.org = wrote: >=20 >>> On 6. Nov 2024, at 15:12, Mark Millard wrote: >>>=20 >>> On Nov 6, 2024, at 01:44, tuexen@freebsd.org = wrote: >>>=20 >>>> is debugging a kernel panic by using kgdb or lldb on a core file >>>> supposed to work? At least it is not right now for me... >>>=20 >>> # kgdb /boot/kernel.GENERIC-NODEBUG/kernel /var/crash/vmcore.2 >>> GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD] >>> Copyright (C) 2024 Free Software Foundation, Inc. >>> License GPLv3+: GNU GPL version 3 or later = >>> This is free software: you are free to change and redistribute it. >>> There is NO WARRANTY, to the extent permitted by law. >>> Type "show copying" and "show warranty" for details. >>> This GDB was configured as "aarch64-portbld-freebsd15.0". >>> Type "show configuration" for configuration details. >>> For bug reporting instructions, please see: >>> . >>> Find the GDB manual and other documentation resources online at: >>> . >>>=20 >>> For help, type "help". >>> Type "apropos word" to search for commands related to "word"... >>> Reading symbols from /boot/kernel.GENERIC-NODEBUG/kernel... >>> Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/kernel.debug... >>>=20 >>> Unread portion of the kernel message buffer: >>> KDB: enter: manual escape to debugger >>>=20 >>> Reading symbols from /boot/kernel.GENERIC-NODEBUG/uhid.ko... >>> Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/uhid.ko.debug... >>> Reading symbols from /boot/kernel.GENERIC-NODEBUG/wmt.ko... >>> Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/wmt.ko.debug... >>> Reading symbols from /boot/kernel.GENERIC-NODEBUG/ums.ko... >>> Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/ums.ko.debug... >>> Reading symbols from /boot/kernel.GENERIC-NODEBUG/zfs.ko... >>> Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/zfs.ko.debug... >>> 0xffff00000050f3f0 in doadump (textdump=3D0, = textdump@entry=3D212431136) at = /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c:404 >>> warning: 404 /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c: = No such file or directory >>> (kgdb) bt >>> #0 0xffff00000050f3f0 in doadump (textdump=3D0, = textdump@entry=3D212431136) at = /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c:404 >>> #1 0xffff0000000ee6a8 in db_dump (dummy=3D, = dummy2=3D, dummy3=3D, dummy4=3D) at /home/pkgbuild/worktrees/main/sys/ddb/db_command.c:596 >>> #2 0xffff0000000ee478 in db_command (last_cmdp=3D, = cmd_table=3D, dopager=3Dtrue) at = /home/pkgbuild/worktrees/main/sys/ddb/db_command.c:508 >>> #3 0xffff0000000ee150 in db_command_loop () at = /home/pkgbuild/worktrees/main/sys/ddb/db_command.c:555 >>> #4 0xffff0000000f1ff4 in db_trap (type=3D, = code=3D) at = /home/pkgbuild/worktrees/main/sys/ddb/db_main.c:267 >>> #5 0xffff000000568b0c in kdb_trap (type=3D60, code=3D0, = tf=3D) at = /home/pkgbuild/worktrees/main/sys/kern/subr_kdb.c:790 >>> #6 >>> #7 kdb_enter (why=3D, msg=3D) at = /home/pkgbuild/worktrees/main/sys/kern/subr_kdb.c:556 >>> #8 0xffff0000003625cc in vt_machine_kbdevent (vd=3D, = c=3D) at = /home/pkgbuild/worktrees/main/sys/dev/vt/vt_core.c:761 >>> #9 vt_processkey (kbd=3D0xffffa000803caa80, vd=3D0xffff000000d24360 = , c=3D-2147483514) at = /home/pkgbuild/worktrees/main/sys/dev/vt/vt_core.c:903 >>> #10 vt_kbdevent (kbd=3D0xffffa000803caa80, event=3D, = arg=3D0xffff000000d24360 ) at = /home/pkgbuild/worktrees/main/sys/dev/vt/vt_core.c:1030 >>> #11 0xffff0000001ea048 in kbdmux_intr (kbd=3D0xffffa000803caa80, = arg=3D) at = /home/pkgbuild/worktrees/main/sys/dev/kbdmux/kbdmux.c:565 >>> #12 0xffff0000005839ac in taskqueue_run_locked = (queue=3Dqueue@entry=3D0xffffa000803c9c00) at = /home/pkgbuild/worktrees/main/sys/kern/subr_taskqueue.c:517 >>> #13 0xffff000000583714 in taskqueue_run (queue=3D0xffffa000803c9c00) = at /home/pkgbuild/worktrees/main/sys/kern/subr_taskqueue.c:532 >>> #14 0xffff0000004bc114 in intr_event_execute_handlers = (ie=3D0xffffa0008028ec00, p=3D) at = /home/pkgbuild/worktrees/main/sys/kern/kern_intr.c:1183 >>> #15 ithread_execute_handlers (ie=3D0xffffa0008028ec00, p=3D) at /home/pkgbuild/worktrees/main/sys/kern/kern_intr.c:1196 >>> #16 ithread_loop (arg=3D, = arg@entry=3D0xffffa000803de5a0) at = /home/pkgbuild/worktrees/main/sys/kern/kern_intr.c:1289 >>> #17 0xffff0000004b700c in fork_exit (callout=3D0xffff0000004bbd78 = , arg=3D0xffffa000803de5a0, frame=3D0xffff00010ca97a00) at = /home/pkgbuild/worktrees/main/sys/kern/kern_fork.c:1151 >>> #18 >>>=20 >>> The context here was from an official PkgBase kernel and world >>> installation. >>>=20 >>> . . . (deletion) . . . >>>=20 >>> You may have to be more explicit about the specific of the >>> problem(s) you are seeing. >> OK. Here is what I am referring to: >>=20 >> tuexen@head:~ % sudo kgdb -c /var/crash/vmcore.last = /boot/kernel/kernel >=20 > That command does not match the parameter order in the man page or > in my example. >=20 > man kgdb output shows kernel first, then core: >=20 > SYNOPSIS > kgdb [-a | -f | -fullname] [-b rate] [-q | -quiet] [-v] [-w] > [-d crashdir] [-c core | -n dumpnr | -r device] [kernel = [core]] >=20 > My example: # kgdb /boot/kernel.GENERIC-NODEBUG/kernel = /var/crash/vmcore.2 >=20 > You might want to see if using the other order makes a difference. No it doesn't. I'm specifying the core via the -c core option instead of=20= the second argument... Best regards Michael >=20 >> Password: >> GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD] >> Copyright (C) 2024 Free Software Foundation, Inc. >> License GPLv3+: GNU GPL version 3 or later = >> This is free software: you are free to change and redistribute it. >> There is NO WARRANTY, to the extent permitted by law. >> Type "show copying" and "show warranty" for details. >> This GDB was configured as "aarch64-portbld-freebsd15.0". >> Type "show configuration" for configuration details. >> For bug reporting instructions, please see: >> . >> Find the GDB manual and other documentation resources online at: >> . >>=20 >> For help, type "help". >> Type "apropos word" to search for commands related to "word"... >> Reading symbols from /boot/kernel/kernel... >> Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug... >>=20 >> Unread portion of the kernel message buffer: >> panic: tcp_do_segment: sent too much >> cpuid =3D 1 >> time =3D 1730910226 >> KDB: stack backtrace: >> db_trace_self() at db_trace_self >> db_trace_self_wrapper() at db_trace_self_wrapper+0x38 >> vpanic() at vpanic+0x1a0 >> panic() at panic+0x48 >> tcp_do_segment() at tcp_do_segment+0x2794 >> tcp_input_with_port() at tcp_input_with_port+0xcbc >> tcp_input() at tcp_input+0x10 >> ip_input() at ip_input+0x35c >> netisr_dispatch_src() at netisr_dispatch_src+0xd8 >> tunwrite() at tunwrite+0x2a8 >> devfs_write_f() at devfs_write_f+0x108 >> dofilewrite() at dofilewrite+0x7c >> kern_writev() at kern_writev+0x4c >> sys_writev() at sys_writev+0x40 >> do_el0_sync() at do_el0_sync+0x60c >> handle_el0_sync() at handle_el0_sync+0x4c >> --- exception, esr 0x56000000 >> KDB: enter: panic >>=20 >> Reading symbols from /boot/kernel/tcp_rack.ko... >> Reading symbols from /usr/lib/debug//boot/kernel/tcp_rack.ko.debug... >> Reading symbols from /boot/kernel/tcphpts.ko... >> Reading symbols from /usr/lib/debug//boot/kernel/tcphpts.ko.debug... >> Reading symbols from /boot/kernel/if_bridge.ko... >> Reading symbols from = /usr/lib/debug//boot/kernel/if_bridge.ko.debug... >> Reading symbols from /boot/kernel/bridgestp.ko... >> Reading symbols from = /usr/lib/debug//boot/kernel/bridgestp.ko.debug... >> Reading symbols from /boot/kernel/uhid.ko... >> Reading symbols from /usr/lib/debug//boot/kernel/uhid.ko.debug... >> Reading symbols from /boot/kernel/wmt.ko... >> Reading symbols from /usr/lib/debug//boot/kernel/wmt.ko.debug... >> Reading symbols from /boot/kernel/cc_newreno.ko... >> Reading symbols from = /usr/lib/debug//boot/kernel/cc_newreno.ko.debug... >> 0xffff0000004b5644 in doadump (textdump=3D0) at = /usr/home/tuexen/freebsd-src/sys/kern/kern_shutdown.c:404 >> 404 dump_savectx(); >> (kgdb) up >> #1 0x3fdb0000000e99f0 in ?? () >> (kgdb) Initial frame selected; you cannot go up. >> (kgdb) Initial frame selected; you cannot go up. >> (kgdb) quit >> tuexen@head:~ % pkg info gdb gdb-15.1 >> Name : gdb >> Version : 15.1 >> Installed on : Thu Oct 24 10:32:17 2024 CEST >> Origin : devel/gdb >> Architecture : FreeBSD:15:aarch64 >> Prefix : /usr/local >> Categories : devel >> Licenses : GPLv3 >> Maintainer : pizzamig@FreeBSD.org >> WWW : https://www.gnu.org/software/gdb/ >> Comment : GNU Project Debugger >> Options : >> BUNDLED_READLINE: off >> BUNDLED_ZLIB : off >> DEBUGINFOD : off >> GDB_LINK : on >> GUILE : off >> KGDB : on >> NLS : on >> PORT_ICONV : on >> PORT_READLINE : on >> PYTHON : on >> SOURCE_HIGHLIGHT: on >> SYSTEM_ICONV : off >> SYSTEM_ZLIB : on >> TUI : on >> XXHASH : on >> Shared Libs required: >> libzstd.so.1 >> libxxhash.so.0 >> libsource-highlight.so.4 >> libreadline.so.8 >> libpython3.11.so.1.0 >> libmpfr.so.6 >> libintl.so.8 >> libiconv.so.2 >> libgmp.so.10 >> libexpat.so.1 >> libboost_regex.so.1.85.0 >> Annotations : >> FreeBSD_version: 1500025 >> build_timestamp: 2024-10-16T20:27:27+0000 >> built_by : poudriere-git-3.4.2 >> cpe : cpe:2.3:a:gnu:gdb:15.1:::::freebsd15:aarch64 >> flavor : py311 >> port_checkout_unclean: no >> port_git_hash : 82beca9e630 >> ports_top_checkout_unclean: no >> ports_top_git_hash: 94c4ac6b071 >> repo_type : binary >> repository : FreeBSD >> Flat size : 58.5MiB >> Description : >> GDB is a source-level debugger for Ada, C, C++, Objective-C, Pascal = and >> many other languages. GDB can target (i.e., debug programs running = on) >> more than a dozen different processor architectures, and GDB itself = can >> run on most popular GNU/Linux, Unix and Microsoft Windows variants. >> tuexen@head:~ %=20 >>=20 >> Using kgdb from "pkg install gdb" and locally built world and kernel. >>=20 >> Best regards >> Michael >>>=20 >>> For reference: >>>=20 >>> . . . (deletion) . . . >=20 >=20 >=20 > =3D=3D=3D > Mark Millard > marklmi at yahoo.com >=20