From nobody Wed Nov 06 22:24:15 2024 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XkKWK2BKyz5bvn1 for ; Wed, 06 Nov 2024 22:24:29 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from drew.franken.de (mail-n.franken.de [193.175.24.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "Certum Domain Validation CA SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XkKWK0qYQz4Fpr for ; Wed, 6 Nov 2024 22:24:29 +0000 (UTC) (envelope-from tuexen@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (unknown [IPv6:2001:67c:1232:144:44b0:da02:1d6e:fd0a]) (Authenticated sender: micmac) by drew.franken.de (Postfix) with ESMTPSA id 4BF99721E283A; Wed, 6 Nov 2024 23:24:16 +0100 (CET) Content-Type: text/plain; charset=utf-8 List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.200.121\)) Subject: Re: kernel core debugging From: tuexen@freebsd.org In-Reply-To: Date: Wed, 6 Nov 2024 22:24:15 +0000 Cc: Mark Millard , FreeBSD ARM List Content-Transfer-Encoding: quoted-printable Message-Id: <4B430EC2-C328-4FAB-BEBF-CA21D208F78C@freebsd.org> References: <34E3792D-E464-45A8-938A-AD266F2EB49F@freebsd.org> <92795A92-653D-4DED-ADB2-3BAAF181B813@freebsd.org> <078E5E0E-E90E-48D3-BED0-66C21343DBF5@yahoo.com> To: Bakul Shah X-Mailer: Apple Mail (2.3826.200.121) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:680, ipnet:193.174.0.0/15, country:DE] X-Rspamd-Queue-Id: 4XkKWK0qYQz4Fpr X-Spamd-Bar: ---- > On 6. Nov 2024, at 19:27, Bakul Shah wrote: >=20 > My guess: either the core and the kernel don't match or the core is = corrupted. I'm running a test script resulting in the panic. Doing the same on an amd64 box results in kgdb being usable to debug the problem. Just want to use the arm64 box for it... Best regards Michael >=20 >> On Nov 6, 2024, at 11:19=E2=80=AFAM, Mark Millard = wrote: >>=20 >> On Nov 6, 2024, at 10:07, tuexen@freebsd.org = wrote: >>>=20 >>>> On 6. Nov 2024, at 17:51, Mark Millard wrote: >>>>=20 >>>>=20 >>>>=20 >>>> On Nov 6, 2024, at 09:28, tuexen@freebsd.org = wrote: >>>>=20 >>>>>> On 6. Nov 2024, at 15:12, Mark Millard wrote: >>>>>>=20 >>>>>> On Nov 6, 2024, at 01:44, tuexen@freebsd.org = wrote: >>>>>>=20 >>>>>>> is debugging a kernel panic by using kgdb or lldb on a core file >>>>>>> supposed to work? At least it is not right now for me... >>>>>>=20 >>>>>> # kgdb /boot/kernel.GENERIC-NODEBUG/kernel /var/crash/vmcore.2 >>>>>> GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD] >>>>>> Copyright (C) 2024 Free Software Foundation, Inc. >>>>>> License GPLv3+: GNU GPL version 3 or later = >>>>>> This is free software: you are free to change and redistribute = it. >>>>>> There is NO WARRANTY, to the extent permitted by law. >>>>>> Type "show copying" and "show warranty" for details. >>>>>> This GDB was configured as "aarch64-portbld-freebsd15.0". >>>>>> Type "show configuration" for configuration details. >>>>>> For bug reporting instructions, please see: >>>>>> . >>>>>> Find the GDB manual and other documentation resources online at: >>>>>> . >>>>>>=20 >>>>>> For help, type "help". >>>>>> Type "apropos word" to search for commands related to "word"... >>>>>> Reading symbols from /boot/kernel.GENERIC-NODEBUG/kernel... >>>>>> Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/kernel.debug... >>>>>>=20 >>>>>> Unread portion of the kernel message buffer: >>>>>> KDB: enter: manual escape to debugger >>>>>>=20 >>>>>> Reading symbols from /boot/kernel.GENERIC-NODEBUG/uhid.ko... >>>>>> Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/uhid.ko.debug... >>>>>> Reading symbols from /boot/kernel.GENERIC-NODEBUG/wmt.ko... >>>>>> Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/wmt.ko.debug... >>>>>> Reading symbols from /boot/kernel.GENERIC-NODEBUG/ums.ko... >>>>>> Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/ums.ko.debug... >>>>>> Reading symbols from /boot/kernel.GENERIC-NODEBUG/zfs.ko... >>>>>> Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/zfs.ko.debug... >>>>>> 0xffff00000050f3f0 in doadump (textdump=3D0, = textdump@entry=3D212431136) at = /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c:404 >>>>>> warning: 404 = /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c: No such file or = directory >>>>>> (kgdb) bt >>>>>> #0 0xffff00000050f3f0 in doadump (textdump=3D0, = textdump@entry=3D212431136) at = /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c:404 >>>>>> #1 0xffff0000000ee6a8 in db_dump (dummy=3D, = dummy2=3D, dummy3=3D, dummy4=3D) at /home/pkgbuild/worktrees/main/sys/ddb/db_command.c:596 >>>>>> #2 0xffff0000000ee478 in db_command (last_cmdp=3D, cmd_table=3D, dopager=3Dtrue) at = /home/pkgbuild/worktrees/main/sys/ddb/db_command.c:508 >>>>>> #3 0xffff0000000ee150 in db_command_loop () at = /home/pkgbuild/worktrees/main/sys/ddb/db_command.c:555 >>>>>> #4 0xffff0000000f1ff4 in db_trap (type=3D, = code=3D) at = /home/pkgbuild/worktrees/main/sys/ddb/db_main.c:267 >>>>>> #5 0xffff000000568b0c in kdb_trap (type=3D60, code=3D0, = tf=3D) at = /home/pkgbuild/worktrees/main/sys/kern/subr_kdb.c:790 >>>>>> #6 >>>>>> #7 kdb_enter (why=3D, msg=3D) at = /home/pkgbuild/worktrees/main/sys/kern/subr_kdb.c:556 >>>>>> #8 0xffff0000003625cc in vt_machine_kbdevent (vd=3D, c=3D) at = /home/pkgbuild/worktrees/main/sys/dev/vt/vt_core.c:761 >>>>>> #9 vt_processkey (kbd=3D0xffffa000803caa80, = vd=3D0xffff000000d24360 , c=3D-2147483514) at = /home/pkgbuild/worktrees/main/sys/dev/vt/vt_core.c:903 >>>>>> #10 vt_kbdevent (kbd=3D0xffffa000803caa80, event=3D, arg=3D0xffff000000d24360 ) at = /home/pkgbuild/worktrees/main/sys/dev/vt/vt_core.c:1030 >>>>>> #11 0xffff0000001ea048 in kbdmux_intr (kbd=3D0xffffa000803caa80, = arg=3D) at = /home/pkgbuild/worktrees/main/sys/dev/kbdmux/kbdmux.c:565 >>>>>> #12 0xffff0000005839ac in taskqueue_run_locked = (queue=3Dqueue@entry=3D0xffffa000803c9c00) at = /home/pkgbuild/worktrees/main/sys/kern/subr_taskqueue.c:517 >>>>>> #13 0xffff000000583714 in taskqueue_run = (queue=3D0xffffa000803c9c00) at = /home/pkgbuild/worktrees/main/sys/kern/subr_taskqueue.c:532 >>>>>> #14 0xffff0000004bc114 in intr_event_execute_handlers = (ie=3D0xffffa0008028ec00, p=3D) at = /home/pkgbuild/worktrees/main/sys/kern/kern_intr.c:1183 >>>>>> #15 ithread_execute_handlers (ie=3D0xffffa0008028ec00, = p=3D) at = /home/pkgbuild/worktrees/main/sys/kern/kern_intr.c:1196 >>>>>> #16 ithread_loop (arg=3D, = arg@entry=3D0xffffa000803de5a0) at = /home/pkgbuild/worktrees/main/sys/kern/kern_intr.c:1289 >>>>>> #17 0xffff0000004b700c in fork_exit (callout=3D0xffff0000004bbd78 = , arg=3D0xffffa000803de5a0, frame=3D0xffff00010ca97a00) at = /home/pkgbuild/worktrees/main/sys/kern/kern_fork.c:1151 >>>>>> #18 >>>>>>=20 >>>>>> The context here was from an official PkgBase kernel and world >>>>>> installation. >>>>>>=20 >>>>>> . . . (deletion) . . . >>>>>>=20 >>>>>> You may have to be more explicit about the specific of the >>>>>> problem(s) you are seeing. >>>>> OK. Here is what I am referring to: >>>>>=20 >>>>> tuexen@head:~ % sudo kgdb -c /var/crash/vmcore.last = /boot/kernel/kernel >>>>=20 >>>> That command does not match the parameter order in the man page or >>>> in my example. >>>>=20 >>>> man kgdb output shows kernel first, then core: >>>>=20 >>>> SYNOPSIS >>>> kgdb [-a | -f | -fullname] [-b rate] [-q | -quiet] [-v] [-w] >>>> [-d crashdir] [-c core | -n dumpnr | -r device] [kernel [core]] >>>>=20 >>>> My example: # kgdb /boot/kernel.GENERIC-NODEBUG/kernel = /var/crash/vmcore.2 >>>>=20 >>>> You might want to see if using the other order makes a difference. >>> No it doesn't. I'm specifying the core via the -c core option = instead of=20 >>> the second argument... >>=20 >> Of course. Sorry for the noise. (Does not look like this is >> going to be one of my better mornings.) >>=20 >> I guess about all we learn is that your issue is somehow more >> specific to your context rather than it being an example of >> kgdb being generally broken. >>=20 >>> Best regards >>> Michael >>>>=20 >>>>> Password: >>>>> GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD] >>>>> Copyright (C) 2024 Free Software Foundation, Inc. >>>>> License GPLv3+: GNU GPL version 3 or later = >>>>> This is free software: you are free to change and redistribute it. >>>>> There is NO WARRANTY, to the extent permitted by law. >>>>> Type "show copying" and "show warranty" for details. >>>>> This GDB was configured as "aarch64-portbld-freebsd15.0". >>>>> Type "show configuration" for configuration details. >>>>> For bug reporting instructions, please see: >>>>> . >>>>> Find the GDB manual and other documentation resources online at: >>>>> . >>>>>=20 >>>>> For help, type "help". >>>>> Type "apropos word" to search for commands related to "word"... >>>>> Reading symbols from /boot/kernel/kernel... >>>>> Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug... >>>>>=20 >>>>> Unread portion of the kernel message buffer: >>>>> panic: tcp_do_segment: sent too much >>>>> cpuid =3D 1 >>>>> time =3D 1730910226 >>>>> KDB: stack backtrace: >>>>> db_trace_self() at db_trace_self >>>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x38 >>>>> vpanic() at vpanic+0x1a0 >>>>> panic() at panic+0x48 >>>>> tcp_do_segment() at tcp_do_segment+0x2794 >>>>> tcp_input_with_port() at tcp_input_with_port+0xcbc >>>>> tcp_input() at tcp_input+0x10 >>>>> ip_input() at ip_input+0x35c >>>>> netisr_dispatch_src() at netisr_dispatch_src+0xd8 >>>>> tunwrite() at tunwrite+0x2a8 >>>>> devfs_write_f() at devfs_write_f+0x108 >>>>> dofilewrite() at dofilewrite+0x7c >>>>> kern_writev() at kern_writev+0x4c >>>>> sys_writev() at sys_writev+0x40 >>>>> do_el0_sync() at do_el0_sync+0x60c >>>>> handle_el0_sync() at handle_el0_sync+0x4c >>>>> --- exception, esr 0x56000000 >>>>> KDB: enter: panic >>>>>=20 >>>>> Reading symbols from /boot/kernel/tcp_rack.ko... >>>>> Reading symbols from = /usr/lib/debug//boot/kernel/tcp_rack.ko.debug... >>>>> Reading symbols from /boot/kernel/tcphpts.ko... >>>>> Reading symbols from = /usr/lib/debug//boot/kernel/tcphpts.ko.debug... >>>>> Reading symbols from /boot/kernel/if_bridge.ko... >>>>> Reading symbols from = /usr/lib/debug//boot/kernel/if_bridge.ko.debug... >>>>> Reading symbols from /boot/kernel/bridgestp.ko... >>>>> Reading symbols from = /usr/lib/debug//boot/kernel/bridgestp.ko.debug... >>>>> Reading symbols from /boot/kernel/uhid.ko... >>>>> Reading symbols from /usr/lib/debug//boot/kernel/uhid.ko.debug... >>>>> Reading symbols from /boot/kernel/wmt.ko... >>>>> Reading symbols from /usr/lib/debug//boot/kernel/wmt.ko.debug... >>>>> Reading symbols from /boot/kernel/cc_newreno.ko... >>>>> Reading symbols from = /usr/lib/debug//boot/kernel/cc_newreno.ko.debug... >>>>> 0xffff0000004b5644 in doadump (textdump=3D0) at = /usr/home/tuexen/freebsd-src/sys/kern/kern_shutdown.c:404 >>>>> 404 dump_savectx(); >>>>> (kgdb) up >>>>> #1 0x3fdb0000000e99f0 in ?? () >>=20 >> Somehow it went from referencing the apparently correct/expected >> 0xffff0000004b5644 (doadump) to referencing 0x3fdb0000000e99f0 . >> There is also the odd "404 dump_savectx();". >>=20 >> If bt is used as the first command at the prompt, what does it >> show for the backtrace? Anything interesting? Just #0 for >> 0xffff0000004b5644 and #1 for 0x3fdb0000000e99f0 ? >>=20 >>=20 >> My doadump line also has ", textdump@entry=3D212431136": >>=20 >> 0xffff00000050f3f0 in doadump (textdump=3D0, = textdump@entry=3D212431136) at = /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c:404 >>=20 >> But I'm not aware of the PkbBase build configuration information >> being published for making comparisons with. >>=20 >>=20 >>>>> (kgdb) Initial frame selected; you cannot go up. >>>>> (kgdb) Initial frame selected; you cannot go up. >>>>> (kgdb) quit >>>>> tuexen@head:~ % pkg info gdb gdb-15.1 >>>>> Name : gdb >>>>> Version : 15.1 >>>>> Installed on : Thu Oct 24 10:32:17 2024 CEST >>>>> Origin : devel/gdb >>>>> Architecture : FreeBSD:15:aarch64 >>>>> Prefix : /usr/local >>>>> Categories : devel >>>>> Licenses : GPLv3 >>>>> Maintainer : pizzamig@FreeBSD.org >>>>> WWW : https://www.gnu.org/software/gdb/ >>>>> Comment : GNU Project Debugger >>>>> Options : >>>>> BUNDLED_READLINE: off >>>>> BUNDLED_ZLIB : off >>>>> DEBUGINFOD : off >>>>> GDB_LINK : on >>>>> GUILE : off >>>>> KGDB : on >>>>> NLS : on >>>>> PORT_ICONV : on >>>>> PORT_READLINE : on >>>>> PYTHON : on >>>>> SOURCE_HIGHLIGHT: on >>>>> SYSTEM_ICONV : off >>>>> SYSTEM_ZLIB : on >>>>> TUI : on >>>>> XXHASH : on >>>>> Shared Libs required: >>>>> libzstd.so.1 >>>>> libxxhash.so.0 >>>>> libsource-highlight.so.4 >>>>> libreadline.so.8 >>>>> libpython3.11.so.1.0 >>>>> libmpfr.so.6 >>>>> libintl.so.8 >>>>> libiconv.so.2 >>>>> libgmp.so.10 >>>>> libexpat.so.1 >>>>> libboost_regex.so.1.85.0 >>>>> Annotations : >>>>> FreeBSD_version: 1500025 >>>>> build_timestamp: 2024-10-16T20:27:27+0000 >>>>> built_by : poudriere-git-3.4.2 >>>>> cpe : cpe:2.3:a:gnu:gdb:15.1:::::freebsd15:aarch64 >>>>> flavor : py311 >>>>> port_checkout_unclean: no >>>>> port_git_hash : 82beca9e630 >>>>> ports_top_checkout_unclean: no >>>>> ports_top_git_hash: 94c4ac6b071 >>>>> repo_type : binary >>>>> repository : FreeBSD >>>>> Flat size : 58.5MiB >>>>> Description : >>>>> GDB is a source-level debugger for Ada, C, C++, Objective-C, = Pascal and >>>>> many other languages. GDB can target (i.e., debug programs = running on) >>>>> more than a dozen different processor architectures, and GDB = itself can >>>>> run on most popular GNU/Linux, Unix and Microsoft Windows = variants. >>=20 >> Same gdb package version installation as in my context. >> kgdb, of itself, should not be a source of the behavior >> differences. >>=20 >>>>> tuexen@head:~ %=20 >>>>>=20 >>>>> Using kgdb from "pkg install gdb" and locally built world and = kernel. >>>>>=20 >>>>> Best regards >>>>> Michael >>>>>>=20 >>>>>> For reference: >>>>>>=20 >>>>>> . . . (deletion) . . . >>>=20 >>=20 >> I'm not identifying anything else to investigate. >>=20 >>=20 >> =3D=3D=3D >> Mark Millard >> marklmi at yahoo.com >>=20 >>=20 >=20