From nobody Wed Nov 06 19:27:43 2024 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XkFbc4S64z1GtCZ for ; Wed, 06 Nov 2024 19:27:56 +0000 (UTC) (envelope-from bakul@iitbombay.org) Received: from mail-pg1-x52a.google.com (mail-pg1-x52a.google.com [IPv6:2607:f8b0:4864:20::52a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XkFbc1xqwz4sKd for ; Wed, 6 Nov 2024 19:27:56 +0000 (UTC) (envelope-from bakul@iitbombay.org) Authentication-Results: mx1.freebsd.org; none Received: by mail-pg1-x52a.google.com with SMTP id 41be03b00d2f7-7e9f377a3c9so21937a12.3 for ; Wed, 06 Nov 2024 11:27:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iitbombay.org; s=google; t=1730921275; x=1731526075; darn=freebsd.org; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=FWT70RRgBvGyQj89HROidyyRzn6V52+tFgv2tEGdLYA=; b=W9FnLZMKZoTa7BP7+4DX1ycttCT73Lfl1IQPXFMoQ9meHPWo7+O5MJRwxfNSw1NhPR G7QZ9ZfOfRiYs1gtP/rmwWpg5h5Oe2DPqMtoO7bXczFmpsMd60pBUsjm/WItjK2HiHXI 9hxhx4zEagZkGHq1MjnpSUiBaoJkmGp3vmxAI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730921275; x=1731526075; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FWT70RRgBvGyQj89HROidyyRzn6V52+tFgv2tEGdLYA=; b=gyaHzbY0d8749OT4989TRcUWCdQA09fk29rxsqW53Ws7mePweZQObtOkMvQ2oi9fwl 5gXpcGWszeRTrD1tXS2pS8IyYsAwW9KneebhaEfhABUU91gT7lYS0tF5EQ75pMANt2bF ntSXR5jkRiYx0665E3CspK9eBMmBa9XXpgSYIkjoHilHWxWCvvPD4T9dv55i4eBd+VlD Te8PXOIovbHaWG9bXy0t/VB8zLkqmf6uVSzpzomP0DuAYyIzVguICgK/HSG94n7UXQKl ch5TH0QoemIun5W+ZIuhq4ROgzirxujyeLxdqU/DQYRouFa024IHGsm4zDW8Fq1JdemM bWww== X-Forwarded-Encrypted: i=1; AJvYcCVmzrlDPW3HB9wu1K/9cWzt6lHXZ2Q7J3gxALnAnDyFq9jZ637xuhrCDVLc8Rvzt2qXNzcv4xkI7TX9cQ==@freebsd.org X-Gm-Message-State: AOJu0YzlVpAqFm0KxEp6ej7dwHQmyWbmPkQbNB9hcDASLR7n+JK9JfCi D/BTQuMJ/1Ciu5+N+PvypUs+lUSB3Qz2ToJPz2N0TnGBRXWIbGk6bMNN3prP/A== X-Google-Smtp-Source: AGHT+IEVTs9aUHPEJxSxh5TBGtW005RKU589LJxlwmzeERipi9/Ph9HZ7ACCZCfiekm0c4p+d36xJg== X-Received: by 2002:a05:6a20:2452:b0:1cf:3f39:c46a with SMTP id adf61e73a8af0-1dc0fac9fbbmr446303637.6.1730921274717; Wed, 06 Nov 2024 11:27:54 -0800 (PST) Received: from smtpclient.apple (107-215-223-229.lightspeed.sntcca.sbcglobal.net. [107.215.223.229]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7ee45a0eda2sm11683517a12.86.2024.11.06.11.27.53 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Nov 2024 11:27:54 -0800 (PST) Content-Type: text/plain; charset=utf-8 List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.200.121\)) Subject: Re: kernel core debugging From: Bakul Shah In-Reply-To: Date: Wed, 6 Nov 2024 11:27:43 -0800 Cc: "tuexen@freebsd.org" , FreeBSD ARM List Content-Transfer-Encoding: quoted-printable Message-Id: References: <34E3792D-E464-45A8-938A-AD266F2EB49F@freebsd.org> <92795A92-653D-4DED-ADB2-3BAAF181B813@freebsd.org> <078E5E0E-E90E-48D3-BED0-66C21343DBF5@yahoo.com> To: Mark Millard X-Mailer: Apple Mail (2.3826.200.121) X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4XkFbc1xqwz4sKd X-Spamd-Bar: ---- My guess: either the core and the kernel don't match or the core is = corrupted. > On Nov 6, 2024, at 11:19=E2=80=AFAM, Mark Millard = wrote: >=20 > On Nov 6, 2024, at 10:07, tuexen@freebsd.org = wrote: >>=20 >>> On 6. Nov 2024, at 17:51, Mark Millard wrote: >>>=20 >>>=20 >>>=20 >>> On Nov 6, 2024, at 09:28, tuexen@freebsd.org = wrote: >>>=20 >>>>> On 6. Nov 2024, at 15:12, Mark Millard wrote: >>>>>=20 >>>>> On Nov 6, 2024, at 01:44, tuexen@freebsd.org = wrote: >>>>>=20 >>>>>> is debugging a kernel panic by using kgdb or lldb on a core file >>>>>> supposed to work? At least it is not right now for me... >>>>>=20 >>>>> # kgdb /boot/kernel.GENERIC-NODEBUG/kernel /var/crash/vmcore.2 >>>>> GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD] >>>>> Copyright (C) 2024 Free Software Foundation, Inc. >>>>> License GPLv3+: GNU GPL version 3 or later = >>>>> This is free software: you are free to change and redistribute it. >>>>> There is NO WARRANTY, to the extent permitted by law. >>>>> Type "show copying" and "show warranty" for details. >>>>> This GDB was configured as "aarch64-portbld-freebsd15.0". >>>>> Type "show configuration" for configuration details. >>>>> For bug reporting instructions, please see: >>>>> . >>>>> Find the GDB manual and other documentation resources online at: >>>>> . >>>>>=20 >>>>> For help, type "help". >>>>> Type "apropos word" to search for commands related to "word"... >>>>> Reading symbols from /boot/kernel.GENERIC-NODEBUG/kernel... >>>>> Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/kernel.debug... >>>>>=20 >>>>> Unread portion of the kernel message buffer: >>>>> KDB: enter: manual escape to debugger >>>>>=20 >>>>> Reading symbols from /boot/kernel.GENERIC-NODEBUG/uhid.ko... >>>>> Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/uhid.ko.debug... >>>>> Reading symbols from /boot/kernel.GENERIC-NODEBUG/wmt.ko... >>>>> Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/wmt.ko.debug... >>>>> Reading symbols from /boot/kernel.GENERIC-NODEBUG/ums.ko... >>>>> Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/ums.ko.debug... >>>>> Reading symbols from /boot/kernel.GENERIC-NODEBUG/zfs.ko... >>>>> Reading symbols from = /usr/lib/debug//boot/kernel.GENERIC-NODEBUG/zfs.ko.debug... >>>>> 0xffff00000050f3f0 in doadump (textdump=3D0, = textdump@entry=3D212431136) at = /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c:404 >>>>> warning: 404 = /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c: No such file or = directory >>>>> (kgdb) bt >>>>> #0 0xffff00000050f3f0 in doadump (textdump=3D0, = textdump@entry=3D212431136) at = /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c:404 >>>>> #1 0xffff0000000ee6a8 in db_dump (dummy=3D, = dummy2=3D, dummy3=3D, dummy4=3D) at /home/pkgbuild/worktrees/main/sys/ddb/db_command.c:596 >>>>> #2 0xffff0000000ee478 in db_command (last_cmdp=3D, = cmd_table=3D, dopager=3Dtrue) at = /home/pkgbuild/worktrees/main/sys/ddb/db_command.c:508 >>>>> #3 0xffff0000000ee150 in db_command_loop () at = /home/pkgbuild/worktrees/main/sys/ddb/db_command.c:555 >>>>> #4 0xffff0000000f1ff4 in db_trap (type=3D, = code=3D) at = /home/pkgbuild/worktrees/main/sys/ddb/db_main.c:267 >>>>> #5 0xffff000000568b0c in kdb_trap (type=3D60, code=3D0, = tf=3D) at = /home/pkgbuild/worktrees/main/sys/kern/subr_kdb.c:790 >>>>> #6 >>>>> #7 kdb_enter (why=3D, msg=3D) at = /home/pkgbuild/worktrees/main/sys/kern/subr_kdb.c:556 >>>>> #8 0xffff0000003625cc in vt_machine_kbdevent (vd=3D, c=3D) at = /home/pkgbuild/worktrees/main/sys/dev/vt/vt_core.c:761 >>>>> #9 vt_processkey (kbd=3D0xffffa000803caa80, vd=3D0xffff000000d24360= , c=3D-2147483514) at = /home/pkgbuild/worktrees/main/sys/dev/vt/vt_core.c:903 >>>>> #10 vt_kbdevent (kbd=3D0xffffa000803caa80, event=3D, arg=3D0xffff000000d24360 ) at = /home/pkgbuild/worktrees/main/sys/dev/vt/vt_core.c:1030 >>>>> #11 0xffff0000001ea048 in kbdmux_intr (kbd=3D0xffffa000803caa80, = arg=3D) at = /home/pkgbuild/worktrees/main/sys/dev/kbdmux/kbdmux.c:565 >>>>> #12 0xffff0000005839ac in taskqueue_run_locked = (queue=3Dqueue@entry=3D0xffffa000803c9c00) at = /home/pkgbuild/worktrees/main/sys/kern/subr_taskqueue.c:517 >>>>> #13 0xffff000000583714 in taskqueue_run (queue=3D0xffffa000803c9c00)= at /home/pkgbuild/worktrees/main/sys/kern/subr_taskqueue.c:532 >>>>> #14 0xffff0000004bc114 in intr_event_execute_handlers = (ie=3D0xffffa0008028ec00, p=3D) at = /home/pkgbuild/worktrees/main/sys/kern/kern_intr.c:1183 >>>>> #15 ithread_execute_handlers (ie=3D0xffffa0008028ec00, = p=3D) at = /home/pkgbuild/worktrees/main/sys/kern/kern_intr.c:1196 >>>>> #16 ithread_loop (arg=3D, = arg@entry=3D0xffffa000803de5a0) at = /home/pkgbuild/worktrees/main/sys/kern/kern_intr.c:1289 >>>>> #17 0xffff0000004b700c in fork_exit (callout=3D0xffff0000004bbd78 = , arg=3D0xffffa000803de5a0, frame=3D0xffff00010ca97a00) at = /home/pkgbuild/worktrees/main/sys/kern/kern_fork.c:1151 >>>>> #18 >>>>>=20 >>>>> The context here was from an official PkgBase kernel and world >>>>> installation. >>>>>=20 >>>>> . . . (deletion) . . . >>>>>=20 >>>>> You may have to be more explicit about the specific of the >>>>> problem(s) you are seeing. >>>> OK. Here is what I am referring to: >>>>=20 >>>> tuexen@head:~ % sudo kgdb -c /var/crash/vmcore.last = /boot/kernel/kernel >>>=20 >>> That command does not match the parameter order in the man page or >>> in my example. >>>=20 >>> man kgdb output shows kernel first, then core: >>>=20 >>> SYNOPSIS >>> kgdb [-a | -f | -fullname] [-b rate] [-q | -quiet] [-v] [-w] >>> [-d crashdir] [-c core | -n dumpnr | -r device] [kernel = [core]] >>>=20 >>> My example: # kgdb /boot/kernel.GENERIC-NODEBUG/kernel = /var/crash/vmcore.2 >>>=20 >>> You might want to see if using the other order makes a difference. >> No it doesn't. I'm specifying the core via the -c core option instead = of=20 >> the second argument... >=20 > Of course. Sorry for the noise. (Does not look like this is > going to be one of my better mornings.) >=20 > I guess about all we learn is that your issue is somehow more > specific to your context rather than it being an example of > kgdb being generally broken. >=20 >> Best regards >> Michael >>>=20 >>>> Password: >>>> GNU gdb (GDB) 15.1 [GDB v15.1 for FreeBSD] >>>> Copyright (C) 2024 Free Software Foundation, Inc. >>>> License GPLv3+: GNU GPL version 3 or later = >>>> This is free software: you are free to change and redistribute it. >>>> There is NO WARRANTY, to the extent permitted by law. >>>> Type "show copying" and "show warranty" for details. >>>> This GDB was configured as "aarch64-portbld-freebsd15.0". >>>> Type "show configuration" for configuration details. >>>> For bug reporting instructions, please see: >>>> . >>>> Find the GDB manual and other documentation resources online at: >>>> . >>>>=20 >>>> For help, type "help". >>>> Type "apropos word" to search for commands related to "word"... >>>> Reading symbols from /boot/kernel/kernel... >>>> Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug... >>>>=20 >>>> Unread portion of the kernel message buffer: >>>> panic: tcp_do_segment: sent too much >>>> cpuid =3D 1 >>>> time =3D 1730910226 >>>> KDB: stack backtrace: >>>> db_trace_self() at db_trace_self >>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x38 >>>> vpanic() at vpanic+0x1a0 >>>> panic() at panic+0x48 >>>> tcp_do_segment() at tcp_do_segment+0x2794 >>>> tcp_input_with_port() at tcp_input_with_port+0xcbc >>>> tcp_input() at tcp_input+0x10 >>>> ip_input() at ip_input+0x35c >>>> netisr_dispatch_src() at netisr_dispatch_src+0xd8 >>>> tunwrite() at tunwrite+0x2a8 >>>> devfs_write_f() at devfs_write_f+0x108 >>>> dofilewrite() at dofilewrite+0x7c >>>> kern_writev() at kern_writev+0x4c >>>> sys_writev() at sys_writev+0x40 >>>> do_el0_sync() at do_el0_sync+0x60c >>>> handle_el0_sync() at handle_el0_sync+0x4c >>>> --- exception, esr 0x56000000 >>>> KDB: enter: panic >>>>=20 >>>> Reading symbols from /boot/kernel/tcp_rack.ko... >>>> Reading symbols from = /usr/lib/debug//boot/kernel/tcp_rack.ko.debug... >>>> Reading symbols from /boot/kernel/tcphpts.ko... >>>> Reading symbols from = /usr/lib/debug//boot/kernel/tcphpts.ko.debug... >>>> Reading symbols from /boot/kernel/if_bridge.ko... >>>> Reading symbols from = /usr/lib/debug//boot/kernel/if_bridge.ko.debug... >>>> Reading symbols from /boot/kernel/bridgestp.ko... >>>> Reading symbols from = /usr/lib/debug//boot/kernel/bridgestp.ko.debug... >>>> Reading symbols from /boot/kernel/uhid.ko... >>>> Reading symbols from /usr/lib/debug//boot/kernel/uhid.ko.debug... >>>> Reading symbols from /boot/kernel/wmt.ko... >>>> Reading symbols from /usr/lib/debug//boot/kernel/wmt.ko.debug... >>>> Reading symbols from /boot/kernel/cc_newreno.ko... >>>> Reading symbols from = /usr/lib/debug//boot/kernel/cc_newreno.ko.debug... >>>> 0xffff0000004b5644 in doadump (textdump=3D0) at = /usr/home/tuexen/freebsd-src/sys/kern/kern_shutdown.c:404 >>>> 404 dump_savectx(); >>>> (kgdb) up >>>> #1 0x3fdb0000000e99f0 in ?? () >=20 > Somehow it went from referencing the apparently correct/expected > 0xffff0000004b5644 (doadump) to referencing 0x3fdb0000000e99f0 . > There is also the odd "404 dump_savectx();". >=20 > If bt is used as the first command at the prompt, what does it > show for the backtrace? Anything interesting? Just #0 for > 0xffff0000004b5644 and #1 for 0x3fdb0000000e99f0 ? >=20 >=20 > My doadump line also has ", textdump@entry=3D212431136": >=20 > 0xffff00000050f3f0 in doadump (textdump=3D0, textdump@entry=3D212431136)= at /home/pkgbuild/worktrees/main/sys/kern/kern_shutdown.c:404 >=20 > But I'm not aware of the PkbBase build configuration information > being published for making comparisons with. >=20 >=20 >>>> (kgdb) Initial frame selected; you cannot go up. >>>> (kgdb) Initial frame selected; you cannot go up. >>>> (kgdb) quit >>>> tuexen@head:~ % pkg info gdb gdb-15.1 >>>> Name : gdb >>>> Version : 15.1 >>>> Installed on : Thu Oct 24 10:32:17 2024 CEST >>>> Origin : devel/gdb >>>> Architecture : FreeBSD:15:aarch64 >>>> Prefix : /usr/local >>>> Categories : devel >>>> Licenses : GPLv3 >>>> Maintainer : pizzamig@FreeBSD.org >>>> WWW : https://www.gnu.org/software/gdb/ >>>> Comment : GNU Project Debugger >>>> Options : >>>> BUNDLED_READLINE: off >>>> BUNDLED_ZLIB : off >>>> DEBUGINFOD : off >>>> GDB_LINK : on >>>> GUILE : off >>>> KGDB : on >>>> NLS : on >>>> PORT_ICONV : on >>>> PORT_READLINE : on >>>> PYTHON : on >>>> SOURCE_HIGHLIGHT: on >>>> SYSTEM_ICONV : off >>>> SYSTEM_ZLIB : on >>>> TUI : on >>>> XXHASH : on >>>> Shared Libs required: >>>> libzstd.so.1 >>>> libxxhash.so.0 >>>> libsource-highlight.so.4 >>>> libreadline.so.8 >>>> libpython3.11.so.1.0 >>>> libmpfr.so.6 >>>> libintl.so.8 >>>> libiconv.so.2 >>>> libgmp.so.10 >>>> libexpat.so.1 >>>> libboost_regex.so.1.85.0 >>>> Annotations : >>>> FreeBSD_version: 1500025 >>>> build_timestamp: 2024-10-16T20:27:27+0000 >>>> built_by : poudriere-git-3.4.2 >>>> cpe : cpe:2.3:a:gnu:gdb:15.1:::::freebsd15:aarch64 >>>> flavor : py311 >>>> port_checkout_unclean: no >>>> port_git_hash : 82beca9e630 >>>> ports_top_checkout_unclean: no >>>> ports_top_git_hash: 94c4ac6b071 >>>> repo_type : binary >>>> repository : FreeBSD >>>> Flat size : 58.5MiB >>>> Description : >>>> GDB is a source-level debugger for Ada, C, C++, Objective-C, Pascal = and >>>> many other languages. GDB can target (i.e., debug programs running = on) >>>> more than a dozen different processor architectures, and GDB itself = can >>>> run on most popular GNU/Linux, Unix and Microsoft Windows variants. >=20 > Same gdb package version installation as in my context. > kgdb, of itself, should not be a source of the behavior > differences. >=20 >>>> tuexen@head:~ %=20 >>>>=20 >>>> Using kgdb from "pkg install gdb" and locally built world and = kernel. >>>>=20 >>>> Best regards >>>> Michael >>>>>=20 >>>>> For reference: >>>>>=20 >>>>> . . . (deletion) . . . >>=20 >=20 > I'm not identifying anything else to investigate. >=20 >=20 > =3D=3D=3D > Mark Millard > marklmi at yahoo.com >=20 >=20