From nobody Tue Dec 21 17:01:50 2021 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id AAF3418F9245 for ; Tue, 21 Dec 2021 17:02:01 +0000 (UTC) (envelope-from imb@protected-networks.net) Received: from mail.protected-networks.net (mail.protected-networks.net [202.12.127.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.protected-networks.net", Issuer "R3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JJN6J4yTNz4YRh; Tue, 21 Dec 2021 17:02:00 +0000 (UTC) (envelope-from imb@protected-networks.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d= protected-networks.net; h=content-transfer-encoding:content-type :content-type:in-reply-to:from:from:references:content-language :subject:subject:user-agent:mime-version:date:date:message-id; s=201508; t=1640106110; bh=ba5iIFJ+KTboDWHvx1O6mUyzfCJ3MWqCo6qE awP99cM=; b=LYYJ21RGlf8KkKMXGz1c6us313CK9FuEHXZcxCdTvBcx8d6d9chj zt9efeNO6ggl0hJXSC9UYlVub0ky6S26tqT+mnnpMOcNnuDc7TSuLyOPXG5bVH11 +glzeW4Fx0kAijWuEozIVTW31qfpN9b7PzK82+jjlj7GXbdF1w+vFjw= Received: from [192.168.1.10] (toshi.auburn.protected-networks.net [192.168.1.10]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: imb@mail.protected-networks.net) by mail.protected-networks.net (Postfix) with ESMTPSA id BB9DA31CA0; Tue, 21 Dec 2021 12:01:50 -0500 (EST) Message-ID: <311ce6f4-9fa8-0514-193d-6be841af26b2@protected-networks.net> Date: Tue, 21 Dec 2021 12:01:50 -0500 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:91.0) Gecko/20100101 Thunderbird/91.4.1 Subject: Re: Panic: Page Fault in Kernel: Yesterday's CURRENT Content-Language: en-NZ To: Larry Rosenman , Mark Johnston Cc: Alexander Motin , Freebsd current References: <3d1b5249a2c51670de496fad9e8b054c@lerctr.org> <9852ae04-6dd0-1cd4-13fe-e97c68e71b37@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4JJN6J4yTNz4YRh X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=protected-networks.net header.s=201508 header.b=LYYJ21RG; dmarc=pass (policy=reject) header.from=protected-networks.net; spf=pass (mx1.freebsd.org: domain of imb@protected-networks.net designates 202.12.127.228 as permitted sender) smtp.mailfrom=imb@protected-networks.net X-Spamd-Result: default: False [-1.91 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[protected-networks.net:s=201508]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-0.999]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_MEDIUM(0.92)[0.917]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[protected-networks.net:+]; DMARC_POLICY_ALLOW(-0.50)[protected-networks.net,reject]; NEURAL_HAM_SHORT(-0.83)[-0.827]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:5716, ipnet:202.12.127.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Reply-To: imb@protected-networks.net From: Michael Butler via freebsd-current X-Original-From: Michael Butler X-ThisMailContainsUnwantedMimeParts: N I have an old pentium-3 that also won't boot kernels built after Dec 6th. I suspect the commits listed below but, with the device being remote and having no DRAC, I'm struggling to test this theory. The relevant commits .. commit 553af8f1ec71d397b5b4fd5876622b9269936e63 Author: Mark Johnston Date: Mon Dec 6 10:42:19 2021 -0500 x86: Perform late TSC calibration before LAPIC timer calibration commit 62d09b46ad7508ae74d462e49234f0a80f91ff69 Author: Mark Johnston Date: Mon Dec 6 10:42:10 2021 -0500 x86: Defer LAPIC calibration until after timecounters are available It's currently running git rev e43d081f352 and I have a kernel at git rev f06f1d1fdb969fa7a0a6eefa030d8536f365eb6e to test later this evening, Michael On 12/17/21 15:07, Larry Rosenman wrote: > On 12/17/2021 1:36 pm, Mark Johnston wrote: >> On Fri, Dec 10, 2021 at 10:43:19AM -0600, Larry Rosenman wrote: >>> 14-2021_12_07-1217             -      -          1.87G 2021-12-07 12:17 >>> 14-2021_12_09-1957             NR     /          121G  2021-12-09 19:57 >>> >>> If that's any help >> >> I can't tell what this is saying.  A kernel built on the 7th does not >> crash, or...?  Which revision did you update from before you started >> seeing crashes? >> >> From a kgdb session it'd be useful to see output from >> >> (kgdb) frame 8 >> (kgdb) p/x *tmp >> >> to start. >> > > Correct, the 7th didn't panic, but the 9th did, and yesterday's too. > > Grrr > ler in borg in /mnt🔒 on ☁️  (us-east-1) > ❯ kgdb -c /var/crash/vmcore.0  /mnt/boot/kernel/kernel > GNU gdb (GDB) 11.1 [GDB v11.1 for FreeBSD] > Copyright (C) 2021 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. > Type "show copying" and "show warranty" for details. > This GDB was configured as "x86_64-portbld-freebsd14.0". > Type "show configuration" for configuration details. > For bug reporting instructions, please see: > . > Find the GDB manual and other documentation resources online at: >     . > > For help, type "help". > Type "apropos word" to search for commands related to "word"... > Reading symbols from /mnt/boot/kernel/kernel... > (No debugging symbols found in /mnt/boot/kernel/kernel) > Failed to open vmcore: /var/crash/vmcore.0: Permission denied > (kgdb) bt > No stack. > quitb) > > ler in borg in /mnt🔒 on ☁️  (us-east-1) took 6s > ❯ sudo chmod +r /var/crash/* > > ler in borg in /mnt🔒 on ☁️  (us-east-1) > ❯ kgdb -c /var/crash/vmcore.0  /mnt/boot/kernel/kernel > GNU gdb (GDB) 11.1 [GDB v11.1 for FreeBSD] > Copyright (C) 2021 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. > Type "show copying" and "show warranty" for details. > This GDB was configured as "x86_64-portbld-freebsd14.0". > Type "show configuration" for configuration details. > For bug reporting instructions, please see: > . > Find the GDB manual and other documentation resources online at: >     . > > For help, type "help". > Type "apropos word" to search for commands related to "word"... > Reading symbols from /mnt/boot/kernel/kernel... > (No debugging symbols found in /mnt/boot/kernel/kernel) > /wrkdirs/usr/ports/devel/gdb/work-py37/gdb-11.1/gdb/thread.c:1345: > internal-error: void switch_to_thread(thread_info *): Assertion `thr != > NULL' failed. > A problem internal to GDB has been detected, > further debugging may prove unreliable. > Quit this debugging session? (y or n) n > > This is a bug, please report it.  For instructions, see: > . > > /wrkdirs/usr/ports/devel/gdb/work-py37/gdb-11.1/gdb/thread.c:1345: > internal-error: void switch_to_thread(thread_info *): Assertion `thr != > NULL' failed. > A problem internal to GDB has been detected, > further debugging may prove unreliable. > Create a core file of GDB? (y or n) n > Command aborted. > (kgdb) bt > No thread selected. > (kgdb) fr 8 > No thread selected. > (kgdb) > >>> On 12/10/2021 10:36 am, Alexander Motin wrote: >>> > Hi Larry, >>> > >>> > This looks like some use-after-free or otherwise corrupted callout >>> > structure.  Unfortunately the backtrace does not tell what was the >>> > callout.  When was the previous update to look what could change? >>> > >>> > On 10.12.2021 11:24, Larry Rosenman wrote: >>> >> FreeBSD borg.lerctr.org 14.0-CURRENT FreeBSD 14.0-CURRENT #15 >>> >> main-n251537-ab639f2398b: Thu Dec  9 19:45:37 CST 2021 >>> >> root@borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/LER-MINIMAL >>> >> amd64 >>> >> >>> >> VMCORE *IS* available. >>> >> >>> >> >>> >> >>> >> >>> >> Unread portion of the kernel message buffer: >>> >> kernel trap 12 with interrupts disabled >>> >> >>> >> >>> >> Fatal trap 12: page fault while in kernel mode >>> >> cpuid = 0; apic id = 20 >>> >> fault virtual address   = 0x0 >>> >> fault code              = supervisor write data, page not present >>> >> instruction pointer     = 0x20:0xffffffff804e0db4 >>> >> stack pointer           = 0x0:0xfffffe0434de4e10 >>> >> frame pointer           = 0x0:0xfffffe0434de4e70 >>> >> code segment            = base 0x0, limit 0xfffff, type 0x1b >>> >>                         = DPL 0, pres 1, long 1, def32 0, gran 1 >>> >> processor eflags        = resume, IOPL = 0 >>> >> current process         = 82990 (c++) >>> >> trap number             = 12 >>> >> panic: page fault >>> >> cpuid = 0 >>> >> time = 1639111198 >>> >> KDB: stack backtrace: >>> >> #0 0xffffffff8050fc95 at kdb_backtrace+0x65 >>> >> #1 0xffffffff804c468f at vpanic+0x17f >>> >> #2 0xffffffff804c4503 at panic+0x43 >>> >> #3 0xffffffff807a2195 at trap_fatal+0x385 >>> >> #4 0xffffffff807a21ef at trap_pfault+0x4f >>> >> #5 0xffffffff80779c78 at calltrap+0x8 >>> >> #6 0xffffffff8045ddb8 at handleevents+0x188 >>> >> #7 0xffffffff8045ea3e at timercb+0x24e >>> >> #8 0xffffffff807ca9eb at lapic_handle_timer+0x9b >>> >> #9 0xffffffff8077b9b1 at Xtimerint+0xb1 >>> >> Uptime: 2h28m57s >>> >> Dumping 12829 out of 131023 >>> >> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% >>> >> >>> >> __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 >>> >> 55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" >>> >> (offsetof(struct pcpu, >>> >> (kgdb) #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 >>> >> #1  doadump (textdump=) >>> >>     at /usr/src/sys/kern/kern_shutdown.c:399 >>> >> #2  0xffffffff804c428c in kern_reboot (howto=260) >>> >>     at /usr/src/sys/kern/kern_shutdown.c:487 >>> >> #3  0xffffffff804c46fe in vpanic (fmt=0xffffffff807e1276 "%s", >>> >>     ap=) at /usr/src/sys/kern/kern_shutdown.c:920 >>> >> #4  0xffffffff804c4503 in panic (fmt=) >>> >>     at /usr/src/sys/kern/kern_shutdown.c:844 >>> >> #5  0xffffffff807a2195 in trap_fatal (frame=0xfffffe0434de4d50, >>> eva=0) >>> >>     at /usr/src/sys/amd64/amd64/trap.c:946 >>> >> #6  0xffffffff807a21ef in trap_pfault (frame=0xfffffe0434de4d50, >>> >>     usermode=false, signo=, ucode=) >>> >>     at /usr/src/sys/amd64/amd64/trap.c:765 >>> >> #7  >>> >> #8  0xffffffff804e0db4 in callout_process >>> >> (now=now@entry=38385536922300) >>> >>     at /usr/src/sys/kern/kern_timeout.c:488 >>> >> #9  0xffffffff8045ddb8 in handleevents (now=now@entry=38385536922300, >>> >>     fake=fake@entry=0) at /usr/src/sys/kern/kern_clocksource.c:213 >>> >> #10 0xffffffff8045ea3e in timercb (et=0xffffffff80d475e0 , >>> >>     arg=) at /usr/src/sys/kern/kern_clocksource.c:357 >>> >> #11 0xffffffff807ca9eb in lapic_handle_timer >>> >> (frame=0xfffffe0434de4f40) >>> >>     at /usr/src/sys/x86/x86/local_apic.c:1364 >>> >> #12 >>> >> #13 0x000000080df42bb6 in ?? () >>> >> Backtrace stopped: Cannot access memory at address 0x7ffffdef2c90 >>> >> (kgdb) >