panic: Too many early devmap mappings

[Mark Millard]

[I have a git bisect result for the failure: bbfa199cbc16.]

On 2021-Jan-12, at 16:24, bob prohaska <fbsd at www.zefox.net> wrote:

> On Tue, Jan 12, 2021 at 03:59:44PM -0800, Mark Millard wrote:
>> 
>> 
>> On 2021-Jan-12, at 15:49, bob prohaska <fbsd at www.zefox.net> wrote:
>> 
>>> An RPi3 running -current updated on Jan. 10 installed a new world/kernel and 
>>> when rebooted promptly crashed with 
>>> 
>>> ---<<BOOT>>---
>>> panic: Too many early devmap mappings
>>> cpuid = 0
>>> time = 1
>>> KDB: stack backtrace:
>>> (null)() at 0xffff00000011ad90
>>> 	 pc = 0xffff000000760f70  lr = 0xffff00000011ad90
>>> 	 sp = 0xffff0000011df330  fp = 0xffff0000011df530
>>> 
>>> (null)() at 0xffff00000045c2d4
>>> 	 pc = 0xffff00000011ad90  lr = 0xffff00000045c2d4
>>> 	 sp = 0xffff0000011df540  fp = 0xffff0000011df5a0
>>> 
>>> (null)() at 0xffff00000045c07c
>>> 	 pc = 0xffff00000045c2d4  lr = 0xffff00000045c07c
>>> 	 sp = 0xffff0000011df5b0  fp = 0xffff0000011df660
>>> 
>>> (null)() at 0xffff0000007d8380
>>> 	 pc = 0xffff00000045c07c  lr = 0xffff0000007d8380
>>> 	 sp = 0xffff0000011df670  fp = 0xffff0000011df670
>>> 
>>> (null)() at 0xffff00000075dc98
>>> 	 pc = 0xffff0000007d8380  lr = 0xffff00000075dc98
>>> 	 sp = 0xffff0000011df680  fp = 0xffff0000011df6a0
>>> 
>>> (null)() at 0xffff0000007710e4
>>> 	 pc = 0xffff00000075dc98  lr = 0xffff0000007710e4
>>> 	 sp = 0xffff0000011df6b0  fp = 0xffff0000011df6d0
>>> 
>>> (null)() at 0xffff00000028850c
>>> 	 pc = 0xffff0000007710e4  lr = 0xffff00000028850c
>>> 	 sp = 0xffff0000011df6e0  fp = 0xffff0000011df7a0
>>> 
>>> (null)() at 0xffff0000007c8788
>>> 	 pc = 0xffff00000028850c  lr = 0xffff0000007c8788
>>> 	 sp = 0xffff0000011df7b0  fp = 0xffff0000011df830
>>> 
>>> (null)() at 0xffff00000028a64c
>>> 	 pc = 0xffff0000007c8788  lr = 0xffff00000028a64c
>>> 	 sp = 0xffff0000011df840  fp = 0xffff0000011df850
>>> 
>>> (null)() at 0xffff00000039b340
>>> 	 pc = 0xffff00000028a64c  lr = 0xffff00000039b340
>>> 	 sp = 0xffff0000011df860  fp = 0xffff0000011df870
>>> 
>>> (null)() at 0xffff0000004a6950
>>> 	 pc = 0xffff00000039b340  lr = 0xffff0000004a6950
>>> 	 sp = 0xffff0000011df880  fp = 0xffff0000011df8b0
>>> 
>>> (null)() at 0xffff00000076d73c
>>> 	 pc = 0xffff0000004a6950  lr = 0xffff00000076d73c
>>> 	 sp = 0xffff0000011df8c0  fp = 0xffff0000011dfa00
>>> 
>>> (null)() at 0xffff00000000089c
>>> 	 pc = 0xffff00000076d73c  lr = 0xffff00000000089c
>>> 	 sp = 0xffff0000011dfa10  fp = 0x0000000000000000
>>> 
>>> KDB: enter: panic
>>> [ thread pid 0 tid 0 ]
>>> Stopped at      0xffff0000004a6550
>>> db> reboot
>>> cpu_reset failed
>>> 
>>> It had to be power-cycled to restart. It came back up readily with
>>> kernel.old, which reports main-c255664-g4d64c7243d26 compiled Jan 9.
>>> 
>>> In particular, how does one recognize which revision fixes 
>>> this problem, assuming it's a bug and not operator error? 
>>> Presumably, it'll take at least several days to reach git.
>> 
>> Discovered last night on 8GiByte RPi4B's relative to this:
>> Booting without a monitor changes the memory use and avoids
>> the panic. WIth the 1920x1080 monitor it fails. (Only kernels
>> with INVARIANTS make the check that panics, but need not
>> mean that others are operating well, even if it is not
>> obvious in a specific context.)
>> 
>> Quoted from part of a message list item from last night:
>> 
>> QUOTE
>> Going back to my 19cca0b9613d based debug kernel build that
>> has the printf's reporting the values used in the test, but
>> with no monitor attached, it boots fine and reports:
>> 
>> pmap_mapdev early_boot: akva_devmap_vaddr: ffff007ffffff000 size: 1000
>> pmap_mapdev early_boot: va: ffff007fffffe000 VM_MAX_KERNEL_ADDRESS: ffff008000000000 L2_SIZE: 200000
>> 
>> That compares to the previously reported failure figures from
>> having the monitor attached for that debug kernel:
>> 
>> pmap_mapdev early_boot: akva_devmap_vaddr: ffff007fff816000 size: 1000
>> pmap_mapdev early_boot: va: ffff007fff815000 VM_MAX_KERNEL_ADDRESS: ffff008000000000 L2_SIZE: 200000
>> panic: Too many early devmap mappings
>> 
>> where the code does:
>> 
>>              KASSERT(va >= VM_MAX_KERNEL_ADDRESS - L2_SIZE,
>>                  ("Too many early devmap mappings"));
>> 
>> 
>> Looks like akva_devmap_vaddr gets smaller to make room above
>> for monitor related data and so va can end up being too small
>> by the criteria of this test.
>> 
>> I've no clue who would be appropriate for dealing with this.
>> END QUOTE
>> 
>> You may have provided a bound for a bisection
>> 
> 
> It looks as if unplugging the HDMI monitor (1920x1200) fixed the
> panic on the RPi3B+ as well. 
> 
> [the original subject line said "devmatch", which confused me hugely 8-)] 
> 

A git bisect sequence on a 8 GiBYte RPi4B with a monitor plugged
in (to make it use more high kernel RAM such that the KASSERT
indicated above fails) resulted in:

# git bisect good
bbfa199cbc1698631a0e932848e62dd76559d4d7 is the first bad commit
commit bbfa199cbc1698631a0e932848e62dd76559d4d7
Author: mhorne <mhorne at FreeBSD.org>
Date:   Wed Dec 9 16:38:42 2020 -0400

    arm64: gdb(4) machine-dependent bits
    
    Everything required for remote kernel debugging over a serial
    connection. For FDT-based systems, a debug port can be specified by
    setting hw.fdt.dbgport to the desired device tree node in loader.conf.
    For example, hw.fdt.dbgport="uart1", or
    hw.fdt.dbgport="serial at ff1a0000".
    
    Looks good:     emaste
    Tested by:      rwatson
    MFC after:      2 weeks
    Sponsored by:   The FreeBSD Foundation
    Differential Revision:  https://reviews.freebsd.org/D27727

 sys/arm64/arm64/gdb_machdep.c   | 112 ++++++++++++++++++++++++++++++++++++++++
 sys/arm64/conf/GENERIC          |   2 +-
 sys/arm64/include/gdb_machdep.h |  81 +++++++++++++++++++++++++++++
 sys/conf/files.arm64            |   1 +
 4 files changed, 195 insertions(+), 1 deletion(-)
 create mode 100644 sys/arm64/arm64/gdb_machdep.c
 create mode 100644 sys/arm64/include/gdb_machdep.h


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)