panic: Too many early devmap mappings

Wed Jan 13 06:53:08 UTC 2021

On 2021-Jan-12, at 16:55, Mark Millard <marklmi at yahoo.com> wrote:

> [I have a git bisect result for the failure: bbfa199cbc16.]
> 
> On 2021-Jan-12, at 16:24, bob prohaska <fbsd at www.zefox.net> wrote:
> 
>> On Tue, Jan 12, 2021 at 03:59:44PM -0800, Mark Millard wrote:
>>> 
>>> 
>>> On 2021-Jan-12, at 15:49, bob prohaska <fbsd at www.zefox.net> wrote:
>>> 
>>>> An RPi3 running -current updated on Jan. 10 installed a new world/kernel and 
>>>> when rebooted promptly crashed with 
>>>> 
>>>> ---<<BOOT>>---
>>>> panic: Too many early devmap mappings
>>>> cpuid = 0
>>>> time = 1
>>>> KDB: stack backtrace:
>>>> (null)() at 0xffff00000011ad90
>>>> 	 pc = 0xffff000000760f70  lr = 0xffff00000011ad90
>>>> 	 sp = 0xffff0000011df330  fp = 0xffff0000011df530
>>>> 
>>>> (null)() at 0xffff00000045c2d4
>>>> 	 pc = 0xffff00000011ad90  lr = 0xffff00000045c2d4
>>>> 	 sp = 0xffff0000011df540  fp = 0xffff0000011df5a0
>>>> 
>>>> (null)() at 0xffff00000045c07c
>>>> 	 pc = 0xffff00000045c2d4  lr = 0xffff00000045c07c
>>>> 	 sp = 0xffff0000011df5b0  fp = 0xffff0000011df660
>>>> 
>>>> (null)() at 0xffff0000007d8380
>>>> 	 pc = 0xffff00000045c07c  lr = 0xffff0000007d8380
>>>> 	 sp = 0xffff0000011df670  fp = 0xffff0000011df670
>>>> 
>>>> (null)() at 0xffff00000075dc98
>>>> 	 pc = 0xffff0000007d8380  lr = 0xffff00000075dc98
>>>> 	 sp = 0xffff0000011df680  fp = 0xffff0000011df6a0
>>>> 
>>>> (null)() at 0xffff0000007710e4
>>>> 	 pc = 0xffff00000075dc98  lr = 0xffff0000007710e4
>>>> 	 sp = 0xffff0000011df6b0  fp = 0xffff0000011df6d0
>>>> 
>>>> (null)() at 0xffff00000028850c
>>>> 	 pc = 0xffff0000007710e4  lr = 0xffff00000028850c
>>>> 	 sp = 0xffff0000011df6e0  fp = 0xffff0000011df7a0
>>>> 
>>>> (null)() at 0xffff0000007c8788
>>>> 	 pc = 0xffff00000028850c  lr = 0xffff0000007c8788
>>>> 	 sp = 0xffff0000011df7b0  fp = 0xffff0000011df830
>>>> 
>>>> (null)() at 0xffff00000028a64c
>>>> 	 pc = 0xffff0000007c8788  lr = 0xffff00000028a64c
>>>> 	 sp = 0xffff0000011df840  fp = 0xffff0000011df850
>>>> 
>>>> (null)() at 0xffff00000039b340
>>>> 	 pc = 0xffff00000028a64c  lr = 0xffff00000039b340
>>>> 	 sp = 0xffff0000011df860  fp = 0xffff0000011df870
>>>> 
>>>> (null)() at 0xffff0000004a6950
>>>> 	 pc = 0xffff00000039b340  lr = 0xffff0000004a6950
>>>> 	 sp = 0xffff0000011df880  fp = 0xffff0000011df8b0
>>>> 
>>>> (null)() at 0xffff00000076d73c
>>>> 	 pc = 0xffff0000004a6950  lr = 0xffff00000076d73c
>>>> 	 sp = 0xffff0000011df8c0  fp = 0xffff0000011dfa00
>>>> 
>>>> (null)() at 0xffff00000000089c
>>>> 	 pc = 0xffff00000076d73c  lr = 0xffff00000000089c
>>>> 	 sp = 0xffff0000011dfa10  fp = 0x0000000000000000
>>>> 
>>>> KDB: enter: panic
>>>> [ thread pid 0 tid 0 ]
>>>> Stopped at      0xffff0000004a6550
>>>> db> reboot
>>>> cpu_reset failed
>>>> 
>>>> It had to be power-cycled to restart. It came back up readily with
>>>> kernel.old, which reports main-c255664-g4d64c7243d26 compiled Jan 9.
>>>> 
>>>> In particular, how does one recognize which revision fixes 
>>>> this problem, assuming it's a bug and not operator error? 
>>>> Presumably, it'll take at least several days to reach git.
>>> 
>>> Discovered last night on 8GiByte RPi4B's relative to this:
>>> Booting without a monitor changes the memory use and avoids
>>> the panic. WIth the 1920x1080 monitor it fails. (Only kernels
>>> with INVARIANTS make the check that panics, but need not
>>> mean that others are operating well, even if it is not
>>> obvious in a specific context.)
>>> 
>>> Quoted from part of a message list item from last night:
>>> 
>>> QUOTE
>>> Going back to my 19cca0b9613d based debug kernel build that
>>> has the printf's reporting the values used in the test, but
>>> with no monitor attached, it boots fine and reports:
>>> 
>>> pmap_mapdev early_boot: akva_devmap_vaddr: ffff007ffffff000 size: 1000
>>> pmap_mapdev early_boot: va: ffff007fffffe000 VM_MAX_KERNEL_ADDRESS: ffff008000000000 L2_SIZE: 200000
>>> 
>>> That compares to the previously reported failure figures from
>>> having the monitor attached for that debug kernel:
>>> 
>>> pmap_mapdev early_boot: akva_devmap_vaddr: ffff007fff816000 size: 1000
>>> pmap_mapdev early_boot: va: ffff007fff815000 VM_MAX_KERNEL_ADDRESS: ffff008000000000 L2_SIZE: 200000
>>> panic: Too many early devmap mappings
>>> 
>>> where the code does:
>>> 
>>>             KASSERT(va >= VM_MAX_KERNEL_ADDRESS - L2_SIZE,
>>>                 ("Too many early devmap mappings"));
>>> 
>>> 
>>> Looks like akva_devmap_vaddr gets smaller to make room above
>>> for monitor related data and so va can end up being too small
>>> by the criteria of this test.
>>> 
>>> I've no clue who would be appropriate for dealing with this.
>>> END QUOTE
>>> 
>>> You may have provided a bound for a bisection
>>> 
>> 
>> It looks as if unplugging the HDMI monitor (1920x1200) fixed the
>> panic on the RPi3B+ as well. 
>> 
>> [the original subject line said "devmatch", which confused me hugely 8-)] 
>> 
> 
> A git bisect sequence on a 8 GiBYte RPi4B with a monitor plugged
> in (to make it use more high kernel RAM such that the KASSERT
> indicated above fails) resulted in:
> 
> # git bisect good
> bbfa199cbc1698631a0e932848e62dd76559d4d7 is the first bad commit
> commit bbfa199cbc1698631a0e932848e62dd76559d4d7
> Author: mhorne <mhorne at FreeBSD.org>
> Date:   Wed Dec 9 16:38:42 2020 -0400
> 
>    arm64: gdb(4) machine-dependent bits
> 
>    Everything required for remote kernel debugging over a serial
>    connection. For FDT-based systems, a debug port can be specified by
>    setting hw.fdt.dbgport to the desired device tree node in loader.conf.
>    For example, hw.fdt.dbgport="uart1", or
>    hw.fdt.dbgport="serial at ff1a0000".
> 
>    Looks good:     emaste
>    Tested by:      rwatson
>    MFC after:      2 weeks
>    Sponsored by:   The FreeBSD Foundation
>    Differential Revision:  https://reviews.freebsd.org/D27727
> 
> sys/arm64/arm64/gdb_machdep.c   | 112 ++++++++++++++++++++++++++++++++++++++++
> sys/arm64/conf/GENERIC          |   2 +-
> sys/arm64/include/gdb_machdep.h |  81 +++++++++++++++++++++++++++++
> sys/conf/files.arm64            |   1 +
> 4 files changed, 195 insertions(+), 1 deletion(-)
> create mode 100644 sys/arm64/arm64/gdb_machdep.c
> create mode 100644 sys/arm64/include/gdb_machdep.h
> 

I forgot to list the bugzilla for this:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252541

I have added the notes, including:

QUOTE
Turns out that this "too much high kernel memory in use" issue happens for
a combination of 2 things being true at the same time:

A) Monitor attached (sufficiently large pixel count?)
B) GDB enabled, per bbfa199cbc16 .
END QUOTE

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)