panic: Too many early devmap mappings

Wed Jan 13 21:33:39 UTC 2021

On Wed, Jan 13, 2021 at 2:53 AM Mark Millard <marklmi at yahoo.com> wrote:
>
> On 2021-Jan-12, at 16:55, Mark Millard <marklmi at yahoo.com> wrote:
>
> > [I have a git bisect result for the failure: bbfa199cbc16.]
> >
> > On 2021-Jan-12, at 16:24, bob prohaska <fbsd at www.zefox.net> wrote:
> >
> >> On Tue, Jan 12, 2021 at 03:59:44PM -0800, Mark Millard wrote:
> >>>
> >>>
> >>> On 2021-Jan-12, at 15:49, bob prohaska <fbsd at www.zefox.net> wrote:
> >>>
> >>>> An RPi3 running -current updated on Jan. 10 installed a new world/kernel and
> >>>> when rebooted promptly crashed with
> >>>>
> >>>> ---<<BOOT>>---
> >>>> panic: Too many early devmap mappings
> >>>> cpuid = 0
> >>>> time = 1
> >>>> KDB: stack backtrace:
> >>>> (null)() at 0xffff00000011ad90
> >>>>     pc = 0xffff000000760f70  lr = 0xffff00000011ad90
> >>>>     sp = 0xffff0000011df330  fp = 0xffff0000011df530
> >>>>
> >>>> (null)() at 0xffff00000045c2d4
> >>>>     pc = 0xffff00000011ad90  lr = 0xffff00000045c2d4
> >>>>     sp = 0xffff0000011df540  fp = 0xffff0000011df5a0
> >>>>
> >>>> (null)() at 0xffff00000045c07c
> >>>>     pc = 0xffff00000045c2d4  lr = 0xffff00000045c07c
> >>>>     sp = 0xffff0000011df5b0  fp = 0xffff0000011df660
> >>>>
> >>>> (null)() at 0xffff0000007d8380
> >>>>     pc = 0xffff00000045c07c  lr = 0xffff0000007d8380
> >>>>     sp = 0xffff0000011df670  fp = 0xffff0000011df670
> >>>>
> >>>> (null)() at 0xffff00000075dc98
> >>>>     pc = 0xffff0000007d8380  lr = 0xffff00000075dc98
> >>>>     sp = 0xffff0000011df680  fp = 0xffff0000011df6a0
> >>>>
> >>>> (null)() at 0xffff0000007710e4
> >>>>     pc = 0xffff00000075dc98  lr = 0xffff0000007710e4
> >>>>     sp = 0xffff0000011df6b0  fp = 0xffff0000011df6d0
> >>>>
> >>>> (null)() at 0xffff00000028850c
> >>>>     pc = 0xffff0000007710e4  lr = 0xffff00000028850c
> >>>>     sp = 0xffff0000011df6e0  fp = 0xffff0000011df7a0
> >>>>
> >>>> (null)() at 0xffff0000007c8788
> >>>>     pc = 0xffff00000028850c  lr = 0xffff0000007c8788
> >>>>     sp = 0xffff0000011df7b0  fp = 0xffff0000011df830
> >>>>
> >>>> (null)() at 0xffff00000028a64c
> >>>>     pc = 0xffff0000007c8788  lr = 0xffff00000028a64c
> >>>>     sp = 0xffff0000011df840  fp = 0xffff0000011df850
> >>>>
> >>>> (null)() at 0xffff00000039b340
> >>>>     pc = 0xffff00000028a64c  lr = 0xffff00000039b340
> >>>>     sp = 0xffff0000011df860  fp = 0xffff0000011df870
> >>>>
> >>>> (null)() at 0xffff0000004a6950
> >>>>     pc = 0xffff00000039b340  lr = 0xffff0000004a6950
> >>>>     sp = 0xffff0000011df880  fp = 0xffff0000011df8b0
> >>>>
> >>>> (null)() at 0xffff00000076d73c
> >>>>     pc = 0xffff0000004a6950  lr = 0xffff00000076d73c
> >>>>     sp = 0xffff0000011df8c0  fp = 0xffff0000011dfa00
> >>>>
> >>>> (null)() at 0xffff00000000089c
> >>>>     pc = 0xffff00000076d73c  lr = 0xffff00000000089c
> >>>>     sp = 0xffff0000011dfa10  fp = 0x0000000000000000
> >>>>
> >>>> KDB: enter: panic
> >>>> [ thread pid 0 tid 0 ]
> >>>> Stopped at      0xffff0000004a6550
> >>>> db> reboot
> >>>> cpu_reset failed
> >>>>
> >>>> It had to be power-cycled to restart. It came back up readily with
> >>>> kernel.old, which reports main-c255664-g4d64c7243d26 compiled Jan 9.
> >>>>
> >>>> In particular, how does one recognize which revision fixes
> >>>> this problem, assuming it's a bug and not operator error?
> >>>> Presumably, it'll take at least several days to reach git.
> >>>
> >>> Discovered last night on 8GiByte RPi4B's relative to this:
> >>> Booting without a monitor changes the memory use and avoids
> >>> the panic. WIth the 1920x1080 monitor it fails. (Only kernels
> >>> with INVARIANTS make the check that panics, but need not
> >>> mean that others are operating well, even if it is not
> >>> obvious in a specific context.)
> >>>
> >>> Quoted from part of a message list item from last night:
> >>>
> >>> QUOTE
> >>> Going back to my 19cca0b9613d based debug kernel build that
> >>> has the printf's reporting the values used in the test, but
> >>> with no monitor attached, it boots fine and reports:
> >>>
> >>> pmap_mapdev early_boot: akva_devmap_vaddr: ffff007ffffff000 size: 1000
> >>> pmap_mapdev early_boot: va: ffff007fffffe000 VM_MAX_KERNEL_ADDRESS: ffff008000000000 L2_SIZE: 200000
> >>>
> >>> That compares to the previously reported failure figures from
> >>> having the monitor attached for that debug kernel:
> >>>
> >>> pmap_mapdev early_boot: akva_devmap_vaddr: ffff007fff816000 size: 1000
> >>> pmap_mapdev early_boot: va: ffff007fff815000 VM_MAX_KERNEL_ADDRESS: ffff008000000000 L2_SIZE: 200000
> >>> panic: Too many early devmap mappings
> >>>
> >>> where the code does:
> >>>
> >>>             KASSERT(va >= VM_MAX_KERNEL_ADDRESS - L2_SIZE,
> >>>                 ("Too many early devmap mappings"));
> >>>
> >>>
> >>> Looks like akva_devmap_vaddr gets smaller to make room above
> >>> for monitor related data and so va can end up being too small
> >>> by the criteria of this test.
> >>>
> >>> I've no clue who would be appropriate for dealing with this.
> >>> END QUOTE
> >>>
> >>> You may have provided a bound for a bisection
> >>>
> >>
> >> It looks as if unplugging the HDMI monitor (1920x1200) fixed the
> >> panic on the RPi3B+ as well.
> >>
> >> [the original subject line said "devmatch", which confused me hugely 8-)]
> >>
> >
> > A git bisect sequence on a 8 GiBYte RPi4B with a monitor plugged
> > in (to make it use more high kernel RAM such that the KASSERT
> > indicated above fails) resulted in:
> >
> > # git bisect good
> > bbfa199cbc1698631a0e932848e62dd76559d4d7 is the first bad commit
> > commit bbfa199cbc1698631a0e932848e62dd76559d4d7
> > Author: mhorne <mhorne at FreeBSD.org>
> > Date:   Wed Dec 9 16:38:42 2020 -0400
> >
> >    arm64: gdb(4) machine-dependent bits
> >
> >    Everything required for remote kernel debugging over a serial
> >    connection. For FDT-based systems, a debug port can be specified by
> >    setting hw.fdt.dbgport to the desired device tree node in loader.conf.
> >    For example, hw.fdt.dbgport="uart1", or
> >    hw.fdt.dbgport="serial at ff1a0000".
> >
> >    Looks good:     emaste
> >    Tested by:      rwatson
> >    MFC after:      2 weeks
> >    Sponsored by:   The FreeBSD Foundation
> >    Differential Revision:  https://reviews.freebsd.org/D27727
> >
> > sys/arm64/arm64/gdb_machdep.c   | 112 ++++++++++++++++++++++++++++++++++++++++
> > sys/arm64/conf/GENERIC          |   2 +-
> > sys/arm64/include/gdb_machdep.h |  81 +++++++++++++++++++++++++++++
> > sys/conf/files.arm64            |   1 +
> > 4 files changed, 195 insertions(+), 1 deletion(-)
> > create mode 100644 sys/arm64/arm64/gdb_machdep.c
> > create mode 100644 sys/arm64/include/gdb_machdep.h
> >
>
> I forgot to list the bugzilla for this:
>
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252541
>
> I have added the notes, including:
>
> QUOTE
> Turns out that this "too much high kernel memory in use" issue happens for
> a combination of 2 things being true at the same time:
>
> A) Monitor attached (sufficiently large pixel count?)
> B) GDB enabled, per bbfa199cbc16 .
> END QUOTE
>
>
> ===
> Mark Millard
> marklmi at yahoo.com
> ( dsl-only.net went
> away in early 2018-Mar)
>

Hi all,

The problem should be fixed with commit 818390ce0ca5 to main. Please
let me know if you still encounter any issues after updating.

Cheers,
Mitchell