panic: UMA: Increase vm.boot_pages on Dell R920 r279210

[Patrick Kelsey]

On Sat, May 2, 2015 at 10:25 PM, Adrian Chadd <adrian at freebsd.org> wrote:

> hi,
>
> Hm, should we be upping this limit automatically? Can we get cpu
> counts or memory amount early enough in boot to have a hope of
> auto-tuning?
>
> 64 seems low, 1024 seems high as a default. :)
>


What is it that's exhausting the boot_pages?  I'm semi-guessing it's the
number of vm radix tree nodes needed for the TiB of memory.  The only thing
I'm aware of (allow for ignorance here) that consumes boot_pages and scales
with the cpu count is the uma zone used for uma cache objects, but on amd64
this zone only needs 640 + cpus * 128 bytes, or about 4 pages for 120
cpus.  vm radix nodes are 144 bytes each on amd64, and by my
back-of-the-envelope calculations (using traces of non-vm-radix boot_page
use from another amd64 system), 64 boot_pages would be exhausted after
about 1000 vm radix nodes were allocated.  It would be interesting to know
how many boot_pages were actually required for this particular system.

In any event, since startup_alloc() is designed to exhaust all the
boot_pages before switching to the normal allocators, it doesn't seem
necessarily harmful to err on the high side either in bumping up the static
default or introducing an auto-tuned value (provided the excess is not so
perversely large that startup_alloc() isn't able to make use of an
embarrassment of pages due to zone creation timing and usage patterns).  We
know the number of cpus at the time boot_pages is put to use, but I don't
think  we know how much memory there is (and even less sure that even if we
did, we'd really want to try to estimate things the vm radix tree size in a
generic way).

Something like a default of boot_pages = max(64, 32 + k * cpus) might be
sufficient for k = 4 or 8 (gathering some data points would give a clue
here), and palatable since it is at a minimum the current value that's been
in use, and at the other end approaches a modest commitment of 16 or 32 KiB
per cpu in the worst case (unused and unreclaimed boot_pages with high cpu
count).

-Patrick



>
>
> On 24 March 2015 at 13:00, Keith White <kwhite at site.uottawa.ca> wrote:
> > On Tue, 24 Mar 2015, Rui Paulo wrote:
> >
> >> On Mar 24, 2015, at 04:19, kwhite at site.uottawa.ca wrote:
> >>>
> >>>
> >>> I'm using /boot/loader.conf. Is there another place I should be doing
> >>> this?
> >>
> >>
> >> No, that's correct, but apparently there's a problem: the RDTUN sysctl
> is
> >> not picked up early enough.  Can you try this patch?  I haven't really
> >> tested it. :-)
> >>
> >> diff --git a/sys/vm/vm_page.c b/sys/vm/vm_page.c
> >> index 79665ba..a764788 100644
> >> --- a/sys/vm/vm_page.c
> >> +++ b/sys/vm/vm_page.c
> >> @@ -134,8 +134,9 @@ long first_page;
> >> int vm_page_zero_count;
> >>
> >> static int boot_pages = UMA_BOOT_PAGES;
> >> -SYSCTL_INT(_vm, OID_AUTO, boot_pages, CTLFLAG_RDTUN, &boot_pages, 0,
> >> -       "number of pages allocated for bootstrapping the VM system");
> >> +SYSCTL_INT(_vm, OID_AUTO, boot_pages, CTLFLAG_RDTUN | CTLFLAG_NOFETCH,
> >> +    &boot_pages, 0,
> >> +    "number of pages allocated for bootstrapping the VM system");
> >>
> >> static int pa_tryrelock_restart;
> >> SYSCTL_INT(_vm, OID_AUTO, tryrelock_restart, CTLFLAG_RD,
> >> @@ -349,6 +350,7 @@ vm_page_startup(vm_offset_t vaddr)
> >>         * Allocate memory for use when boot strapping the kernel memory
> >>         * allocator.
> >>         */
> >> +       TUNABLE_INT_FETCH("vm.boot_pages", &boot_pages);
> >>        new_end = end - (boot_pages * UMA_SLAB_SIZE);
> >>        new_end = trunc_page(new_end);
> >>        mapped = pmap_map(&vaddr, new_end, end,
> >> @@ -443,7 +445,7 @@ vm_page_startup(vm_offset_t vaddr)
> >>
> >>
> >> --
> >> Rui Paulo
> >
> >
> > Patch tried.  Success!
> >
> > I now get this after setting vm.boot_pages=1024 in /boot/loader.conf:
> >
> > Booting...
> > GDB: no debug ports present
> > KDB: debugger backends: ddb
> > KDB: current backend: ddb
> > Copyright (c) 1992-2015 The FreeBSD Project.
> > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> >         The Regents of the University of California. All rights reserved.
> > FreeBSD is a registered trademark of The FreeBSD Foundation.
> > FreeBSD 11.0-CURRENT #1: Tue Mar 24 13:44:48 UTC 2015
> >     root@:/usr/obj/usr/src/sys/GENERIC amd64
> > FreeBSD clang version 3.5.1 (tags/RELEASE_351/final 225668) 20150115
> > WARNING: WITNESS option enabled, expect reduced performance.
> > UMA startup boot_pages: 1024
> > ...
> >
> > And can start all 120 processors.
> >
> > Thanks!
> >
> > ...keith
> > --
> > Keith White, genie.uottawa.ca engineering.uottawa.ca
> > kwhite at uottawa.ca [+1 613 562 5800 x6681]
> > _______________________________________________
> > freebsd-current at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-current
> > To unsubscribe, send any mail to "
> freebsd-current-unsubscribe at freebsd.org"
> _______________________________________________
> freebsd-current at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe at freebsd.org"
>