kernel address space & auto-tuning

[Ian Lepore]

On Sat, 2013-02-16 at 13:05 -0600, Alan Cox wrote:
> On 02/16/2013 12:31, Ian Lepore wrote:
> > On Mon, 2013-02-11 at 11:37 -0600, Alan Cox wrote:
> >> Ian and others here,
> >>
> >> Could you please test the attached patch?  However, before you apply
> >> this patch, can you run
> >>
> >>     sysctl vm.max_kernel_address
> >>
> >> and record the output.  I'd like to know how this output changes after
> >> the patch is applied.
> >>
> >> Here is an explanation of what the patch does:
> >>
> >> 1. Recently, a #define for VM_KMEM_SIZE_SCALE was added to
> >> arm/include/vmparam.h.  This is good.  However, it will create a problem
> >> as soon as someone gets arm hardware with more than ~1.5GB of RAM.  This
> >> patch introduces a cap on the size of the kmem submap so that booting on
> >> future larger-memory systems won't fail.
> >>
> >> 2. It appears that arm is more like sparc64 than x86 in the respect that
> >> the ending address of the kernel address space can vary from machine to
> >> machine.  To be more precise, I'm talking about the kernel address space
> >> into which we can dynamically map and unmap code/data and I'm not
> >> including regions for device access or direct maps.  Thus, the current
> >> #define for VM_MAX_KERNEL_ADDRESS on arm is really incorrect or at least
> >> inconsistent with how this is defined on other architectures.  This
> >> patch borrows from sparc64 in defining a global variable,
> >> vm_max_kernel_address, rather than a constant #define to represent the
> >> end of the kernel address space.
> >>
> >> Thanks,
> >> Alan
> >>
> > Took me a while to get to this, here's the results...
> >
> > DreamPlug (armv5)
> >
> >     real memory  = 536870912 (512 MB)
> >     avail memory = 516612096 (492 MB)
> >
> >     before: vm.max_kernel_address: 4294967295 0xFFFFFFFF
> >     after:  vm.max_kernel_address: 4026531840 0xF0000000
> >
> > rpi (armv6)
> >
> >     real memory  = 536870912 (512 MB)
> >     avail memory = 386789376 (368 MB)
> >
> >     before: vm.max_kernel_address: 4294967295 0xFFFFFFFF
> >     after:  vm.max_kernel_address: 3741319168 0xDF000000
> >
> > Beaglebone (armv7)
> >
> >     real memory  = 268435456 (256 MB)
> >     avail memory = 254808064 (243 MB)
> >
> >     before: vm.max_kernel_address: 4294967295 0xFFFFFFFF
> >     after:  vm.max_kernel_address: 3741319168 0xDF000000
> >
> > It appears that the values that go into making the max kernel address on
> > various arm platforms may be choosen somewhat arbitrarily.  Most SoCs
> > map the on-chip register space "somewhere" way up near the end of the
> > 32-bit virtual address space, and set the max kernel address to that
> > point.  0xD0000000, 0xE0000000, and 0xF0000000 all seem to be popular
> > choices.  I just changed the value for DreamPlug, which only needs 1MB
> > of register mappings, from 0xF0000000 to 0xFE000000 and it seems to be
> > working fine.
> >
> > So, given that the variable ending address may be arbitrary and not much
> > different than the current 0xFFFFFFFF constant, I wonder if your changes
> > are going to have the intended effect?
> 
> There are two parts to my change.  One is making VM_MAX_KERNEL_ADDRESS
> accurately reflect the end of the kernel's address space by making it a
> variable.  The other is placing a cap on the size of the kernel's heap,
> i.e., the kmem submap.  Without that cap, machines with larger physical
> memories won't boot.  In fact, I just saw an e-mail a few hours ago from
> Oleksandr Tymoshenko saying that his 1GB Beaglebone won't boot anymore.
> 
> > ...  It may be that the important
> > part of the change would come next: pick better arbitrary ending
> > addresses.
> 
> Is the location of the register mapping something that the kernel
> chooses?  Or is it something that is passed to the kernel at boot-time?
> 

As near as I can tell, it's a pretty much arbitrary choice on a per-SoC
basis, always hard-coded with a named constant of some sort in kernel
source code.  0xE0000000 seems to be a popular choice, with comments
about it being the address used to bootstrap the initial devmap.  That
appears to be true for only one SoC.  I think maybe there's been some
cut and paste propagation here.

-- Ian