kernel address space & auto-tuning

[Ian Lepore]

On Mon, 2013-02-11 at 11:37 -0600, Alan Cox wrote:
> Ian and others here,
> 
> Could you please test the attached patch?  However, before you apply
> this patch, can you run
> 
>     sysctl vm.max_kernel_address
> 
> and record the output.  I'd like to know how this output changes after
> the patch is applied.
> 
> Here is an explanation of what the patch does:
> 
> 1. Recently, a #define for VM_KMEM_SIZE_SCALE was added to
> arm/include/vmparam.h.  This is good.  However, it will create a problem
> as soon as someone gets arm hardware with more than ~1.5GB of RAM.  This
> patch introduces a cap on the size of the kmem submap so that booting on
> future larger-memory systems won't fail.
> 
> 2. It appears that arm is more like sparc64 than x86 in the respect that
> the ending address of the kernel address space can vary from machine to
> machine.  To be more precise, I'm talking about the kernel address space
> into which we can dynamically map and unmap code/data and I'm not
> including regions for device access or direct maps.  Thus, the current
> #define for VM_MAX_KERNEL_ADDRESS on arm is really incorrect or at least
> inconsistent with how this is defined on other architectures.  This
> patch borrows from sparc64 in defining a global variable,
> vm_max_kernel_address, rather than a constant #define to represent the
> end of the kernel address space.
> 
> Thanks,
> Alan
> 

Took me a while to get to this, here's the results...

DreamPlug (armv5)

    real memory  = 536870912 (512 MB)
    avail memory = 516612096 (492 MB)

    before: vm.max_kernel_address: 4294967295 0xFFFFFFFF
    after:  vm.max_kernel_address: 4026531840 0xF0000000

rpi (armv6)

    real memory  = 536870912 (512 MB)
    avail memory = 386789376 (368 MB)

    before: vm.max_kernel_address: 4294967295 0xFFFFFFFF
    after:  vm.max_kernel_address: 3741319168 0xDF000000

Beaglebone (armv7)

    real memory  = 268435456 (256 MB)
    avail memory = 254808064 (243 MB)

    before: vm.max_kernel_address: 4294967295 0xFFFFFFFF
    after:  vm.max_kernel_address: 3741319168 0xDF000000

It appears that the values that go into making the max kernel address on
various arm platforms may be choosen somewhat arbitrarily.  Most SoCs
map the on-chip register space "somewhere" way up near the end of the
32-bit virtual address space, and set the max kernel address to that
point.  0xD0000000, 0xE0000000, and 0xF0000000 all seem to be popular
choices.  I just changed the value for DreamPlug, which only needs 1MB
of register mappings, from 0xF0000000 to 0xFE000000 and it seems to be
working fine.

So, given that the variable ending address may be arbitrary and not much
different than the current 0xFFFFFFFF constant, I wonder if your changes
are going to have the intended effect?  It may be that the important
part of the change would come next: pick better arbitrary ending
addresses.  Or do what sparc seems to be doing, and choose a size then
set the address accordingly.

Speaking of what sparc is doing, this sparc code looks suspicious:

	virtsz = roundup(5 / 3 * physsz, PAGE_SIZE_4M <<
	    (PAGE_SHIFT - TTE_SHIFT));

I wonder if the intent there was "3 * physsz / 5"?

-- Ian