MFS ROOT over 100MB

[Julian Elischer]

Peter Wemm wrote:
> On Fri, Mar 28, 2008 at 1:12 PM, Julian Elischer <julian at elischer.org> wrote:
>> Miroslav Chlastak wrote:
>>  > Hi,
>>  >
>>  >
>>  > it's possible create MFS ROOT over 100MB?
>>  > If I create MFS ROOT less then 90MB, system boot normally.
>>  > With size 90MB-100MB of MFS ROOT kernel panic with boot and
>>  > over 100MB system is rebooted in loop.
>>  >
>>
>>  I suspect that the MFS root is loaded into kenrelVM which is a limited
>>  resource. in fact it must be I think because it is linked with the
>>  kernel by the loader.
> 
> There isn't a lot of spare kernel virtual memory by default, but there
> are other things that are likely to be being hit first.
> 
> First, have a look at the NKPT kernel compile option.  This is
> probably what is being run into.
> 
> By default, for an i386, non-PAE kernel, the value is 30.  This is the
> initial number of page table pages that are statically allocated for
> early kernel bootup.  Once the VM system is fully up and running, we
> can allocate more.  But during early bootup, this is what we have.
> 
> In this case, 30 page table pages, and there are 1024 entries per page
> table page, with 4K per entry.  ie: 30 x 1024 x 4k = 122MB.  If the VM
> system isn't fully up and running by the time this 122MB of kernel VM
> is consumed, bad things will happen.
> 
> This is where the 90-100MB threshold in MFS root image fits in.  While
> there is 1GB of KVM available in total, the MFS image has to fit
> inside this 122MB chunk, **and** the kernel has to complete bootstrap
> with what's left.
> 
> On an i386 kernel without PAE, the KVA_PAGES option is 256.  It is
> also means 256 x 1024 x 4k = 1GB.  In theory, you could increase the
> NKPT option to close to this amount.  I don't know what the exact
> threshold is, but it'll be pretty close.
> 
> ie: you should be able to add 'options NKPT=220' to the kernel config.
>  This would let you load a mfsroot in the order of about 900MB, **IN
> THEORY**.  The problem is that you'll then run into other kernel VM
> exhaustion.  It still has a total of 1GB available by default.  This
> is changeable too.
> 
> If you're trying to do a massive mfsroot, you'll probably have to
> increase KVA_PAGES too.  For example, 'options KVA_PAGES=512' gives
> you 2GB of kernel VM, and 2GB left for user processes.  You could then
> set NKPT to 300+ and easily get a 1GB mfsroot ramdisk loaded.
> 
> For PAE, page table pages have different numbers of entries.  You have
> to double the numbers because they're in units of 512 x 4k, or 2MB per
> page table page.  So with PAE, you'd need 'options KVA_PAGES=1024' and
> 'options NKPT=600' to get the same 2G/2G user/kernel split, and have
> the 1gb mfsroot image.
> 
> I hope this is helpful.  You'll need to experiment to make this work
> and not waste too much memory.  There is no way to avoid compiling
> custom kernels to do this.

You could probably write code to store the memory disk in physical
memeory and leaqe it unmapped except when needed. With PAE you could 
then have an 8GB ram disk. I'm told that this can be done even without 
changing the rest of the system to use PAE.. (but that's 2nd hand info).


>>  > MFS ROOT image is ok (i can mount this image via mdconfig and mount).
>>  > I'am testing this on RELENG_6_2 and RELENG_7_0.
>>  >
>>  > The same problem like this:
>>  > http://lists.freebsd.org/pipermail/freebsd-stable/2008-February/040626.html
>>  >
>>  > Thanks,
>>  > Mira
>