Machines with >= 4GB of RAM

[Terry Lambert]

Andrew Kinney wrote:
> On 17 Dec 2003 at 15:44, Julian Elischer wrote:
> <snip>
> > options  KVA_PAGES=512
> >
> > may be a start, but is it still required, and do I have to change
> > anything else to match it? (where does the Makefile work out where to
> > link the kernel for?)
> >   Is a value of 512 enough for a machine with 16GB of RAM?
> >
> > Any hints, (even a better google search string) appreciated.
> 
> We have a 4GB machine running 4.8-RELEASE, so we aren't using PAE,
> but we had to make changes similar to what you're asking about for a
> different reason.
> 
> Your requirements will vary depending on the version of FreeBSD
> you're running, but in general, increasing KVA_PAGES will help
> considerably with stability on large memory machines.  It should be
> noted that releases prior to 4.8 required more changes than just
> KVA_PAGES, but the documentation is a bit muddied on that subject.

In general, the kmem_map size and other kernel memory usage, including
page tables necessary to reference the full memory, end up taking more
than 1G, so the 3G user:1G kernel ratio that's the default for older
FreeBSD won't work at all.  I usually recommend that people make it
1G user:3G kernel; you can get away with 2G:2G if you aren't going to
be allocating lots of mbufs or supporting lots of open sockets, etc.,
but in general most people throw 4G+ into a box because they plan on
building a network server and then throwing some serious load on it.


> I don't know if it is required, but we rebuilt the world after
> changing KVA_PAGES just to make sure that any hidden dependencies on
> that value were handled in things other than the kernel.

Depends.  The normal case where this will be required is for prebuilt
kernel modules.  The only user space code I'm aware of which cares is
the Linux threads package (and anything that links against it), since
the threads mailboxes are in a fixed location apriori known to both
the kernel module and the threads library, and the location has to be
changed when the KVA space changes, since it assumes a 3:1 or whatever
was in effect when it was compiled.


> As far as 512 being a large enough setting for a 16GB machine, that
> depends entirely on what you plan to do with the machine and its
> usage pattern of various system resources.

In my personal experience, th kernel and data structures consume over
1G in a 4G box, du to the auto-tuning cruft trying to b smarter than
it actually is, and making bad decisions.

In the 4.7/4.8 time frame, this was catastrophic with more than 4G,
since it didn't stop scaling at 4G (the kernel can only address 4G,
without PAE, no matter what, since pointers are 32 bits).  Scaling
above that point tris to allocate more memory for the kernel than
the kernel is capable of addressing.


> For instance, on our 4GB machine, it does a lot of heavy web serving,
> databases, and email.  We needed the 2GB KVA on that machine because
> of large numbers of files, large network buffers, and some weirdness
> relating to Apache and pv entries.  If your usage patterns were
> similar and you wanted to make full use of the 16GB without getting
> trap 12 panics, then 2GB KVA may be inadequate.

Older boxes won't even boot with 3:1 if you jam 4G in them, priod.

-- Terry