Understanding i386 PAE

[John Baldwin]

On Monday, May 09, 2011 12:31:48 pm Brad Waite wrote:
> Hi all, I've got some questions about i386 PAE. I was pointed towards
> this list, but if another is more appropriate, just let me know.
> 
> I also know that i386 PAE is the ugly, red-headed step-child of FreeBSD.
> Kindly keep the standard "why don't you use amd64" statements to a
> minimum. My aim is to gain an accurate understanding of how things work
> with my antiquated and lowly 32-bit hardware.
> 
> Based on my research, hardware w/PAE support can address up to 64G of
> memory via a 36-bit addressing scheme. However, since an i386 process
> can only address 32 bits, each process gets at most 4G of that 64G. In
> my case, my Intel SE7501BR2, dual 2GHz Xeon server supports 8G max and I
> want to use 6G, maximizing the kernel memory.
> 
> I understand that KVA_PAGES determines the amount of memory reserved for
> the kernel, and therefore userland processes. But where does the kernel
> fit into that 32-bit/4G limit?
> 
> My original understanding was that the kernel was a process, just like
> userland ones, and could address 4G and therefore I could split my 6G
> into 4G for the kernel and 2G for userland (KVA_PAGES = 2048).
> 
> I've been informed that PAE doesn't actually work that way. Instead, the
> kernel-reserved memory is per-process and setting KVA_PAGES to 2048
> means the kernel takes up all of the 4G of a process' addressable
> memory, which obviously won't work.
> 
> Is the latter actually the case or is my original understanding correct?
> Or is it somewhere in between?

FreeBSD uses a shared address space on x86 (both i386 and amd64) where
the kernel is mapped into every process' address space at a fixed address.  
This makes it cheaper and easier to copy data between a user process and the 
kernel since you don't have to temporarily map user pages into the kernel's 
address space, etc.

It is possible to use separate address spaces for the kernel and userland 
(other OS's have done this) so that you could have 4G for the kernel and 4G 
for userland.  However, this would require a good bit of work in the kernel 
(e.g. copyin/copyout would have to start mapping user pages at temporary 
addresses in the kernel).

As you have noted, PAE does not increas your virtual address space, merely 
your phyiscal address space by changing PTEs to use 64-bit physical addresses 
instead of 32-bit physical addresses.  However, each process can only address 
up to 4G of RAM at a time.

-- 
John Baldwin