hw.physmem/hw.realmem question

[Daniel Braniss]

Thanks Chris!
It will take me some time do fully digets all this!,
but at least the picture is less murky.
	danny

> >for example, this host has has 32G of physical memory ...
> >[snip - dmesg:]
> >real memory  = 34359738368 (32768 MB)
> >avail memory = 32191340544 (30700 MB)
> >[snip]
> >and from sysctl:
> >hw.physmem: 34284916736
> >hw.usermem: 32964923392
> >hw.realmem: 36507222016
> >
> >after setting
> >	hw.physmem=16G
> >
> >from dmesg:
> >real memory  = 34359738368 (32768 MB)
> >avail memory = 13999382528 (13350 MB)
> >
> >and sysctl:
> >
> >	hw.physmem: 14957563904
> >	hw.usermem: 10094678016
> >	hw.realmem: 17179869184
> >
> >from the numbers, I can assume that realmem is the real physical memory,
> >(or whatever is set in hw.physmem),
> 
> I think there are three numbers (though one is not always a single
> number) of interest, which I believe were the original intent of
> the three sysctl values ("real", "phys", and "user" memory,
> specially called out in sysctl.h under the "HW" section).  These
> are:
> 
>   - The RAM present in hardware.  This would be 32 GB on
>     your particular system.
> 
>     On some machines, RAM is not even close to physically
>     contiguous, e.g., there might be 4 GB of RAM starting at
>     address 8 TB, then 32 GB of RAM starting at address 16 TB,
>     then 1 TB of RAM starting at address 24 TB, on a machine where
>     each DIMM slot is simply located at each 8-TB position (with
>     "architectural room" for up to 8 TB per slot), and "slot 0"
>     reserved for boot ROMs (hence physical space starting at 8
>     TB).  These enormous "holes" in the physical space should (my
>     opinion) be reflected in whatever interface gets you the RAM
>     map.  That makes this one not a "single number".  Of course,
>     if you can't or won't show the holes, you can make up a single
>     number.
> 
>     (On x86 systems you don't have holes this big, although you do
>     have the historical "I/O space" windows, e.g., that annoying
>     gap at 640kB :-) .  Worse, there's always a half-GB gap at
>     3.5GB, although some memory controllers work around this.
>     See:
> 
>   http://lists.freebsd.org/pipermail/freebsd-amd64/2005-August/005849.html
> 
>     for example.)
> 
>   - The amount of "usable" RAM for the OS.  This subtracts off
>     any spaces reserved by boot ROMs / BIOSes / what-have-you, and
>     in the case of (e.g.) FreeBSD-9 on amd64, the 1 TB direct-map
>     limit (which you must take care of manually with the loader's
>     hw.physmem setting).
> 
>     This is what "phys mem" should be and mostly is.  If you boot
>     a machine with 1.5 TB of RAM but the OS is limited to 1 TB,
>     hw.physmem should be 1 TB minus a bit for the BIOS, etc.
> 
>   - The amount of memory left after subtracting more or less fixed
>     kernel resources, such as kernel text and data (including loaded
>     modules), page table pages, "vm_page" array data structures, and
>     so on.  This will shift over time.
> 
>     This is what "user mem" is, more or less -- to be exact, it's:
> 
> 	ctob(physmem - cnt.v_wire_count)
> 
>     where "ctob" is "clicks to bytes" which is really "pages to
>     bytes", as these things count in terms of pages (4K at a time,
>     on the x86).  The "wire count" is the number of pages that are
>     not page-in/out-able, so subtracting that from physmem gives
>     you the number of pageable, hence user-use-able, pages.
> 
> There's a fourth number, which is not really very useful, but I
> need to describe for the below:
> 
>   - The highest useable physical address in the system (or
>     really, just after that -- the same way if the machine is to
>     count to 4, you go "0 1 2 3" and that gives you 4).  This
>     is called "Maxmem" in amd64/amd64/machdep.c (and pmap.c).
> 
> Note that the printf output:
> 
>     real memory = <number> (<number> MB)
> 
> generally comes from the amount reported by the BIOS, which is
> different yet again!  My box with 8GB says:
> 
>     real memory  = 8589934592 (8192 MB)
> 
> but on my machine, Maxmem is 8.5 GB, which also shows up in
> sysctl (more on that in a moment):
> 
>     hw.realmem: 9126805504
> 
> I have the Intel memory controller that "moves up" the shadowed
> (by PCI hole) RAM at 3.5 GB. If I were to set hw.physmem to 8
> GB in my loader.conf, I would actually give up half a gigabyte.
> 
> So, on to this:
> 
> >if so, where did almost 2G go? (realmem - physmem) ...
> >what is physmem and realmem, and what's the relationship - if any
> >- between them?
> 
> "hw.realmem" is a snapshot of the value of Maxmem, before a few
> adjustments are made.  If you have 32 GB of physical RAM and you
> have not limited it (and you don't have the remapping noted
> below), "realmem" will be 32 GB.  If you *have* limited it,
> realmem captures the limit (which I think is wrong -- the snapshot
> should happen earlier, at the least).
> 
> "hw.physmem" results from counting up useable pages.  After
> finding Maxmem, which gives you "maximum valid address plus 1",
> the machdep.c code goes through all the segments -- segments being
> stuff like "64k to 640k", with the first 64k off limits because
> BIOSes tend to munch on it and the 640k limit due to the ISA hole
> -- and checks that each page in that segment for useability.  If
> the page is good, it's added to physmem.
> 
> Your 2 GB (542555 pages, to be exact) is space eaten up by your
> BIOS and architectural holes, including the large PCI hole.  Your
> BIOS and motherboard etc may (or may not) allow you to remap some
> of your "hidden" or "shadowed" RAM (out of the PCI hole),
> increasing the boot-time value of Maxmem, and hence also
> increasing both "hw.realmem" and actual, useable pages.
> 
> Note: the x86's architectural holes still use up some dedicated
> kernel memory, even with shadowed-memory remapping: if addresses
> from zero to 8.5 GB are valid (as they are on my box), the kernel
> allocates enough "vm_page" data structures to have one for all 8.5
> GB, even though there's a .5 GB PCI hole with no RAM behind it.
> Those pages are marked "not here, never use these" -- but they
> still take sizeof(struct vm_page) bytes (120 bytes) to represent.
> 512 MB of hole = 512*1024*1024 / 4096 = 131072 pages, which means
> the kernel is using 15728640 bytes (15 MB) to track this empty
> area.  So my remapping hardware gains me 512 MB and then the
> kernel loses 15, for a net of 497 MB recovered.
> 
> Chris