top shows only part of available physmem

Thu Jan 27 03:21:44 UTC 2011

On Thu, Jan 27, 2011 at 01:59:01AM +0100, Bartosz Stec wrote:
> W dniu 2011-01-26 19:44, John Baldwin pisze:
> >On Wednesday, January 26, 2011 1:04:02 pm Marco van Tol wrote:
> >>On Wed, Jan 26, 2011 at 12:35:56PM -0500, John Baldwin wrote:
> >>>On Wednesday, January 26, 2011 8:20:28 am Bartosz Stec wrote:
> >>>>W dniu 2011-01-26 14:06, John Baldwin pisze:
> >>>>>On Wednesday, January 26, 2011 7:20:34 am Bartosz Stec wrote:
> >>>>>>Guys,
> >>>>>>
> >>>>>>could someone explain me this?
> >>>>>>
> >>>>>>      # sysctl hw.realmem
> >>>>>>      hw.realmem: 2139029504
> >>>>>>
> >>>>>>top line shows:
> >>>>>>
> >>>>>>      Mem: 32M Active, 35M Inact, 899M Wired, 8392K Cache, 199M Buf, 58M Free
> >>>>>>
> >>>>>>32+35+899+8+199+58 = 1231MB
> >>>>>>
> >>>>>>Shouldn't that sum to all available ram? Or maybe I'm reading it wrong?
> >>>>>>This machine has indeed 2GB of ram on board and showed in BIOS.
> >>>>>>i386  FreeBSD 8.2-PRERELEASE #16: Mon Jan 17 22:28:53 CET 2011
> >>>>>>Cheers.
> >>>>>First, don't include 'buf' as isn't a separate set of RAM, it is only a range
> >>>>>of the virtual address space in the kernel.  It used to be relevant when the
> >>>>>buffer cache was separate from the VM page cache, but now it is mostly
> >>>>>irrelevant (arguably it should just be dropped from top output).
> >>>>Thanks for the explanation. So 1231MB - 199MB Buf and we got about 1GB
> >>>>of memory instead of 2B.
> >>>>
> >>>>>However, look at what hw.physmem says (and the realmem and availmem lines in
> >>>>>dmesg).  realmem is actually not that useful as it is not a count of the
> >>>>>amount of memory, but the address of the highest memory page available.  There
> >>>>>can be less memory available than that due to "holes" in the address space for
> >>>>>PCI memory BARs, etc.
> >>>>>
> >>>>OK, here you go:
> >>>># sysctl hw | grep mem
> >>>>
> >>>>     hw.physmem: 2125893632
> >>>>     hw.usermem: 1212100608
> >>>>     hw.realmem: 2139029504
> >>>>     hw.pci.host_mem_start: 2147483648
> >>>Humm, you should still have 2GB of RAM then.  All the memory you set aside
> >>>for ARC should be counted in the 'wired' count, so I'm not sure why you see
> >>>1GB of RAM rather than 2GB.
> >>For what its worth (seems to be the same values top shows), the sysctl's
> >>I use to make cacti graphs of memory usage are: (Counts are in pages)
> >>
> >>vm.stats.vm.v_page_size
> >>
> >>vm.stats.vm.v_wire_count
> >>vm.stats.vm.v_active_count
> >>vm.stats.vm.v_inactive_count
> >>vm.stats.vm.v_cache_count
> >>vm.stats.vm.v_free_count
> >>
> >>Using the output of those sysctls I allways get a cacti graph which at
> >>least very much seems to account for all memory, and has a flat surface
> >>in a stacked graph.
> >These sysctls are exactly what top uses.  There is also a 'v_page_count'
> >which is a total count of pages.
> >
> So here's additional sysctl output from now:
> 
>    fbsd# sysctl hw | grep mem
>    hw.physmem: 2125893632
>    hw.usermem: 1392594944
>    hw.realmem: 2139029504
>    hw.pci.host_mem_start: 2147483648
> 
>    fbsd# sysctl vm.stats.vm
>    vm.stats.vm.v_kthreadpages: 0
>    vm.stats.vm.v_rforkpages: 0
>    vm.stats.vm.v_vforkpages: 1422927
>    vm.stats.vm.v_forkpages: 4606557
>    vm.stats.vm.v_kthreads: 40
>    vm.stats.vm.v_rforks: 0
>    vm.stats.vm.v_vforks: 9917
>    vm.stats.vm.v_forks: 30429
>    vm.stats.vm.v_interrupt_free_min: 2
>    vm.stats.vm.v_pageout_free_min: 34
>    vm.stats.vm.v_cache_max: 27506
>    vm.stats.vm.v_cache_min: 13753
>    vm.stats.vm.v_cache_count: 20312
>    vm.stats.vm.v_inactive_count: 18591
>    vm.stats.vm.v_inactive_target: 20629
>    vm.stats.vm.v_active_count: 1096
>    vm.stats.vm.v_wire_count: 179027
>    vm.stats.vm.v_free_count: 6193
>    vm.stats.vm.v_free_min: 3260
>    vm.stats.vm.v_free_target: 13753
>    vm.stats.vm.v_free_reserved: 713
>    vm.stats.vm.v_page_count: 509752
>    vm.stats.vm.v_page_size: 4096
>    vm.stats.vm.v_tfree: 196418851
>    vm.stats.vm.v_pfree: 2837177
>    vm.stats.vm.v_dfree: 0
>    vm.stats.vm.v_tcached: 1305893
>    vm.stats.vm.v_pdpages: 3527455
>    vm.stats.vm.v_pdwakeups: 187
>    vm.stats.vm.v_reactivated: 83786
>    vm.stats.vm.v_intrans: 3053
>    vm.stats.vm.v_vnodepgsout: 134384
>    vm.stats.vm.v_vnodepgsin: 29213
>    vm.stats.vm.v_vnodeout: 96249
>    vm.stats.vm.v_vnodein: 29213
>    vm.stats.vm.v_swappgsout: 19730
>    vm.stats.vm.v_swappgsin: 8573
>    vm.stats.vm.v_swapout: 5287
>    vm.stats.vm.v_swapin: 2975
>    vm.stats.vm.v_ozfod: 83338
>    vm.stats.vm.v_zfod: 2462557
>    vm.stats.vm.v_cow_optim: 330
>    vm.stats.vm.v_cow_faults: 1239253
>    vm.stats.vm.v_vm_faults: 5898471
> 
>    fbsd# sysctl vm.vmtotal
>    vm.vmtotal:
>    System wide totals computed every five seconds: (values in kilobytes)
>    ===============================================
>    Processes:              (RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 60)
>    Virtual Memory:         (Total: 4971660K Active: 699312K)
>    Real Memory:            (Total: 540776K Active: 29756K)
>    Shared Virtual Memory:  (Total: 41148K Active: 19468K)
>    Shared Real Memory:     (Total: 4964K Active: 3048K)
>    Free Memory Pages:      105308K
> 
> 
>    /usr/bin/top line: Mem: 4664K Active, 73M Inact, 700M Wired, 79M
>    Cache, 199M Buf, 23M Free
>    Sum (Without Buf): 879,5 MB
> 
>    So what are we looking at? Wrong sysctls/top output or maybe
>    actually FreeBSD doesn't use all available RAM for some reason?
>    Could it be hardware problem? Maybe I should provide some additional
>    data?

Does the behaviour become more expected if you remove ZFS from the
picture?  Please try this (yes really).

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |