Invalid memory stats from vmstat and sysctl vm.vmtotal?

[Andriy Gapon]

on 30/11/2011 14:39 Steven Hartland said the following:
> We're seeing some impossible memory usage stats reported on machines
> here from vmstat and sysctl vm.vmtotal.
> 
> We have machines reporting to be using 31GB total when they only have
> 8GB physical and are not using any swap.
> 
> Here's an output from one of our machines:-
> vmstat -c 2 -w 1 -n 0 procs      memory      page                     
> faults         cpu
> r b w     avm    fre   flt  re  pi  po    fr  sr   in   sy   cs us sy id
> 0 0 0  31768M  2112M   586   0   0   0   421   0  106  270  569  0  6 94
> 0 0 0  31768M  2112M     2   0   0   0     0   0  370 8139 3996  0  1 99
> 
> The raw output is:-
> vmstat -c 2 -w 1 -n 0 -H
> procs      memory      page                      faults         cpu
> r b w     avm    fre   flt  re  pi  po    fr  sr   in   sy   cs us sy id
> 0 0 0 32530228  2162524   586   0   0   0   421   0  106  270  569  0  6 94
> 0 0 0 32530228  2162524     2   0   0   0     0   0  286 8234 4347  0  1 99
> 
> Top shows:-
> last pid:  6665;  load averages:  0.00,  0.00, 0.01 up 80+01:24:12  09:35:28
> 1893 processes:1 running, 1892 sleeping
> CPU:  0.0% user,  0.0% nice,  0.3% system,  0.0% interrupt, 99.7% idle
> Mem: 3754M Active, 84M Inact, 1976M Wired, 4K Cache, 2109M Free
> Swap: 4096M Total, 4096M Free
> 
> sysctl vm.vmtotal
> vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes)
> ===============================================
> Processes:              (RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 1893)
> Virtual Memory:         (Total: 1106403532K Active: 32540260K)
> Real Memory:            (Total: 4563648K Active: 3921644K)
> Shared Virtual Memory:  (Total: 19976K Active: 16396K)
> Shared Real Memory:     (Total: 9040K Active: 8436K)
> Free Memory Pages:      2161740K
> 
> As mentioned this machine has 8GB of ram and according to both top and
> swapinfo is using no swap at all
> 
>> From dmesg:-
> real memory  = 8589934592 (8192 MB)
> avail memory = 8255553536 (7873 MB)
> 
> swapinfo
> Device          1K-blocks     Used    Avail Capacity
> /dev/gptid/09f211f7-39ce-11e0-8   4194304        0  4194304     0%
> 
> uname -a
> FreeBSD test 8.2-RELEASE FreeBSD 8.2-RELEASE #2: Thu Mar 24 17:28:55 UTC 2011    
> root at test:/usr/obj/usr/src/sys/MULTIPLAY  amd64
> 
> sysctl hw.pagesize
> hw.pagesize: 4096
> 
> It looks like it may be out by a factor of 4, possibly due to the fact
> the its a 4k page size not 1k as indicated by the vmstat man page:-
> 
> memory  Information about the usage of virtual and real memory.  Virtual
>             pages (reported in units of 1024 bytes) are considered active if
>             they belong to processes which are running or have run in the
>             last 20 seconds.
> 
>             avm     active virtual pages
>             fre     size of the free list
> 

I think that the description of avm is not sufficiently complete as it doesn't
explain meaning of the "virtual memory" it refers too.
Reference to units of 1024 bytes when talking about pages also sounds a bit vague.
 The values reported are actually byte sizes, non-humanized numbers are in KB
("units of 1024 bytes" in the poetic speech), humanized numbers carry an
appropriate suffix.

> Totalling up RSS from ps axo "rss" gives a total in the region of that if
> the vm stats are out by a factor of 4, in this case it should be: 8132557
> which is 7.75GB a much more realistic value.
> 
> Am I totally missing something or is there problem here?

Likely more of the former than of latter.  Those virtual sizes are not
sufficiently explained, but you have been warned that those are not physical
sizes, so I am not sure why you try to compare the virtual figures with the
physical figures.
Here's an example.  Let' say you mmap-ed a 1GB file into a process memory space,
here you immediately increased your virtual size counts by 1GB, even if you hadn't
accessed any bytes in the file yet and so none of them were in physical memory.
The same applies to anonymous memory.

P.S. the above is reveled by a cursory look through the code (which is publicly
available btw) :-)

-- 
Andriy Gapon