Invalid memory stats from vmstat and sysctl vm.vmtotal?

[Steven Hartland]

----- Original Message ----- 
From: "Andriy Gapon" <avg at FreeBSD.org>

>> Totalling up RSS from ps axo "rss" gives a total in the region of that if
>> the vm stats are out by a factor of 4, in this case it should be: 8132557
>> which is 7.75GB a much more realistic value.
>> 
>> Am I totally missing something or is there problem here?
> 
> Likely more of the former than of latter.  Those virtual sizes are not
> sufficiently explained, but you have been warned that those are not physical
> sizes, so I am not sure why you try to compare the virtual figures with the
> physical figures.

My miss-understanding was due to what "virtual" actually meant.

> Here's an example.  Let' say you mmap-ed a 1GB file into a process memory space,
> here you immediately increased your virtual size counts by 1GB, even if you hadn't
> accessed any bytes in the file yet and so none of them were in physical memory.
> The same applies to anonymous memory.
> 
> P.S. the above is reveled by a cursory look through the code (which is publicly
> available btw) :-)

Yer I did have a dig around before posting and ended up the code for vm.vmtotal,
which is where vmstat gets its info from but that's just a summation of each object's
size from vm_object_list. Thats where I got lost without an insight into what
a vm_object.size actually represents.

Your info about mmap'ed files helped point me in the right direction as it 
identified space that shows as virtual but doesn't show in swap or real ram,
which is what I was missing.

Given this starting point the following links provided me with addtional
information:-
http://www.freebsd.org/doc/en/books/arch-handbook/vm.html
http://www.freebsd.org/doc/en/books/design-44bsd/overview-memory-management.html
http://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/
http://www.cse.chalmers.se/edu/course/EDA203/unix4.pdf

I was under the incorrect impression that Virtual Memory (VM) was so named as it
was a unified physical memory and swap (virtual memory), but its not that simple,
as other items such as file-backed objects also count to this total which would
never show in physical or swap allocation of other tools such as top and swapinfo.

So what I believe is now the big cause of virtual memory uplift vs the memory
totals shown by ps / top is that the vm totals include things like file backed
memory mapped process binaries, shared libs etc many multiple times.

This would explain why this specific machine shows the applification more than
others here as it runs thousands of very small lightweight processes.

Thanks for pointer Andy, I now understand a lot more about the BSD VMS :)

What do people think about expanding that entry in the man page of vmstat to
clarify just what active "virtual pages" really means?

    Regards
    Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster at multiplay.co.uk.