memory leak and swapfile

[Kevin Oberman]

> Date: Sat, 27 Nov 2010 04:17:43 -0800
> From: Jeremy Chadwick <freebsd at jdc.parodius.com>
> 
> On Fri, Nov 26, 2010 at 07:12:59PM -0800, Kevin Oberman wrote:
> > > From: "Jack Raats" <jack at jarasoft.net>
> > > Date: Fri, 26 Nov 2010 19:17:05 +0100
> > > Sender: owner-freebsd-stable at freebsd.org
> > > 
> > > It looks like that there may be a memory leak of my swap space with one of 
> > > the processes that is running.
> > > Big question: How can I determine which process is responsible.
> > > 
> > > Any suggestions?
> > 
> > ps -aux Look for processes with large values in the VSZ column. 
> > 
> > I'm sure that there are other ways to see this, but that's an easy
> > one. You can, of course, pipe the output to sort and use the -k 5 -n
> > options. 
> 
> I believe he should be looking for a process that has a large value in
> RSS ("RES" in top), not VSZ ("SIZE" in top).

I believe it's not that simple, but I think my answer is more likely to
pint to the culprit than yours.

I am not terribly familiar with the details of the FreeBSD virtual
memory system, but I assume that it is similar to other I have known very
well over the years, primarily VMS and OSF/1 where I did a lot of kernel
programming.

FreeBSD does not do "greedy" swap allocation. Some systems will
"reserve" space in swap for all active memory. FreeBSD only uses swap
space when it is needed. RES shows the amount of physical memory a
process is using while VSZ. VSZ is in KB while RES is in pages (I
think), so the numbers wind up looking "odd".

If VSZ is bigger than (RES * page-size in KB), then the entire process
memory space is not in physical memory. It is in one of three other
places: 
1. Imaginary memory (demand-zero pages)
2. Unmapped space (any pages of the image that have not been loaded into
physical memory)
3. Swap

It's very hard to determine how much is where, though unread image pages
are not likely to be significant. Some applications set up huge buffers
of demand-zero memory which may never be used. This is the virtual memory
equivalent of a sparse file. Until a demand zero page is written to, it
takes a page table slot, but does not use either physical memory nor
swap space.

That all said, memory leakage is memory that has been used, but not
freed. It is never accessed, so drops into swap space when memory
pressure triggers the system to look for pages not recently
accessed. It goes to swap and stays thee until the process exists and
VSZ just keeps growing.

If you monitor VSZ and it just keeps growing when the process is not
doing anything that should require ever increasing memory, it's probably
a memory leak.

While RES alone tells you only what is in memory and nothing about swap
use,  leaky process will start by growing RES eventually start having
old pages swapped out so RES stops growing and VSZ keeps growing. If
some process grows with pages that are being actively accessed (not a
leak), RES may get large, but unless memory pressure is great enough,
will use little swap.

Bottom line is that, if the system works the way I believe it does, VSZ
is the best, if not ideal check for memory leaks that fill swap.
-- 
R. Kevin Oberman, Network Engineer
Energy Sciences Network (ESnet)
Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab)
E-mail: oberman at es.net			Phone: +1 510 486-8634
Key fingerprint:059B 2DDF 031C 9BA3 14A4  EADA 927D EBB3 987B 3751