memory leak and swapfile

[Jeremy Chadwick]

On Sun, Nov 28, 2010 at 01:13:37AM +1100, Ian Smith wrote:
> On Sat, 27 Nov 2010, Jeremy Chadwick wrote:
>  > On Fri, Nov 26, 2010 at 07:12:59PM -0800, Kevin Oberman wrote:
>  > > > From: "Jack Raats" <jack at jarasoft.net>
>  > > > Date: Fri, 26 Nov 2010 19:17:05 +0100
>  > > > Sender: owner-freebsd-stable at freebsd.org
>  > > > 
>  > > > It looks like that there may be a memory leak of my swap space with one of 
>  > > > the processes that is running.
>  > > > Big question: How can I determine which process is responsible.
>  > > > 
>  > > > Any suggestions?
>  > > 
>  > > ps -aux Look for processes with large values in the VSZ column. 
>  > > 
>  > > I'm sure that there are other ways to see this, but that's an easy
>  > > one. You can, of course, pipe the output to sort and use the -k 5 -n
>  > > options. 
>  > 
>  > I believe he should be looking for a process that has a large value in
>  > RSS ("RES" in top), not VSZ ("SIZE" in top).
> 
> Not necessarily.  More noticeable in physical-memory constrained systems 
> (try 160MB!), processes that leak memory badly (say mozilla) push leaked 
> but quiescent memory out to swap, where it lurks forever, growing with 
> every page of eg crappy javascript, while RSS stays more constrained.
> 
> % ps aux | sort -rnk5 | head -2
> smithi 11451  0.0 27.3 119008 42276  ??  S    11Nov10 336:47.74 /usr/local/lib/mozilla/mozilla-bin
> root   45236  1.9  7.8  45328 12132  v7  R    15Aug10 3261:37.36 X :0 -auth /home/smithi/.serverau
> 
> At least 35MB of moz's VSZ is leaked and will disappear once restarted, 
> plus another 15MB or so of VSZ X is hanging onto on mozilla's account.
> 
> Small system, small example; big-iron folks miss out on half the fun :)

Here's an example (and not really from a "big-iron" system), though swap
isn't used in this scenario.

# pstat -s
Device          1K-blocks     Used    Avail Capacity
/dev/ada0s1b     16777216        0 16777216     0%

# top -b | egrep 'COMMAND|mysqld'
  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 1199 mysql      10  44    0   801M   271M sigwai  1  16:28  0.00% mysqld

# ps -auxw | grep 1199
mysql    1199  0.0  3.3 820544 277924  ??  S    19Oct10 123:00.88 /usr/local/libexec/mysqld --defaults-extra-file=/storage/mysql/my.c

# procstat -v 1199 | sort -k5 -n | tail -5
 1199         0x90600000         0x90a00000 rw-  612    0   1   0 -- df
 1199           0x400000           0x8f0000 r-x 1206 1564   2   1 CN vn /usr/local/libexec/mysqld
 1199         0x90c00000         0x91400000 rw- 1276    0   2   0 -- df
 1199         0x8fe00000         0x90600000 rw- 2044    0   1   0 -- df
 1199         0x61600000         0x8fe00000 rw- 60285    0   2   0 -- df

# procstat -v 1199 | sort -k6 -n | tail -5
 1199     0x7fffffbff000     0x7fffffc00000 ---    0    0   0   0 -- --
 1199     0x7ffffffe0000     0x800000000000 rwx   15    0   1   0 -- df
 1199         0x61190000         0x6119c000 r-x   10   14   2   1 CN vn /lib/libgcc_s.so.1
 1199         0x60e68000         0x60f3c000 r-x  126  252   2   1 CN vn /usr/lib/libstdc++.so.6
 1199           0x400000           0x8f0000 r-x 1206 1564   2   1 CN vn /usr/local/libexec/mysqld

# procstat -v 1199 | awk '/1199/ { res += $5; pres += $6 } END { print res " " pres }'
69267 1830

Now what's important to take into mind is that the counts shown in
columns 5 and 6 are page counts, not actual memory usage.  I'm making
the assumption that page size is hw.pagesize even though the system
shows 2MB pages available in hw.pagesizes.  So:

69267 * 4096 = 283717632 bytes
 1830 * 4096 =   7495680 bytes

This more or less matches what's in RSS/RES (the ~10MByte difference
probably has to do with scaling of bytes -> megabytes).

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |