`top` process memory usage: SIZE vs RES

[Jesse Guardiani]

gv-list-freebsdquestions at mygirlfriday.info wrote:

[...]

> J> 1.) Where is my Free memory going?
> 
> given what you say
> custom-python->>>qmail-scanner->clamd->qmail-queue
> 
> This whole scenario is very memory intensive. First you have each email
> "pythonized" and then qmail-scanner is *very* memory intensive, as it has
> initially a very heavy duty perl script for each email before being passed
> off to clamd.

Clamd is a separate issue, since the only clamav command actually run
from the pipeline (and thus under the restrictions of softlimit) is the
clamdscan client, which is NOT memory intensive. Yes, clamd contributes
to the overall memory footprint, but I'm only concerned with getting
softlimit set properly at this point. My machine can always revert to
swap, but the second softlimit is exceeded the email will be temporarily
defered, which I consider a Bad Thing.

Having said that, yes, it is still a very memory intensive pipeline.
I took some time to profile the memory usage a few days ago, and it
looks like the most memory the pipeline should ever use at any given
point in time is ~12780K, with the following processes running:

USER       PID  PPID %CPU %MEM   VSZ  RSS  TT  STAT STARTED      TIME COMMAND
qmaild   24716 24553  0.0  0.2   920  460  ??  I     7:39PM   1:08.07 /var/qmail/bin/qmail-smtpd
qmaild   24718 24716  0.0  0.3   884  488  ??  I     7:39PM   0:08.63 /usr/local/bin/qmail-qfilter /var/qmail/queue-filters/block-forged-sender.py -s
qmailq   24730 24718  9.2  2.1  5052 3988  ??  S     7:41PM   0:55.87 /usr/bin/suidperl -T /dev/fd/4//var/qmail/bin/qmail-scanner-queue.pl (perl)
qmailq   24739 24730 69.7  2.1  5052 3988  ??  R     7:43PM   0:06.55 /usr/bin/suidperl -T /dev/fd/4//var/qmail/bin/qmail-scanner-queue.pl (perl)
qmailq   24740 24739 14.4  0.2   872  400  ??  R     7:43PM   0:01.28 /var/qmail/bin/qmail-queue

(qmail-scanner is silly. For some reason it spawns a copy of itself,
possibly to hand the message off to qmail-queue.)

But even with the softlimit set to 15M, my huge test message to a
server with only about 80M of free RAM (before sending the message.
Free Memory dropped to ~500k while handling the message) somehow
managed to exceed the softlimit. The exact same message, sent to
a machine with ~600M of free RAM and an identical mail server setup,
passed through the pipeline without tripping the softlimit.

>From what I have seen while watching a huge message pass down the
pipeline, none of the processes in the pipeline increase memory
usage in proportion to email size. They're all relatively static.
So I'm a little confused about why the softlimit would be tripped
on a box that had less RAM (128M) but pass through successfully
on a box with more RAM (1G).

Would the act of using more swap effectively increase a process's:

data segment usage?
stack segment usage?
locked physical pages per process?
total of all segments per process?

These are the things that softlimit limits (according to `man softlimit`),
and I admittedly don't understand how any of the above translates to
memory usage as shown by VSZ and RSS under `ps`, or SIZE and RES under
`top`.

Any ideas?



> Maybe running vmstat -w 1 would give you a different perspective also.

I'll check it out.

-- 
Jesse Guardiani, Systems Administrator
WingNET Internet Services,
P.O. Box 2605 // Cleveland, TN 37320-2605
423-559-LINK (v)  423-559-5145 (f)
http://www.wingnet.net