`top` process memory usage: SIZE vs RES

[gv-list-freebsdquestions at mygirlfriday.info]

Hello Jesse,

Thursday, September 11, 2003, 5:15:31 PM, you wrote:

J> I am stress testing a FreeBSD 4.7-RELEASE server (it's a pre-
J> production test server) by sending huge email messages to it
J> via SMTP.

I too am putting together a 4.8 box as we speak for a client.

J> I'm running qmail-1.03 built from source, with the QMAILQUEUE
J> patch, qmail-qfilter, a custom Python script that runs under
J> qmail-qfilter, and qmail-scanner with ClamAV.

J> I test the server by sending a 59M or a 99M email from a
J> remote machine (connected via fxp0).

J> As I watch the email travel down the qmail-smtpd->qmail-qfilter->
custom-python->>qmail-scanner->clamd->qmail-queue "pipeline",
J> I watch the memory usage with `top`.

J> Memory is critical in this type of application, since I run my
J> qmail-smtpd "pipeline" under DJB's softlimit program. I MUST
J> know how much memory to allocate for the upper limit of each
J> pipeline, otherwise qmail-smtpd will terminate the transfer
J> with a 451 SMTP error.

right.

J> Anyway, as I watch `top`, I never see more than 15M being used
J> by the various pipeline programs at any given point in time,
J> but my Free Memory constantly declines until it reaches about
J> 526k.

J> The Questions:
J> ==========================

J> 1.) Where is my Free memory going?

given what you say
custom-python->>>qmail-scanner->clamd->qmail-queue

This whole scenario is very memory intensive. First you have each email
"pythonized" and then qmail-scanner is *very* memory intensive, as it has
initially a very heavy duty perl script for each email before being passed
off to clamd. Multiply this with the default SMTP concurrency limit of
20, and you have a lot of memory usage. Clamd is known to be a memory
piggie too. I switched from that to using fprot which uses less memory and
is faster processing.

J>     I can't account for it
J>     in the SIZE and RES columns of the various processes.
J>     These are relatively constant.

Maybe running vmstat -w 1 would give you a different perspective also.

-- 
Best regards,
Gary