Pernicious problem with vfork / qmail / qmail-scanner

[Justin Baugh, KSC]

Hello list,

I am having a hell of a time with qmail/qmail-scanner, and I am hoping 
someone on the list can help me. Normally this setup has worked 
completely fine on any FreeBSD machine I've set it up on.

This is on 5.1-p10 (which I'm still running as I've yet to upgrade to 
5.2.1R, and I am wondering if that just might be the problem).

I am running a large mailserver which feeds the message to 
qmail-scanner, and then to spamd & a virus scanner (Kaspersky). This is 
a very beefy box (dual 2.4ghz Xeon, 2gb RAM) which should be able to 
handle many, many connections.

My connection limits on tcpserver are set to 60 concurrent connections. 
concurrencyincoming/remote/local are all appropriate for this value. 
When I suddenly get a number of connections (5-10), or when there are 
more than 15-20 concurrent sessions in progress, I begin to get an error 
message from tcpserver and also from qmail-scanner: "Cannot fork: 
Resource temporarily unavailable".

Using truss, I see that the processes are just sitting there doing
vfork/nanosleep over and over and over, in a "Resource temporarily 
unavailable" loop. Once the processes get into this state, they don't 
get out of it, and it seems to build up over time (??) - i.e. more and 
more forking errors until everything is hosed. The only way to fix it is 
to kill off all the qmail-scanner processes.

I have tried messing with every sysctl value I know to change. I've 
checked kern.maxproc, kern.maxfiles, kern.maxfilesperproc, 
kern.maxprocperuid, etc. I've also looked at kern.somaxconn, etc, mostly 
just shooting in the dark. All of these are at sane limits and I am 
nowhere near the limits on open files or the number of processes. I've 
also checked the max datasize (kern.maxdsize), etc etc. I am nowhere 
near the maximum of open files or processes (both of which are
insanely high: maxusers = 384).

At the moment I'm not even using softlimit in the qmail startup script 
in order to eliminate it as a cause. tcpserver is set to allow 60 
concurrent qmail processes; the problems start to occur at about 20 
processes, or when there is a very sudden increase in the number of 
connections At the moment there are no set limits on resource 
consumption for tcpserver or its children - at least not ones that I am 
setting!

Return of limits

Resource limits (current):
   cputime          infinity secs
   filesize         infinity kb
   datasize          1572864 kb
   stacksize          262144 kb
   coredumpsize     infinity kb
   memoryuse        infinity kb
   memorylocked     infinity kb
   maxprocesses         5547
   openfiles           11095
   sbsize           infinity bytes
   vmemoryuse       infinity kb

Return of limits -U {qscand|qmaild} (no surprise here as I checked 
login.conf)

Resource limits for class default:
   cputime          infinity secs
   filesize         infinity kb
   datasize         infinity kb
   stacksize        infinity kb
   coredumpsize     infinity kb
   memoryuse        infinity kb
   memorylocked     infinity kb
   maxprocesses     infinity
   openfiles        infinity
   sbsize           infinity bytes
   vmemoryuse       infinity kb

Return of ulimit as qmaild/qscand (qmail / qmail-scanner users)

core file size        (blocks, -c) unlimited
data seg size         (kbytes, -d) 1572864
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 11095
pipe size          (512 bytes, -p) 1
stack size            (kbytes, -s) 262144
cpu time             (seconds, -t) unlimited
max user processes            (-u) 5547
virtual memory        (kbytes, -v) unlimited

Related sysctls:

kern.maxproc: 6164
kern.maxprocperuid: 5547
kern.maxfiles: 12328
kern.maxfilesperproc: 11095
kern.maxusers: 384

Note, I'm nowhere near these limits:

kern.openfiles: 421
ps -aux | wc -l returns 145

What else could be causing these problems? I would be very grateful for 
any assistance anyone can provide!

-Justin