rblsmtpd/qmail-smtpd hung processes

[Brad Mettee]

>I've run into a bit of a problem and if I don't solve it soon, I'm going 
>to be without e-mail (and so will a lot of other people). This is a bit 
>long, I'm trying to include anything that may help.
>
>The problem is that qmail-smtpd and/or rblsmtpd get stuck and don't close. 
>After a while the server just quits accepting smtp connections, or will 
>connect/disconnect very quickly (connection closed by foreign host). Using 
>kill on these processes just opens another instance of them (they're 
>managed by /var/service). Since I'm not seeing any log activity that shows 
>what's going on, I'm having a hard time figuring it out. I know the 
>processes are supposed to be timing out and closing, but don't seem to be 
>doing that
>
>Has anyone seen this problem? I've tried searching for info on it, but 
>can't find anything that looks similar. I currently have rblsmtpd disabled 
>since it sits between the outside world and qmail-smtpd. The errors don't 
>occur quickly, it seems to take multiple hours to occur so this is hard to 
>debug because I can't create the problem on demand. Any ideas are welcome. 
>(hopefully I'll be able to get mail from this address long enough to fix 
>this - it's one of the ones that's been moved to this machine already)
>
>uname -a
>FreeBSD ns1.pchotshots.com 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Sun Feb 24 
>10:35:36 UTC 
>2008     root at driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>
>Related ports in use:
>ucspi-tcp-0.88_2
>qmail-1.03_7
>qmailadmin-1.2.10,1
>vpopmail-5.4.26_1
>dovecote-1.0.14
>
>A quick bit of history:
>Our email/dns provider is closing up shop in about a week (good friend of 
>mine was a small ISP). I only had about 3 weeks notice on this and I've 
>been configuring a pair of FreeBSD machines (I posted to the list a couple 
>weeks ago, thanks everyone for good replies, ) to take over DNS and E-Mail 
>for about a dozen sites. Basically I've been on a crash course of learning 
>as I go.

I hate responding to my own message, but I think I've finally found the 
solution.

I'm using Matt Simerson's Mail Toaster package. The file "toaster.conf" 
file needs to be 644 so that when rotating logs, the programs can re-read 
the config. Apparently all of the hanging files were dying because they 
couldn't read the config and crashed. Once crashed the supervisor decided 
it needed to start a new copy, which would crash as soon as something 
connected to it, and so on. This also seems to have fixed a problem where 
some e-mails would get stuck in the queue and never leave. They've all been 
delivered/deleted like they should now.

I got lucky in finding this. I happened to spot the rotated log file 
staying empty within a few minutes of seeing the hung processes starting to 
stack up. It gave me a good starting point to googling for a solution.