qmail uses 100% cpu after FreeBSD-5.0 to 5.1 upgrade

[Chris Shenton]

I've been running qmail for years and like it, installed pretty much
per www.LifeWithQmail.org.  My main system was running FreeBSD
5.0-RELEASE and -CURRENT and qmail was fine.  When I just upgraded to
5.1-CURRENT a couple days back, the qmail-send process started using
all CPU.

  last pid: 22793;  load averages:  1.06,  1.02,  1.00  up 0+08:13:46  20:36:32
  74 processes:  2 running, 72 sleeping

  Mem: 38M Active, 51M Inact, 84M Wired, 28K Cache, 73M Buf, 452M Free
  Swap: 2048M Total, 2048M Free

    PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU    CPU COMMAND
    615 qmails   132    0  1228K   616K RUN    483:00 96.88% 96.88% qmail-send

I noticed an identical complaint on the qmail list, to which there have so
far been no replies (except "you should ask the FreeBSD list"):

    From: Luca Morettoni <mailing at morettoni.net>
    Subject: qmail on FreeBSD 5.1-CURRENT
    To: qmail at list.cr.yp.to

    [...] qmail is run under daemontools and all work fine (the configuration
    is 2 years old!), but when I delivery the first mail (localy or remote)
    the qmail-send process fire up to 100% of CPU infinitely

    All other mail are right delivery, and the CPU use is the only problem, I
    see in qmail-send.c that select() function, after the first message,
    allways return 1

A truss shows me it's running in a tight loop over this code:

    open("lock/trigger",0x4,027757775230)            = 8 (0x8)
    stat("todo",0xbfbffa00)                          = 0 (0x0)
    open("todo",0x4,01)                              = 9 (0x9)
    fstat(9,0xbfbffa00)                              = 0 (0x0)
    fcntl(0x9,0x2,0x1)                               = 0 (0x0)
    fstatfs(0x9,0xbfbff900)                          = 0 (0x0)
    getdirentries(0x9,0x8059000,0x1000,0x805a214)    = 512 (0x200)
    gettimeofday(0xbfbffbc8,0x0)                     = 0 (0x0)
    select(0x9,0xbfbffcbc,0xbfbffc3c,0x0,0xbfbffc24) = 1 (0x1)
    gettimeofday(0xbfbffbc8,0x0)                     = 0 (0x0)
    gettimeofday(0xbfbffbc8,0x0)                     = 0 (0x0)
    select(0x9,0xbfbffcbc,0xbfbffc3c,0x0,0xbfbffc24) = 1 (0x1)
    gettimeofday(0xbfbffbc8,0x0)                     = 0 (0x0)
    getdirentries(0x9,0x8059000,0x1000,0x805a214)    = 0 (0x0)
    lseek(9,0x0,0)                                   = 0 (0x0)
    close(9)                                         = 0 (0x0)
    gettimeofday(0xbfbffbc8,0x0)                     = 0 (0x0)
    select(0x9,0xbfbffcbc,0xbfbffc3c,0x0,0xbfbffc24) = 1 (0x1)
    gettimeofday(0xbfbffbc8,0x0)                     = 0 (0x0)
    close(8)                                         = 0 (0x0)
    open("lock/trigger",0x4,027757775230)            = 8 (0x8)

I see nothing besides usual message delivery information in qmail's logs.

Failing that, I rebuilt qmail and it seemed to have fixed it, but I didn't
wait long enough: it's pegged at 100% CPU, constantly.  If what Luca says is
true, maybe it hadn't sent a message yet.

Anyone else seen this or know what in FreeBSD-5.1 might have changed to cause
this?  Any thoughts on how I might go about diagnosing this any better?

Thanks.