Sendmail and high kernel CPU utilisation

[Travis Fitch]

Hello,

I was hoping someone might be able to shed some light on an issue I am
having with sendmail on FreeBSD 5.2.1 running on a Sun V120.

If you look at the snippet from top you can see that several sendmail
process are casing the kernel to use ~ 65% of the CPU. I am having some
issues profiling what is causing this issue. You can also see the load is
quite high.

last pid: 72705;  load averages: 12.22, 12.29, 12.00
up 1+06:09:44  11:42:28
86 processes:  14 running, 72 sleeping
CPU states: 35.5% user,  0.0% nice, 64.5% system,  0.0% interrupt,  0.0%
idle
Mem: 148M Active, 104M Inact, 73M Wired, 520K Cache, 60M Buf, 131M Free
Swap: 2057M Total, 102M Used, 1954M Free, 4% Inuse

  PID USERNAME  PRI NICE   SIZE    RES STATE    TIME   WCPU    CPU COMMAND
20958 root      122    0 23368K  5472K RUN     44:32  8.64%  8.64% sendmail
62864 root      122    0 23432K  5528K RUN     10:45  8.54%  8.54% sendmail
70522 root      122    0 23448K  5544K RUN      2:43  8.35%  8.35% sendmail
91279 smmsp     122    0 12224K  3832K RUN     60:17  8.25%  8.25% sendmail
66302 root      122    0 23392K  5472K RUN      8:36  8.25%  8.25% sendmail
16850 root      122    0 23432K  5528K RUN     51:31  8.20%  8.20% sendmail
66306 root      121    0 23448K  5528K RUN      8:34  8.11%  8.11% sendmail
68330 root      121    0 23432K  5528K RUN      5:37  8.11%  8.11% sendmail
51654 root      121    0 23392K  5472K RUN     16:40  8.06%  8.06% sendmail
66377 root      121    0 23392K  5472K RUN      8:22  8.01%  8.01% sendmail
69364 root      121    0 23432K  5504K RUN      3:50  8.01%  8.01% sendmail


The other interesting thing is that a lot of the SMTP connections hang
around for quite some time. These connection just keep building up and
slowly bring the system to a crawl.

correo:root# ps aux| grep -v grep | grep sendmail
root   62864  8.0  1.1 23432 5528  ??  R    10:06AM  10:49.58 sendmail:
k5J06CMN062864 mx02.globalcenter.net.au [203.89.192.35]: DATA (sendmail)
root   66306  8.0  1.1 23448 5528  ??  R    10:18AM   8:38.72 sendmail:
k5J0IoxC066306 omta02ps.mx.bigpond.com [144.140.83.154]: DATA (sendmail)
root   66377  7.9  1.1 23392 5472  ??  R    10:20AM   8:26.25 sendmail:
k5J0KHGa066377 vpn02.commandhub.net [203.89.202.122]: DATA (sendmail)
root   20958  8.0  1.1 23368 5472  ??  R     5:14AM  44:36.44 sendmail:
k5IJF2bd020958 [83.173.162.41]: DATA (sendmail)
root   68330  8.0  1.1 23432 5528  ??  R    10:43AM   5:41.33 sendmail:
k5J0hOAk068330 mx01.globalcenter.net.au [203.89.192.34]: DATA (sendmail)
root   69364  8.0  1.1 23432 5504  ??  R    10:59AM   3:54.45 sendmail:
k5J0xmg6069364 omta05ps.mx.bigpond.com [144.140.83.195]: DATA (sendmail)
root   70522  8.0  1.1 23448 5544  ??  R    11:11AM   2:47.79 sendmail:
k5J1BY7Y070522 mx01.globalcenter.net.au [203.89.192.34]: DATA (sendmail)
root   66302  8.0  1.1 23392 5472  ??  R    10:18AM   8:40.20 sendmail:
k5J0If8t066302 vpn02.commandhub.net [203.89.202.122]: DATA (sendmail)
smmsp  91279  8.0  0.7 12224 3832  ??  Rs    4:01AM  60:20.92 sendmail:
Queue runner at 00:10:00 for /var/spool/clientmqueue (sendmail)
root   16850  8.0  1.1 23432 5528  ??  R     4:30AM  51:34.91 sendmail:
k5IIUaWm016850 [201.150.67.51]: DATA (sendmail)
root   51654  8.0  1.1 23392 5472  ??  R     9:37AM  16:44.52 sendmail:
k5INbbYo051654 vpn02.commandhub.net [203.89.202.122]: DATA (sendmail)
root   91276  0.0  1.0 22928 5032  ??  Ss    4:01AM   0:03.05 sendmail:
rejecting connections on daemon MSA: load average: 12 (sendmail)

I have rebuilt the world and kernel to see if the resolves my issue, but
alas no luck.

Hopefully someone will be able to point be in the right direction.

Regards,

Travis