Recent stable: bsnmpd eats up memory and cpu

[Trond Endrestøl]

On Mon, 2 May 2016 09:10+0200, Trond Endrestøl wrote:

> On Mon, 2 May 2016 00:01+0200, Wolfgang Zenker wrote:
> 
> > Hi,
> > 
> > after updating some 10-STABLE systems a few days ago, I noticed that on
> > two of those systems bsnmpd started to use up a lot of cpu time, and the
> > available memory shrinked until rendering the system unusable. Killing
> > bsnmpd stops the cpu usage but does not free up memory.
> > Both affected systems are amd64, one having moved from r297555 to
> > r298723, the other from r297555 to r298722. Another amd64 system
> > that went from r297555 to r298722 appears to be not affected.
> > The two affected systems are on an internal LAN segment and there
> > is currently no application connecting to snmp on those machines.
> > 
> > What would be useful debugging data to collect in this case?
> 
> I believe I've seen the very same on my systems. All of them got 
> updated last Friday due to the recent NTP fix. Prior to last Friday, 
> they all ran stable/10 from early March, r296648-ish. Neither of them 
> run bsnmpd, but they offer a lot of network services.
> 
> Three of my i386 systems each with 1 GiB of memory ran out of swap 
> space, Sunday afternoon.
> 
> This night a mail server running i386 with 4 GiB of memory died while 
> handling mail. From the messages I could glean on /dev/ttyvb (due to 
> custom logging) before rebooting, is that it's all networking related.
> 
> SpamAssassin and syslogd on the mail server managed to transmit these 
> lines to the central log host before dying:
> 
> May  2 00:05:17 <mail.err> [HOSTNAME] spamc[63613]: connect to spamd on ::1 failed, retrying (#1 of 3): Connection refused
> May  2 00:05:17 <mail.err> [HOSTNAME] spamc[63613]: connect to spamd on 127.0.0.1 failed, retrying (#1 of 3): Connection refused
> May  2 00:05:18 <mail.err> [HOSTNAME] spamc[63613]: connect to spamd on ::1 failed, retrying (#2 of 3): Connection refused
> May  2 00:05:18 <mail.err> [HOSTNAME] spamc[63613]: connect to spamd on 127.0.0.1 failed, retrying (#2 of 3): Connection refused
> May  2 00:05:19 <mail.err> [HOSTNAME] spamc[63613]: connect to spamd on ::1 failed, retrying (#3 of 3): Connection refused
> May  2 00:05:19 <mail.err> [HOSTNAME] spamc[63613]: connect to spamd on 127.0.0.1 failed, retrying (#3 of 3): Connection refused
> May  2 00:05:19 <mail.err> [HOSTNAME] spamc[63613]: connection attempt to spamd aborted after 3 retries
> 
> May  2 00:52:17 <mail.err> [HOSTNAME] sm-mta[63740]: u41Mp86h063740: Milter (spamassassin): error creating socket: No buffer space available
> May  2 00:52:17 <mail.err> [HOSTNAME] sm-mta[63739]: u41Mp8r9063739: Milter (spamassassin): error creating socket: No buffer space available
> May  2 00:52:17 <mail.info> [HOSTNAME] sm-mta[63740]: u41Mp86h063740: Milter (spamassassin): to error state
> May  2 00:52:17 <mail.info> [HOSTNAME] sm-mta[63739]: u41Mp8r9063739: Milter (spamassassin): to error state
> 
> All of the amd64 systems with 4 GiB or 8 GiB of memory are apparently 
> unaffected.
> 
> Maybe it's time to convert the remaining i386 systems to amd64 
> systems, and add some memory while I'm at it.
> 
> The bug is either in the kernel or in libc, or both.

You might want to try out the patch created by Mark Johnston:

http://lists.freebsd.org/pipermail/freebsd-current/2016-May/061015.html

I'm in the process of testing this patch on one i386 stable/10 
system and on one amd64 stable/10 system.

-- 
+-------------------------------+------------------------------------+
| Vennlig hilsen,               | Best regards,                      |
| Trond Endrestøl,              | Trond Endrestøl,                   |
| IT-ansvarlig,                 | System administrator,              |
| Fagskolen Innlandet,          | Gjøvik Technical College, Norway,  |
| tlf. mob.   952 62 567,       | Cellular...: +47 952 62 567,       |
| sentralbord 61 14 54 00.      | Switchboard: +47 61 14 54 00.      |
+-------------------------------+------------------------------------+