syslogd not draining

[Michael W. Lucas]

On Mon, Mar 27, 2006 at 10:35:11PM +0400, Maxim Konovalov wrote:
> [....]
> > > > > >ns1/etc;netstat -s | grep full
> > > > > >Warning: sysctl(net.inet6.ip6.rip6stats): No such file or directory
> > > > > >        122066 dropped due to full socket buffers
> > > > > >ns1/etc;
> > > > > >
> > > > > >I've doubled kern.ipc.maxsockbuf a couple of times now, and yet it
> > > > > >still happens.
> > >
> > > That's not enough.  You need to teach syslogd to use this new value.
> >
> > I don't see this in syslogd(8); I presume it require source hacking?
> 
> Yes.

OK, I'm going to avoid that for the moment.  I haven't touched C in
five years now, I'd probably confuse it even worse.

Besides, I've had centralized logging hosts with this much activity --
and far more -- previously.  I can't believe that this environment is
so special that it requires that sort of customization.

> [...]
> > > netstat -sp udp | grep 'datagrams received'; sleep 10; \
> > > netstat -sp udp | grep 'datagrams received'
> >
> >         158169 dropped due to full socket buffers
> >         2467251 datagrams received
> > 	sleeping...
> >         158903 dropped due to full socket buffers
> >         2468299 datagrams received
> 
> ~100 datagrams per second, not a lot.  Perhaps they are huge.

Not that I've noticed.  It's syslogd, DHCP, DNS, and flow-capture
from a variety of devices, all generally small packets.

> > > How much cpu time does syslogd use?
> >
> > Not much.  ps -ax | grep syslog gives:
> >
> >  4317 ??  Ss 0:01.60 /usr/sbin/syslogd -l /var/run/log -l
> >  /var/named/var/run/log
> 
> Try to remove a log socket for named and restart syslogd.

Removed the named socket and restarted.  We'll see what happens.

> > Process has been running for about five minutes at that point.
> >
> > Another point that might be of interest:
> >
> > ns1/etc;rc.d/syslogd restart Stopping syslogd. Waiting for PIDS:
> > 4317, 4317, 4317, 4317, 4317, 4317, 4317, 4317, 4317, 4317, 4317,
> > 4317, 4317, 4317, 4317, 4317, 4317, 4317 Starting syslogd.
> 
> What's the /var filesystem type?  Something like gmirror?

Nope.  It's a big SATA drive with a swap partition at the top and the
rest vanilla UFS2:

ad4: 38146MB <WDC WD400JD-75HKA1 14.03G14> at ata2-master SATA150
ad5: 476940MB <Maxtor 6H500F0 HA431C00> at ata2-slave SATA150

ns1~;mount
/dev/ad4s1a on / (ufs, local)
devfs on /dev (devfs, local)
/dev/ad4s1d on /tmp (ufs, local, soft-updates)
/dev/ad4s1e on /usr (ufs, local, soft-updates)
/dev/ad4s1f on /home (ufs, local, soft-updates)
/dev/ad5s1d on /var (ufs, local, soft-updates)
devfs on /var/named/dev (devfs, local)

> diff -u /etc/syslog.conf /usr/src/etc/syslog.conf?

 # $FreeBSD: src/etc/syslog.conf,v 1.28 2005/03/12 12:31:16 glebius Exp $
-#$Id: syslog.conf,v 1.11 2006/03/17 18:56:18 system_mwl Exp system_mwl $
 #
 #      Spaces ARE valid field separators in this file. However,
 #      other *nix-like systems still insist on using tabs as field
 #      separators. If you are sharing this file between systems, you
 #      may want to use only tabs as field separators here.
 #      Consult the syslog.conf(5) manpage.
-*.err;kern.warning;auth.notice;mail.crit;local4.none   /var/log/console.log
-#*.err;kern.warning;auth.notice;mail.crit;local4.none  /dev/console
-*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err;local0.none;local                 1.none;local2.none;local3.none;local4.none;local5.none;local6.none;local7.none /                 var/log/messages
+*.err;kern.warning;auth.notice;mail.crit               /dev/console
+*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err  /var/log/message                 s
 security.*                                     /var/log/security
 auth.info;authpriv.info                                /var/log/auth.log
 mail.info                                      /var/log/maillog
 lpr.info                                       /var/log/lpd-errs
 ftp.info                                       /var/log/xferlog
 cron.*                                         /var/log/cron
-daemon.debug                                   /var/log/daemon.debug
 *.=debug                                       /var/log/debug.log
 *.emerg                                                *
-local0.*                                       /var/log/router
-local1.*                                       /var/log/switch
-#local2.*                                      /var/log/kvm
-#local 2-3 can be used
-local4.*                                       /var/log/pix
-local5.*                                       /var/log/vpn
-local7.*                                       /var/log/dhcpd
 # uncomment this to log all writes to /dev/console to /var/log/console.log
 #console.info                                  /var/log/console.log
 # uncomment this to enable logging of all log messages to /var/log/all.log
 # touch /var/log/all.log and chmod it to mode 600 before it will work
-*.*                                            /var/log/all.log
+#*.*                                           /var/log/all.log
 # uncomment this to enable logging to a remote loghost named loghost
 #*.*                                           @loghost
 # uncomment these if you're running inn
@@ -40,5 +30,3 @@
 *.*                                            /var/log/slip.log
 !ppp
 *.*                                            /var/log/ppp.log
-!flow-capture
-*.*                                            /var/log/flow-capture


-- 
Michael W. Lucas	mwlucas at FreeBSD.org, mwlucas at BlackHelicopters.org
		http://www.BlackHelicopters.org/~mwlucas/

"The cloak of anonymity protects me from the nuisance of caring." -Non Sequitur