Limiting the size of syslogd output files using options in syslog.conf

[Rodney W. Grimes]

> On Tue, 2019-06-25 at 16:06 -0700, Michael Sierchio wrote:
> > Apologies, but I don't understand the point of this at all.  I object
> > especially to idea of patching syslogd when EVERYTHING you seek can
> > be
> > accomplished with appropriate settings in /etc/newsyslog.conf and
> > changing
> > its entry in /etc/crontab to run more frequently. You can control the
> > size
> > and number of files there.
> > 
> > You can also do remote syslog.
> > 
> > Do you need to rotate logs more than once per minute?

It is not a matter of rapid log rotation it is that your
rotate the very data you want out of existance that is
the problem.

> 
> Newsyslog handles expected log growth.  I guess you've never had a
> hardware device or other problem unexpectedly trigger the writing of
> hundreds or thousands of lines of output per minute to syslog?  It
> happens.  You write apps that try to strike a sensible balance between
> logging important events and not logging so much it leads to problems,
> then something unexpected happens and your carefully crafted logging
> starts happening at 200hz and actually contributes to the problem.

:-)  Only 200hz?  

> We've been using this code at $work since 2003; it appears in dozens of
> different products ranging from lab instruments to national timescale
> systems.  People using the products experiencing such problems just
> want the device to keep working as best it can.  They don't want
> repeated mysterious reboots because of things like /var filling up. 
> (In the case of products like lab instruments, they don't really
> perceive the thing to be a computer running an OS at all.)
> 
> And most of all, when something does go wrong and they get our support
> engineers to look into it, they don't want to hear "We can't figure out
> what happened originally to cause this problem because all the logging
> got rotated out of existence."  Newsyslog does exactly the opposite of
> what you want in that situation... it purposely destroys the valuable
> information about the original problem while carefully preserving the
> useless fallout that follows.

Yes, and this is not only an issue on embeded, been in exactly that
position trying to post diagnose server failures, all the logs had
been rotated out and filled with a repeated changing sequence of
error message that was a cascade effect of what had originally gone wrong.

Syslog does very good with rate limiting and colapsing a single
repeated line, but when it is getting 2 or more intertwined lines
this no longer works.  It may make an interesting project to create
a coalescing buffer that holds some N messages for up to some t
time and collapse them into a "Foo bar table top (repeated n times in t)"
that would be a much smarter rate limiter.  You toss anything from the
coalesce buffer that is of some age > t if it only occured once.

> Over the years we've upstreamed every freebsd customization we've ever
> done at $work except this one.  The only reason I've been carrying this
> one local mod is because it seemed too specialized to be useful to
> anyone else.  Then yesterday someone working on an embedded-system
> product asked me on irc how to solve the problem of unexpected log
> spewage, so I offered the solution we've been using.

I agree that this is a useful feature for exactly the reasons
Ian sites.  It is rarely the end of a fifo log that is useful,
but rather some point back in time that occured before the
cascade effect.

This looks to be a simple, but reasonable solution to a long
standing problem that is not solved in any other way.

The 32k circ buffer size should probably be config file tweakable though.

 
> -- Ian
> > On Tue, Jun 25, 2019 at 3:25 PM Ian Lepore <ian at freebsd.org> wrote:
> > 
> > > I've posted a review of a small syslogd change which lets you set a
> > > limit on the size of syslogd output files in /etc/syslog.conf.  The
> > > idea is to prevent filling up a filesystem on emmc or sdcard or
> > > other
> > > small storage device on an embedded system with unexpected logging
> > > triggered by some error or failing hardware.
> > > 
> > > If you're interested, please see the review at
> > > 
> > > https://reviews.freebsd.org/D20764
> > > 
> > > -- Ian

-- 
Rod Grimes                                                 rgrimes at freebsd.org