Automated submission of kernel panic reports: sysutils/panicmail
Alfred Perlstein
alfred at freebsd.org
Mon Nov 4 12:49:24 UTC 2013
On 11/4/13, 2:41 AM, Colin Percival wrote:
> Hi all,
>
> After considerable review on freebsd-hackers (thanks dt71 and jilles!) I have
> now added sysutils/panicmail to the FreeBSD ports tree. If you install this
> and add
> panicmail_enable="YES"
> to your /etc/rc.conf, a panic report will be generated and sent to root@ for
> you to review and submit (via email). You can skip the reviewing step and
> submit panics automatically by setting panicmail_autosubmit="YES".
>
> The panics submitted are encrypted to an RSA key which I hold in order to keep
> them secure in transit; and I intend to keep the raw panic reports confidential
> except to the minimum extent necessary for other developers to help me process
> the incoming reports.
>
> If I receive enough panic reports to be useful, I hope to provide developers
> with aggregate statistics. This may include:
>
> * regular email reports listing the "top panics", to help guide developers
> towards the most fertile areas for stability improvements;
>
> * email to specific developers alerting them to recurring panics in code they
> maintain (especially if it becomes clear that the panic has been recently
> introduced); and
>
> * guidance to re@ and secteam@ about how often a particular panic occurs if
> an errata notice is being considered
>
> as well as other yet-to-be-imagined reports of a similarly aggregate and
> anonymized nature.
>
> So please install the sysutils/panicmail port and enable it in rc.conf! This
> all depends on getting useful data, and I can't do that without your help.
>
Colin, have you had a few minutes to check out the crash reporting
facilities in FreeNAS?
The reason I ask is that:
1) we would like to share code.
2) we have this running for a few months now and have a huge corpus of
information.
3) we are building a nice UI (screenshots attached) over it, we have a
couple of thousands of lines of code we can share for this.
Our scripts can be found here:
1) A startup script that sends us the crashes on system start:
https://github.com/freenas/freenas/blob/master/nanobsd/Files/etc/rc.d/ix_textdump
2) A script to submit data at boot OR from command line that sends more
comprehensive system information "ixdiagnose":
https://github.com/freenas/freenas/blob/master/nanobsd/Files/usr/local/bin/ixdiagnose
3) A very simple script to upload that report:
https://github.com/freenas/freenas/blob/master/nanobsd/Files/usr/local/bin/crashuploader
We send a minimal set of information: kernel stack trace, ddb buffer and
hardware. Just enough to get some very, very handy stuff.
I can share with you offline the crash server code, it's django and
relatively straight forward.
The screenshots can also be seen at:
http://people.freebsd.org/~alfred/crashreporter/
We could modify our framework for FreeBSD to do so by checking for a
sentinel file depending on the host type and only auto-sending if we see
that.
-Alfred
More information about the freebsd-current
mailing list