DEBUG - analysing core dumps

[Andrew Duane]

Damien Fleuriot wrote:
> Hello list,
> 
> 
> 
> We've got these boxes at work running FreeBSD 8.1-STABLE amd64 and
> serving as firewalls and openvpn gateways.
> 
> We use CARP interfaces to provide an active-passive fault tolerant
> system. 
> 
> 
> Today, we received a nagios alert from the master box saying it's
> rsyslogd process had crashed.
> 
> I logged on to it and tried to relaunch it, to no avail:
> pid 2303 (rsyslogd), uid 0: exited on signal 11 (core dumped)
> 
> 
> 
> 
> I would like advice on how to debug the output from the core dump.
> 
> This is what I get from gdb:
> 
> # gdb
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and
> you are welcome to change it and/or distribute copies of it under
> certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for
> details. This GDB was configured as "amd64-marcel-freebsd".
> (gdb) core rsyslogd.core
> Core was generated by `rsyslogd'.
> Program terminated with signal 11, Segmentation fault.
> #0  0x00000000004258ec in ?? ()
> 
> 
> Sadly, getting a backtrace with "bt" gives me more lines with "??",
> which is totally not helpful:
> [SNIP]
> #13 0x00007fffff1f9d70 in ?? ()
> #14 0x0000000000000000 in ?? ()
> #15 0x6f70732f7261762f in ?? ()
> #16 0x6c737973722f6c6f in ?? ()
> #17 0x5f6e70766f2f676f in ?? ()
> #18 0x746174732e676f6c in ?? ()
> #19 0x0000000000000065 in ?? ()
> #20 0x0000000000000000 in ?? ()
> [SNIP]
> 
> I am not sure what steps I should follow to get more information ?
> 
> 
> 
> Also, I believe that often, core dumps with signal 11 = RAM problems
> and I would like a confirmation here.
> 
> I am concerned because rsyslogd is the only process that crashes in
> this way, even after I rebooted the firewall.
> 
> Thanks for your input :)

For what it's worth, the addresses shown in frames 15, 16, 17, and 18 are ASCII:

ops/rav/
lsysr/lo
_npvo/go
tats.gol

/Andrew