DDB scripting, output capture, and textdumps
Nikolay Pavlov
qpadla at gmail.com
Thu Dec 20 05:32:53 PST 2007
On Tuesday 18 December 2007 14:10:46 Robert Watson wrote:
> Dear all:
>
> I've been hacking on-and-off for a while on a side project to improve
> our kernel debugging facilities. Primarily, my concern has been to
> address three problems:
>
> - The complications of employing kernel core dumps for debugging,
> including the large size of dumps making them unwieldy to distribute
> or store for any extended period (even with minidumps), the requirement
> to have relatively synchronized kernel source in order to use the dumps,
> the need to have a kernel with debugging symbols, and the problems with
> fsck causing sufficient swap use to invalidate dumps before they can be
> extracted.
>
> - The decreasing likelihood that notebooks will ship with serial ports
> that can be used for interactive debugging using DDB. Making
> end-users type in stack traces is cruel, photos are a pain, and X11
> rules out both.
>
> - The fact that a great many problems are most easily diagnosed using
> utility routines present in DDB, but not as easily using kgdb for
> offline analysis. I find that for many bugs I analyze, simply
> looking at the DDB output is sufficient to identify the source of the
> problem.
>
> An idea I punted around a bit at BSDCan earlier this year (or perhaps it
> was at EuroBSDCon the previous year) was an idea of a "textdump" -- that
> is, a new type of kernel dump based on capturing automatically extracted
> debugging information generated by DDB. The result would be an ASCII
> text file that could be filed as a bug report, perhaps even
> automatically.
>
> To this end, I have implemented three new facilities for use with DDB:
>
> (1) DDB output capture. The output of DDB is stored in a memory buffer,
> and can be extracted using a sysctl or textdumps (see below). This
> can be turned on and off, both for use manually ("I'll want this
> later, but not that") and as part of scripts (see below).
>
> (2) DDB scripting. A limited number of named scripts can be defined to
> run a series of DDB commands. No loops, etc, just simple command
> lists. These can be caused to run automatically on entering DDB
> for various scenarios, including WITNESS violations and kernel panics.
> They can also be run by hand in order to save a bit of typing if you use
> DDB in a repetitive way (as I do).
>
> (3) Textdumps. A new dump type that stores a series of data files
> containing various pieces of information, including the DDB capture
> buffer, kernel message buffer, kernel configuration (if compiled
> into the kernel), panic message, and kernel version string. These are
> stored in the ustar format inside the dump partition (aligned to the
> end) so can be easily extended, and savecore(8) requires almost no new
> logic to deal with them (it just drops numbered tar files in
> /var/crash). This makes it straight forward to extend the textdump
> format to include new types of information and avoids the issue of how
> to safely simultaneously represent information in many different formats
> in the same file.
>
> These are pretty flexible tools, and you can imagine doing the following
> sorts of things:
>
> - Setting the kdb.enter.panic script to automatically turn on output
> capture, do full backtraces of all threads, show open file
> information, dump UMA stats, and save it all to a textdump and then
> reboot.
>
> - Setting the kdb.enter.witness script to show lock information,
> generate a coredump, and reboot. Or, just to automatically do "show
> allocks" and drop to the DDB prompt.
>
> - Adding a flag to rc.conf to automatically submit textdumps via e-mail
> to a specific address, perhaps including GNATS or an automated bug
> system. These could be unpacked and automatically analyzed, and do to
> the compact size, kept for long-term trend analysis or to identify when
> a problem started occuring.
>
> I've produced an initial snapshot of the above, which can be found here:
>
> http://www.watson.org/~robert/freebsd/20071218-ddb.tgz
>
> This adds three files to DDB, patches quite a few kernel files (to pass
> more information into KDB about why it's being entered, in order to
> trigger the right script), enhancements to savecore(8) to know how to
> extract textdumps, adds a ddb(8) command line tool so that userspace can
> manage DDB scripts from outside the debugger, extensions to the ddb(4)
> man page, and a new textdump(4) man page.
>
> There are a number of known limitations; I've tried to document them at
> the top of the pertinent files where I am aware of them. I also regret
> to say that to date I've been able to test only on i386, and not other
> platforms. I'd welcome any feedback -- I'd like to get these changes
> into CVS in the next week or two.
It looks like some files is not included in the patch.
I have this error:
make -V CFILES -V SYSTEM_CFILES -V GEN_CFILES | MKDEP_CPP="cc -E" CC="cc"
xargs
mkdep -a -f .newdep -O -pipe -std=c99 -g -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/altq -I/usr/src/sys/contrib/ipfilter -I/usr/src/sys/contrib/pf -I/usr/src/sys/dev/ath -I/usr/src/sys/contrib/ngatm -I/usr/src/sys/dev/twa -I/usr/src/sys/gnu/fs/xfs/FreeBSD -I/usr/src/sys/gnu/fs/xfs/FreeBSD/support -I/usr/src/sys/gnu/fs/xfs -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include
opt_global.h -fno-common -finline-limit=8000 --param
inline-unit-growth=100 --param
large-function-growth=1000 -mno-align-long-strings -mpreferred-stack-boundary=2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -ffreestanding
cc: /usr/src/sys/ddb/db_capture.c: No such file or directory
cc: /usr/src/sys/ddb/db_script.c: No such file or directory
cc: /usr/src/sys/ddb/db_textdump.c: No such file or directory
mkdep: compile failed
*** Error code 1
Stop in /usr/obj/usr/src/sys/GENERIC.
*** Error code 1
Stop in /usr/src.
*** Error code 1
Stop in /usr/src.
root at orion-vm:/usr/src# ls -la /usr/src/sys/ddb/
total 424
drwxr-xr-x 2 root wheel 1024 Dec 19 16:49 ./
drwxr-xr-x 53 root wheel 1024 Oct 29 08:24 ../
-rw-r--r-- 1 root wheel 2591 Dec 4 2005 db_access.c
-rw-r--r-- 1 root wheel 1431 Jan 6 2005 db_access.h
-rw-r--r-- 1 root wheel 7737 Jan 6 2005 db_break.c
-rw-r--r-- 1 root wheel 2098 Jan 6 2005 db_break.h
-rw-r--r-- 1 root wheel 16579 Dec 19 16:49 db_command.c
-rw-r--r-- 1 root wheel 15705 Jan 17 2007 db_command.c.orig
-rw-r--r-- 1 root wheel 1633 Dec 19 16:49 db_command.h
-rw-r--r-- 1 root wheel 1588 Jan 6 2005 db_command.h.orig
-rw-r--r-- 1 root wheel 7270 Oct 27 20:19 db_examine.c
-rw-r--r-- 1 root wheel 4811 Jan 6 2005 db_expr.c
-rw-r--r-- 1 root wheel 7981 Dec 19 16:49 db_input.c
-rw-r--r-- 1 root wheel 7931 Jan 6 2005 db_input.c.orig
-rw-r--r-- 1 root wheel 5978 Dec 19 16:49 db_lex.c
-rw-r--r-- 1 root wheel 5304 Jan 6 2005 db_lex.c.orig
-rw-r--r-- 1 root wheel 1951 Dec 19 16:49 db_lex.h
-rw-r--r-- 1 root wheel 1861 Jan 6 2005 db_lex.h.orig
-rw-r--r-- 1 root wheel 5976 Dec 19 16:49 db_main.c
-rw-r--r-- 1 root wheel 5787 Nov 6 2006 db_main.c.orig
-rw-r--r-- 1 root wheel 6889 Dec 19 16:49 db_output.c
-rw-r--r-- 1 root wheel 6639 Oct 10 2006 db_output.c.orig
-rw-r--r-- 1 root wheel 1444 Oct 8 2006 db_output.h
-rw-r--r-- 1 root wheel 2013 Apr 14 2005 db_print.c
-rw-r--r-- 1 root wheel 10956 Nov 13 13:43 db_ps.c
-rw-r--r-- 1 root wheel 8810 Apr 14 2005 db_run.c
-rw-r--r-- 1 root wheel 7815 Jun 16 2006 db_sym.c
-rw-r--r-- 1 root wheel 3591 Jan 6 2005 db_sym.h
-rw-r--r-- 1 root wheel 5143 Jan 17 2007 db_thread.c
-rw-r--r-- 1 root wheel 3402 Jan 6 2005 db_variables.c
-rw-r--r-- 1 root wheel 1854 Jan 6 2005 db_variables.h
-rw-r--r-- 1 root wheel 7218 Nov 17 2006 db_watch.c
-rw-r--r-- 1 root wheel 1509 Jan 6 2005 db_watch.h
-rw-r--r-- 1 root wheel 2264 Jan 6 2005 db_write_cmd.c
-rw-r--r-- 1 root wheel 7467 Dec 19 16:49 ddb.h
-rw-r--r-- 1 root wheel 5776 Jul 12 2006 ddb.h.orig
--
======================================================================
- Best regards, Nikolay Pavlov. <<<-----------------------------------
======================================================================
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
Url : http://lists.freebsd.org/pipermail/freebsd-current/attachments/20071220/d893b2a8/attachment-0001.pgp
More information about the freebsd-current
mailing list