DDB scripting, output capture, and textdumps

Nikolay Pavlov qpadla at gmail.com
Thu Dec 20 05:32:53 PST 2007


On Tuesday 18 December 2007 14:10:46 Robert Watson wrote:
> Dear all:
>
> I've been hacking on-and-off for a while on a side project to improve
> our kernel debugging facilities.  Primarily, my concern has been to
> address three problems:
>
> - The complications of employing kernel core dumps for debugging,
>    including the large size of dumps making them unwieldy to distribute
> or store for any extended period (even with minidumps), the requirement
> to have relatively synchronized kernel source in order to use the dumps,
> the need to have a kernel with debugging symbols, and the problems with
> fsck causing sufficient swap use to invalidate dumps before they can be
> extracted.
>
> - The decreasing likelihood that notebooks will ship with serial ports
>    that can be used for interactive debugging using DDB.  Making
> end-users type in stack traces is cruel, photos are a pain, and X11
> rules out both.
>
> - The fact that a great many problems are most easily diagnosed using
>    utility routines present in DDB, but not as easily using kgdb for
>    offline analysis.  I find that for many bugs I analyze, simply
> looking at the DDB output is sufficient to identify the source of the
> problem.
>
> An idea I punted around a bit at BSDCan earlier this year (or perhaps it
> was at EuroBSDCon the previous year) was an idea of a "textdump" -- that
> is, a new type of kernel dump based on capturing automatically extracted
> debugging information generated by DDB.  The result would be an ASCII
> text file that could be filed as a bug report, perhaps even
> automatically.
>
> To this end, I have implemented three new facilities for use with DDB:
>
> (1) DDB output capture.  The output of DDB is stored in a memory buffer,
>      and can be extracted using a sysctl or textdumps (see below).  This
>      can be turned on and off, both for use manually ("I'll want this
>      later, but not that") and as part of scripts (see below).
>
> (2) DDB scripting.  A limited number of named scripts can be defined to
>      run a series of DDB commands.  No loops, etc, just simple command
>      lists.  These can be caused to run automatically on entering DDB
> for various scenarios, including WITNESS violations and kernel panics.
> They can also be run by hand in order to save a bit of typing if you use
> DDB in a repetitive way (as I do).
>
> (3) Textdumps.  A new dump type that stores a series of data files
>      containing various pieces of information, including the DDB capture
>      buffer, kernel message buffer, kernel configuration (if compiled
> into the kernel), panic message, and kernel version string.  These are
> stored in the ustar format inside the dump partition (aligned to the
> end) so can be easily extended, and savecore(8) requires almost no new
> logic to deal with them (it just drops numbered tar files in
> /var/crash).  This makes it straight forward to extend the textdump
> format to include new types of information and avoids the issue of how
> to safely simultaneously represent information in many different formats
> in the same file.
>
> These are pretty flexible tools, and you can imagine doing the following
> sorts of things:
>
> - Setting the kdb.enter.panic script to automatically turn on output
>    capture, do full backtraces of all threads, show open file
> information, dump UMA stats, and save it all to a textdump and then
> reboot.
>
> - Setting the kdb.enter.witness script to show lock information,
> generate a coredump, and reboot.  Or, just to automatically do "show
> allocks" and drop to the DDB prompt.
>
> - Adding a flag to rc.conf to automatically submit textdumps via e-mail
> to a specific address, perhaps including GNATS or an automated bug
> system. These could be unpacked and automatically analyzed, and do to
> the compact size, kept for long-term trend analysis or to identify when
> a problem started occuring.
>
> I've produced an initial snapshot of the above, which can be found here:
>
>    http://www.watson.org/~robert/freebsd/20071218-ddb.tgz
>
> This adds three files to DDB, patches quite a few kernel files (to pass
> more information into KDB about why it's being entered, in order to
> trigger the right script), enhancements to savecore(8) to know how to
> extract textdumps, adds a ddb(8) command line tool so that userspace can
> manage DDB scripts from outside the debugger, extensions to the ddb(4)
> man page, and a new textdump(4) man page.
>
> There are a number of known limitations; I've tried to document them at
> the top of the pertinent files where I am aware of them.  I also regret
> to say that to date I've been able to test only on i386, and not other
> platforms.  I'd welcome any feedback -- I'd like to get these changes
> into CVS in the next week or two.

It looks like some files is not included in the patch.
I have this error:

make -V CFILES -V SYSTEM_CFILES -V GEN_CFILES |  MKDEP_CPP="cc -E" CC="cc" 
xargs 
mkdep -a -f .newdep -O -pipe  -std=c99 -g -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes  -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual  -Wundef -Wno-pointer-sign -fformat-extensions -nostdinc  -I. -I/usr/src/sys -I/usr/src/sys/contrib/altq -I/usr/src/sys/contrib/ipfilter -I/usr/src/sys/contrib/pf -I/usr/src/sys/dev/ath -I/usr/src/sys/contrib/ngatm -I/usr/src/sys/dev/twa -I/usr/src/sys/gnu/fs/xfs/FreeBSD -I/usr/src/sys/gnu/fs/xfs/FreeBSD/support -I/usr/src/sys/gnu/fs/xfs -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include 
opt_global.h -fno-common -finline-limit=8000 --param 
inline-unit-growth=100 --param 
large-function-growth=1000  -mno-align-long-strings -mpreferred-stack-boundary=2  -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -ffreestanding
cc: /usr/src/sys/ddb/db_capture.c: No such file or directory
cc: /usr/src/sys/ddb/db_script.c: No such file or directory
cc: /usr/src/sys/ddb/db_textdump.c: No such file or directory
mkdep: compile failed
*** Error code 1

Stop in /usr/obj/usr/src/sys/GENERIC.
*** Error code 1

Stop in /usr/src.
*** Error code 1

Stop in /usr/src.

root at orion-vm:/usr/src# ls -la /usr/src/sys/ddb/                                                                                                                             
total 424
drwxr-xr-x   2 root  wheel   1024 Dec 19 16:49 ./
drwxr-xr-x  53 root  wheel   1024 Oct 29 08:24 ../
-rw-r--r--   1 root  wheel   2591 Dec  4  2005 db_access.c
-rw-r--r--   1 root  wheel   1431 Jan  6  2005 db_access.h
-rw-r--r--   1 root  wheel   7737 Jan  6  2005 db_break.c
-rw-r--r--   1 root  wheel   2098 Jan  6  2005 db_break.h
-rw-r--r--   1 root  wheel  16579 Dec 19 16:49 db_command.c
-rw-r--r--   1 root  wheel  15705 Jan 17  2007 db_command.c.orig
-rw-r--r--   1 root  wheel   1633 Dec 19 16:49 db_command.h
-rw-r--r--   1 root  wheel   1588 Jan  6  2005 db_command.h.orig
-rw-r--r--   1 root  wheel   7270 Oct 27 20:19 db_examine.c
-rw-r--r--   1 root  wheel   4811 Jan  6  2005 db_expr.c
-rw-r--r--   1 root  wheel   7981 Dec 19 16:49 db_input.c
-rw-r--r--   1 root  wheel   7931 Jan  6  2005 db_input.c.orig
-rw-r--r--   1 root  wheel   5978 Dec 19 16:49 db_lex.c
-rw-r--r--   1 root  wheel   5304 Jan  6  2005 db_lex.c.orig
-rw-r--r--   1 root  wheel   1951 Dec 19 16:49 db_lex.h
-rw-r--r--   1 root  wheel   1861 Jan  6  2005 db_lex.h.orig
-rw-r--r--   1 root  wheel   5976 Dec 19 16:49 db_main.c
-rw-r--r--   1 root  wheel   5787 Nov  6  2006 db_main.c.orig
-rw-r--r--   1 root  wheel   6889 Dec 19 16:49 db_output.c
-rw-r--r--   1 root  wheel   6639 Oct 10  2006 db_output.c.orig
-rw-r--r--   1 root  wheel   1444 Oct  8  2006 db_output.h
-rw-r--r--   1 root  wheel   2013 Apr 14  2005 db_print.c
-rw-r--r--   1 root  wheel  10956 Nov 13 13:43 db_ps.c
-rw-r--r--   1 root  wheel   8810 Apr 14  2005 db_run.c
-rw-r--r--   1 root  wheel   7815 Jun 16  2006 db_sym.c
-rw-r--r--   1 root  wheel   3591 Jan  6  2005 db_sym.h
-rw-r--r--   1 root  wheel   5143 Jan 17  2007 db_thread.c
-rw-r--r--   1 root  wheel   3402 Jan  6  2005 db_variables.c
-rw-r--r--   1 root  wheel   1854 Jan  6  2005 db_variables.h
-rw-r--r--   1 root  wheel   7218 Nov 17  2006 db_watch.c
-rw-r--r--   1 root  wheel   1509 Jan  6  2005 db_watch.h
-rw-r--r--   1 root  wheel   2264 Jan  6  2005 db_write_cmd.c
-rw-r--r--   1 root  wheel   7467 Dec 19 16:49 ddb.h
-rw-r--r--   1 root  wheel   5776 Jul 12  2006 ddb.h.orig


-- 
======================================================================  
- Best regards, Nikolay Pavlov. <<<-----------------------------------    
======================================================================  

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
Url : http://lists.freebsd.org/pipermail/freebsd-arch/attachments/20071220/d893b2a8/attachment.pgp


More information about the freebsd-arch mailing list