sbflush() panics, new socket-related DDB commands (was: cvs commit: src/sys/conf files src/sys/kern uipc_debug.c (fwd))

[John Utz]

On Thu, 15 Feb 2007 10:12:59 +0000 (GMT)
Robert Watson <rwatson at FreeBSD.org> wrote:

> 
> Dear All,
> 
> Of late we've seen an increase in the number of reports of panics in 
> sbflush().  These occur due to an invariant check when closing
> sockets, which in effect confirms that the amount of actual data in a
> socket buffer matches the socket buffer's cached length from that
> data.  Typically, these panics are a symptom of mbuf chain/mbuf
> packet queue corruption, and in the past have been symptomatic
> primarily of device driver bugs (i.e., the device driver touches mbuf
> fields after passing the mbuf into the stack, etc).  However,
> debugging these problems is very difficult, as sbflush() is called
> only when the socket is closed, and the corruption frequently occurs
> much earlier.

It's possibly worth pointing out that i can get either the same or a
remarkably similar panic 100% reproducibly by using usbaudio. As soon
as mpg321 finishes the song and closes the device i get a panic. 100%
reproducible.

the panic does *not* occur if i use the snd_ich stuff instead of
usbaudio.

i last synced at the following:

minime# uname -a
FreeBSD minime.local 7.0-CURRENT FreeBSD 7.0-CURRENT #20: Mon Feb 12
07:49:10 PST 2007
spaz at minime.local:/usr/src/sys/i386/compile/MINIME  i386


I havent reported it previously because i thought it was my own little
weird problem and i've been attempting to get a more specific failure
point beyond 'device close'

> If you are experiencing these panics, please compile your kernel with
> "options SOCKBUF_DEBUG", which adds more frequent invariant checks of
> socket buffer consistency.  These checks may have a substantial
> performance impact, but unless we can catch the problem(s) closer to
> when it(they) occur(s), the chances of tracking the source are low.
> We could be looking at one or more bugs in some combination of the
> socket, protocol, and network interface code leading to a recent
> increase.  Perhaps socket buffer auto-sizing has increased the
> chances of it occuring, or perhaps we just have increase parallelism
> in test hardware.

Dont think it's network code because it get's tickled by
closing usbaudio.

> I've added some new commands to DDB to make it easier to understand
> the state of the system following panic.  While the same information
> can be extracted with kgdb, core dumps are not reliably available in
> all environments, and certain other information (such as lock
> information) is most easily obtained using DDB.  The commands are:
> 
> show socket <addr>
> show sockbuf <addr>
> show protosw <addr>
> show domain <addr>
> 
> Only the first two of these are likely to be used directly, and they
> invoke protosw and domain printing as required.  It would be helpful
> if people debugging network panics involving sockets could use the
> above (and especially show socket) on socket arguments in stack
> traces.  For example, in an sbflush() panic, the first argument to
> soclose() in the stack trace is the socket pointer, so is ideal to
> pass to "show socket" :-).
> 
> FYI, in general, the first argument to sofoo() functions is the
> socket pointer.  Likewise, the first argument to sbfoo() functions is
> the socket buffer pointer.  This is not the case of soo_foo()
> functions, which take a file descriptor, or filt_sofoo() functions,
> which accept a knote.  In general, if you print the socket contents
> with "show socket", then it isn't necessary to print socket buffers
> separately; "show socket" prints not just the contents of the socket
> buffers, but also the pointers to them so that those pointers can be
> compared with other arguments in the stack.

I will endeavor to resync and rebuild and see if i can provide a more
specific failure point with the usb stuff, but i will not be able to
tackle it before the weekend because this laptop is my only computer
(and i run -current on it, suprisingly well for the most part!)

> Robert N M Watson
> Computer Laboratory
> University of Cambridge
> 
> ---------- Forwarded message ----------
> Date: Thu, 15 Feb 2007 01:28:23 +0000 (UTC)
> From: Robert Watson <rwatson at FreeBSD.org>
> To: src-committers at FreeBSD.org, cvs-src at FreeBSD.org,
> cvs-all at FreeBSD.org Subject: cvs commit: src/sys/conf files
> src/sys/kern uipc_debug.c
> 
> rwatson     2007-02-15 01:28:22 UTC
> 
>    FreeBSD src repository
> 
>    Modified files:
>      sys/conf             files
>    Added files:
>      sys/kern             uipc_debug.c
>    Log:
>    Teach DDB how to print sockets, socket buffers, protosw's, and
> domain structures given pointers to them.
> 
>    Revision  Changes    Path
>    1.1177    +1 -0      src/sys/conf/files
>    1.1       +530 -0    src/sys/kern/uipc_debug.c (new)
> _______________________________________________
> freebsd-current at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to
> "freebsd-current-unsubscribe at freebsd.org"
>