bin/121684: : dump(8) frequently hangs

[Robert Watson]

On Mon, 1 Sep 2008, Jeremy Chadwick wrote:

> On Tue, Sep 02, 2008 at 09:39:20AM +0300, Danny Braniss wrote:
>> take a look at:
>> 	http://www.freebsd.org/cgi/query-pr.cgi?pr=117603 danny
>
> That PR may or may not be relevant, depending upon what FreeBSD version 
> users are using, and what kernel build date.
>
> The bug mentioned in that PR got addressed in HEAD on March 13th, 2008, and 
> the fix MFC'd to RELENG_7 on April 19th, 2008.  It was never MFC'd to 
> RELENG_6.
>
> If there are users on RELENG_7 with kernels built with sources after April 
> 19th 2008 who are experiencing the problem, then the PR is probably not 
> relevant.

Part of the "problem" here is that a whole class of possible bugs have 
near-identical symptoms.  While each of the following means quite different 
things to a kernel developer working on the problem, and may reflect quite 
different types of bugs, they are all often described as "dump hangs" or "dump 
wedges":

- the system deadlocks
- dump fails to complete and/or exit, but cannot be killed
- dump fails to complete and/or exit, but can be killed

When snapshots were first introduced, the problems tended to be at the top end 
of that list, corresponding to VFS locking and resource starvation deadlocks. 
As the snapshot code has matured, new problems in both the kernel scheduler 
and dump code have arisen as parallelism has increased, reflecting a 
combination of old bugs in the dump code and new bugs in the kernel scheduler. 
Unfortunately, these bugs don't tend to get discovered much during testing in 
-CURRENT -- perhaps people don't back up their -CURRENT boxes much :-).

I think we need to rigorously do the following:

- For each bug report, determine whether it is reporting one or more of the
   above types of "hangs".  If multiple types are reported, track them with
   different bug reports.

- Establish as early as possible whether a fix resolves the problems in each
   report.  Because we're dealing with many bugs over time, it's possible to
   end up with accidentally "omnibus" reports that remain open and are never
   closed, even though committed and released fixes may correct the problems
   experienced by the reporter.  It is almost impossible, btw, to rewind and
   years later determine if any particular fix would have corrected any
   particular report, because the original submitter will have moved on.

Dump happens to be particularly sensitive to bugs of these sorts because it 
uses snapshots and it uses multiple workers that signal each other, so it's a 
good lithmus test of stability of both of those features.  However, it's easy 
to conclude that dump is much less stable than it proves in practice because 
we have a lot of stale and confusing bug reports.  What we do need is a dump 
bug report owner, who can keep track of the outstanding set, try to 
agressively close the ones that are fixed, which will among other things allow 
us to better track regressions vs bugs from inception.

Robert N M Watson
Computer Laboratory
University of Cambridge