System deadlock when using mksnap_ffs

[Doug Ambrisko]

Kostik Belousov writes:
| On Thu, Nov 13, 2008 at 02:45:14AM -0800, Jeremy Chadwick wrote:
	[snip]
| > If he can press Control-T, it means SIGINFO can be sent to the
| > mksnap_ffs process, and the process responds with that information.  So,
| > the system is not deadlocked -- meaning, I believe what he experiences
| > is what others experience (the system becomes completely unusable during
| > mksnap_ffs running, but DOES NOT hang or lock up, it just becomes so
| > god-awful slow that processes on the machine literally sit and spin for
| > minutes at a time).
| 
| Unless NOKERNINFO is specified in the local flags in the controlling
| terminal termios, kernel prints one line summary as shown above. This is
| done from the tty discipline input handler (or whatever it is in new tty
| code). No process cooperation is required. On the other hand, actually
| delivering SIGINFO and getting output from the process-installed
| handler do require process to either executing usermode or sleeping
| interruptible.

Also note that "dead-lock" is not just a locking issue but can be
WRT to other chains such as, hit the max buffer cache usage so the
buffer daemon needs to flush things out but it can't since it needs
a buffer but the buffer daemon can't get it since need to flush some.
Things get really bad when the buffer daemon needs a buffer but
can't!  In theory it can go and use "emergency space" just for it
to get out of this situation but it the buffer cache is fragmented
such that all available buffers are to small then the buffer daemon
is stuck on itself.  Note that all stuff works except for anything
that touch the buffer cache, such as a program coming off disk.  A
program in memory is okay.

To really get a good picture of this you need to look at the 
various buffer cache variables via ddb (ie. hi, low, running etc.)
A while back I wrote a debugging function to dump that state of
things every minute or so.  There are various loops you can get into.
So then you start playing wack a mole.  Usually due to the first
bug you can't hit the 2nd, 3rd and so one adding to the fun.
Unfortunately there isn't one magic bullet.

These are not new problems since we hit them in 4.X.  I did start
to go over some of this issue with Tor but ran into ENOTIME on my
side :-(

Snap shots can take a very long time to make depending on the amount 
of stuff it has to snap shot and during that time it has to effectively
lock out everything from the file system or the snap shot will be 
wrong.  This then leads to a need for a good journaling fs that
can be used on "big" disks (big, isn't that big anymore).

Doug A.