8.2-PRERELEASE: live deadlock, almost all processes in "pfault" state

[Kostik Belousov]

On Sat, Jan 08, 2011 at 10:29:09PM +0300, Lev Serebryakov wrote:
> Hello, Kostik.
> You wrote 8 января 2011 г., 22:02:32:
> 
> 
> > There is some weird backtrace at the pid 20, what is g_raid5 ?
>   It is geom_raid5, with two threads -- working one and one for
>  processing finished bios.
> 
> > If I am guessing right, this creature has a classic deadlock when 
> > bio processing requires memory allocation. It seems that tid 100079
>   tid 100079 sleep in waiting for some data in queue.
> 
> > is sleeping not even due to the free page shortage, but due to address
> > space exhaustion. As result, read/write requests are stalled.
>   tid 100078 sleep in malloc(). But geom_raid5 never ever allocate
>  more than 128MiB of memory and it is 64bit system with huge amount of
>  kmem_size/kmem_size_max...
> 
>   How could I explore allocation (like vmstat -m) from kdb to be sure,
> it doesn't allocated more?
Use "show uma" and "show malloc" from ddb.

> 
>   And, if it is "classic deadlock" is here any "classical" solution to
> it?
Do not allocate during bio processing.

> 
>   Really, I'm maintainer of geom_raid5 now, so I need fix this
> deadlock, but I don't really understand, why does it occur? I've
> hit panic with "kernel memory exhausted" symptoms when module allocate
> too much, but not deadlock :(
Hm, I missed the kmem_back() in the stack. Yes, the thread is waiting for page
allocation.

> 
> > Then, syncer is blocked waiting for some physical buffer (look at tid
> > 100075), owning the vnode lock. Other processes also wait for the
> > locked buffers, etc.
> 
> > So my belief is that this is plain driver (g_raid5, whatever is it)
> > i/o loss. Try the same load without it.
>   I can not, because all data is on this GEOM :)
> 
> -- 
> // Black Lion AKA Lev Serebryakov <lev at serebryakov.spb.ru>
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20110108/52b2720e/attachment.pgp