panic ffs_truncate3 (maybe fuse being evil)

[Konstantin Belousov]

On Sun, Jan 10, 2016 at 10:01:57AM -0500, Rick Macklem wrote:
> Hi,
> 
> When fooling around with GlusterFS, I can get this panic intermittently.
> (I had a couple yesterday.) This happens on a Dec. 5, 2015 head kernel.
> 
> panic: ffs_truncate3
> - backtrace without the numbers (I just scribbled it off the screen)
> ffs_truncate()
> ufs_inactive()
> VOP_INACTIVE_APV()
> vinactive()
> vputx()
> kern_unlinkat()
> 
> So, at a glance, it seems that either
>    b_dirty.bv_cnt
> or b_clean.bv_cnt
> is non-zero. (There is another case for the panic, but I thought it
>               was less likely?)
> 
> So, I'm wondering if this might be another side effect of r291460,
> since after that a new vnode isn't completely zero'd out?
> 
> However, shouldn't bo_dirty.bv_cnt and bo_clean.bv_cnt be zero when
> a vnode is recycled?
> Does this make sense or do some fields of v_bufobj need to be zero'd
> out by getnewvnode()?
Look at the _vdrop().  When a vnode is freed to zone, it is asserted
that bufobj queues are empty.  I very much doubt that it is possible
to leak either buffers or counters by reuse.

> 
> GlusterFS is using fuse and I suspect that fuse isn't cleaning out
> the buffers under some circumstance (I already noticed that there
> isn't any code in its fuse_vnop_reclaim() and I vaguely recall that
> there are conditions where VOP_INACTIVE() gets skipped, so that VOP_RECLAIM()
> has to check for anything that would have been done by VOP_INACTIVE()
> and do it, if it isn't already done.)
But even if fuse leaves the buffers around, is it UFS which panics for
you ? I would rather worry about dandling pointers and use after free in
fuse, which is a known issue with it anyway. I.e. it could be that fuse
operates on reclaimed and reused vnode as its own.

> 
> Anyhow, if others have thoughts on this (or other hunches w.r.t. what
> could cause this panic(), please let me know.

The ffs_truncate3 was deterministically triggered by a bug in ffs_balloc().
The routine allocated buffers for indirect blocks, but if the blocks cannot
be allocated, the buffers where left on queue.  See r174973, this was fixed
very long time ago.