panic ffs_truncate3 (maybe fuse being evil)

[Rick Macklem]

Kostik wrote:
> On Sun, Jan 10, 2016 at 10:01:57AM -0500, Rick Macklem wrote:
> > Hi,
> > 
> > When fooling around with GlusterFS, I can get this panic intermittently.
> > (I had a couple yesterday.) This happens on a Dec. 5, 2015 head kernel.
> > 
> > panic: ffs_truncate3
> > - backtrace without the numbers (I just scribbled it off the screen)
> > ffs_truncate()
> > ufs_inactive()
> > VOP_INACTIVE_APV()
> > vinactive()
> > vputx()
> > kern_unlinkat()
> > 
> > So, at a glance, it seems that either
> >    b_dirty.bv_cnt
> > or b_clean.bv_cnt
> > is non-zero. (There is another case for the panic, but I thought it
> >               was less likely?)
> > 
> > So, I'm wondering if this might be another side effect of r291460,
> > since after that a new vnode isn't completely zero'd out?
> > 
> > However, shouldn't bo_dirty.bv_cnt and bo_clean.bv_cnt be zero when
> > a vnode is recycled?
> > Does this make sense or do some fields of v_bufobj need to be zero'd
> > out by getnewvnode()?
> Look at the _vdrop().  When a vnode is freed to zone, it is asserted
> that bufobj queues are empty.  I very much doubt that it is possible
> to leak either buffers or counters by reuse.
> 
Ok. I'll take a look but, yes, it doesn't sound like the fields could
be left bogus when the vnode gets recycled.

> > 
> > GlusterFS is using fuse and I suspect that fuse isn't cleaning out
> > the buffers under some circumstance (I already noticed that there
> > isn't any code in its fuse_vnop_reclaim() and I vaguely recall that
> > there are conditions where VOP_INACTIVE() gets skipped, so that
> > VOP_RECLAIM()
> > has to check for anything that would have been done by VOP_INACTIVE()
> > and do it, if it isn't already done.)
> But even if fuse leaves the buffers around, is it UFS which panics for
> you ? I would rather worry about dandling pointers and use after free in
> fuse, which is a known issue with it anyway. I.e. it could be that fuse
> operates on reclaimed and reused vnode as its own.
> 
> > 
> > Anyhow, if others have thoughts on this (or other hunches w.r.t. what
> > could cause this panic(), please let me know.
> 
> The ffs_truncate3 was deterministically triggered by a bug in ffs_balloc().
> The routine allocated buffers for indirect blocks, but if the blocks cannot
> be allocated, the buffers where left on queue.  See r174973, this was fixed
> very long time ago.
> 
But this was a one month old kernel (around r291900, although I don't know the
exact r#, but it was Dec. 5, 2015), so it definitely has this fix in it.

When I see it again, I will try and see what the v_bufobj fields look like.

Thanks, rick