Hast locking up under 9.2

[Mikolaj Golub]

On Mon, Nov 25, 2013 at 09:32:23AM +0100, Pawel Jakub Dawidek wrote:
> On Sat, Nov 23, 2013 at 11:59:51PM +0200, Mikolaj Golub wrote:
> > On Fri, Nov 22, 2013 at 11:18:29AM +0000, Pete French wrote:
> > 
> > > "Assertion failed: (!hio->hio_done), function write_complete, file
> > >  /usr/src/sbin/hastd/primary.c, line 1130."
> > 
> > It looks like write_complete usage (which should be called once per
> > write request) for memsync is racy.
> > 
> > Consider the following scenario:
> > 
> > 1) remote_recv_thread: memsync ack received, refcount -> 2;
> > 2) local_send_thread: local write completed, refcount -> 1, entering
> >    write_complete()
> > 3) remote_recv_thread: memsync fin received, refcount -> 0, move hio
> >    to done queue, ggate_send_thread gets the hio, checks for
> >    !hio->hio_done and (if loca_send_thread is still in
> >    write_complete()) entering write_complete()
> 
> I don't see how is that possible. The write_complete() function is
> called only when hio_countdown goes from 2 to 1 and because this is
> atomic operation it can only happen in one thread. Can you elaborate on
> how calling write_complete() concurrently for the same request is
> possible?

Yes, hio_countdown protects calling write_complete() concurently by
"component" threads. But it may also be called by ggate_send_thread():

	if (!hio->hio_done)
		write_complete(res, hio);

So if write_complete() has already started executing in
local_send_thread(), and at that time memsync fin is received, the
request is moved to ggate_send_thread, and write_complete can be
reentered if it is still in progress in local_send_thread (hio_done is
set on exiting write_complete).

That is why statement (3) in my patch: write_complete() in component
threads is called only before releasing hio_countdown. Otherwise you
are not protected from running it simultaneously by ggate_send_thread,
or even hio be moved to free before write_complete is finished in
local_send_thread. And so hio_countdown can't be used for detecting
the current memsync state.

-- 
Mikolaj Golub