HAST: primary might get stuck when there are connectivity problems with secondary

[Pawel Jakub Dawidek]

On Sun, Apr 25, 2010 at 02:17:15PM +0300, Mikolaj Golub wrote:
> On Sat, 24 Apr 2010 14:33:53 +0300 Mikolaj Golub wrote:
> 
> > From the code I don't see how hast_proto_recv_hdr() may timeout if the
> > connection is alive, have I missed something?
> 
> I did some experiments adding the code that sets SO_RCVTIMEO socket option
> (see the attached patch). It fixes this issue. After timeout the worker on the
> secondary is restarted with the error:
> 
> Apr 25 13:06:45 hastb hastd: [storage] (secondary) Unable to receive request header: Resource temporarily unavailable.
> Apr 25 13:06:45 hastb hastd: [storage] (secondary) Worker process (pid=1243) exited ungracefully: status=19200.
> 
> On the other hand when the FS is idle (there is no I/O at all) we have the
> worker restart too and the primary is not being connected to the secondary
> until some I/O appears. So it might look not very nicely :-)
> 
> Also note, I had to modify proto_common_recv() to have timeout working. After
> timeout recv() sets errno to EWOULDBLOCK, which has the same number as EAGAIN
> in FreeBSD. The current proto_common_recv() restarts recv() if EAGAIN is
> returned.

Could you see if the following patch fixes the problem for you:

	http://people.freebsd.org/~pjd/patches/hastd_timeout.patch

The patch sets timeout on both incoming and outgoing sockets on primary
and on outgoing socket on secondary. Incoming socket on secondary is
left with no timeout to avoid problem you described above.

-- 
Pawel Jakub Dawidek                       http://www.wheelsystems.com
pjd at FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20100428/6ed749cd/attachment.pgp