[OpenAFS-devel] Re: AFS ... or equivalent ...

[Jan Harkes]

On Thu, Jan 17, 2008 at 08:11:35AM +0000, Robert Watson wrote:
> that more fixes are required to make Coda stable and supportable.  I've 
> added Jan to the CC, perhaps he can opine on what further work is 
> required, and if I missed any further patches he sent.  As with Arla, 

You got everything that was in my queue. That doesn't mean that all
problems have been resolved, but I could walk through the file system
and read and write files without problems.

I notice all the other mails, I'll have to look at those.

> for some period (perhaps still), Coda worked poorly from behind a NAT, a 
> common property of many development environments.

It isn't so much that Coda works poorly behind a NAT, but that a lot of
masquerading firewalls assume that the only UDP traffic is DNS lookups
and maybe NFS.

Which means that they do unexpected things when one side starts to send
several packets in rapid succession to the same destination. I've seen
firewalls that drop everything but the first packet until a reply comes
back. Or they only allow a single "reply" packet direction after a
request. Both of these flaws break our windowed data tranfers because a
single ack will move the window which can result in several outgoing UDP
packets.

Another interesting case is firewalls that don't seem to hash on the
source addr,port,destination address, port tuple, but only remember the
last destination we sent to. So if we send a message to several
(replicated) servers, we only receive a response from a single one (the
last one we talked to). Of course we retry the request to the others,
so eventually we get all answers. But it isn't really very efficient
we end up waiting for the estimated RTT and send 50% more data if
we have 2 replicas. Which affects the estimated RTT, making the next
round even slower.

A more common scenario is a too short period during which the
connection/redirection is considered active. Some masquerading firewalls
have a default value that may be as low as 60 seconds. This means we
have to send at least one packet every 60 seconds, there could be packet
loss, so possibly even more often than that, just to keep the connection
alive. If our packet reaches the firewall outside of this window, it
will either drop the packet, or allocate a new port so that the receiver
thinks this is a connection from a different client. The client ends up
having to revalidate everything in it's cache to make sure it didn't
miss a callback message. Originally a connection keepalive probe only
happened once every 15 minutes if we hadn't heard anything else, the
default is now 150 seconds because a lot of off-the-shelf wireless
routers/nat firewalls seem to have a roughly 3 minute timeout.

And these issues will affect any protocol over UDP that isn't a plain
stateless request-response variant, and in some cases (f.i. dropping
everything but the first packet in a sequence) are broken in all cases
because a packet can be dropped/lost or corrupted after passing the
firewall and the firewall isn't letting the retransmissions through.
But that only happens once in a blue moon, and will in most cases just
show as a stalled/failed DNS lookup and people just hit 'try again'.

Jan