FreeBSD NFS client goes into infinite retry loop

[Steve Polyack]

On 03/19/10 09:23, Steve Polyack wrote:
> On 03/19/10 08:31, John Baldwin wrote:
>> On Friday 19 March 2010 7:34:23 am Steve Polyack wrote:
>>> Hi, we use a FreeBSD 8-STABLE (from shortly after release) system as an
>>> NFS server to provide user home directories which get mounted across a
>>> few machines (all 6.3-RELEASE).  For the past few weeks we have been
>>> running into problems where one particular client will go into an
>>> infinite loop where it is repeatedly trying to write data which causes
>>> the NFS server to return "reply ok 40 write ERROR: Input/output error
>>> PRE: POST:".  This retry loop can cause between 20mbps and 500mbps of
>>> constant traffic on our network, depending on the size of the data
>>> associated with the failed write.
>>>
>>> We spent some time on the issue and determined that something on one of
>>> the clients is deleting a file as it is being written to by another NFS
>>> client.  We were able to enable the NFS lockmgr and use lockf(1) to fix
>>> most of these conditions, and the frequency of this problem has dropped
>>> from once a night to once a week.  However, it's still a problem and we
>>> can't necessarily force all of our users to "play nice" and use 
>>> lockf/flock.
>>>
>>> Has anyone seen this before?  No errors are being logged on the NFS
>>> server itself, but the "Server Ret-Failed" counter begins to increase
>>> rapidly whenever a client gets stuck in this infinite retry loop:
>>> Server Ret-Failed
>>>           224768961
>>>
>>> I have a feeling that using NFS in such a matter may simply be prone to
>>> such problems, but what confuses me is why the NFS client system is
>>> infinitely retrying the write operation and causing itself so much 
>>> grief.
>> Yes, your feeling is correct.  This sort of race is inherent to NFS 
>> if you do
>> not use some sort of locking protocol to resolve the race.  The infinite
>> retries sound like a client-side issue.  Have you been able to try a 
>> newer OS
>> version on a client to see if it still causes the same behavior?
>>
> I can't try a newer FBSD version on the client where we are seeing the 
> problems, but I can recreate the problem fairly easily.  Perhaps I'll 
> try it with an 8.0 client.  If I remember correctly, one of the 
> strange things is that it doesn't seem to hit "critical mass" until a 
> few hours after the operation first fails.  I may be wrong, but I'll 
> double check that when I check vs. 8.0-release.
>
> I forgot to add this in the first post, but these are all TCP NFS v3 
> mounts.
>
> Thanks for the response.

Ok, so I'm still able to trigger what appears to be the same retry loop 
with an 8.0-RELEASE nfsv3 client (going on 1.5 hours now):
$ cat nfs.sh
client#!/usr/local/bin/bash
for a in {1..15} ; do
   sleep 1;
   echo "$a$a$";
done
client$ ./nfs.sh >~/output

the on the server while the above is running:
server$ rm ~/output

What happens is that you will see 3-4 of the same write attempts happen 
per minute via tcpdump.  Our previous logs show that this is how it 
starts, and then ~4 hours later it begins to spiral out of control, 
throwing out up to 3,000 of the same failed write requests per second.