zfs recv causes nfs server to throw NFSERR_IO i/o errors

[Daniel Genis]

Thank you, that probably is what the doctor ordered!
My quick testing shows that it's very likely fixed. Kudos! :-)

On 08/05/2016 04:11 PM, Alan Somers wrote:
> On Fri, Aug 5, 2016 at 7:22 AM, Daniel Genis <daniel.genis at gmx.de> wrote:
>> Hi everyone,
>>
>> we've been tracing an issue where snapshot replication is causing
>> interruptions for the NFS serivce.
>>
>> The problem is as follows:
>>
>> Every time a zfs recv finishes, there is a chance for the NFS server to
>> return an NFSERR_IO for a GETATTR call. This shows up as input/output
>> errors on the nfs clients.
>>
>> Here the tcpdump showing the NFS conversation:
>> https://nopaste.me/view/95d1a79d
>>
>> NFS 202 V3 GETATTR Call (Reply In 6043), FH: 0x8c711a60
>> NFS 98  V3 GETATTR Reply (Call In 6042) Error: NFS3ERR_IO
>> NFS 222 V3 LOOKUP Call (Reply In 6046), DH: 0x6694634f/example.file.txt
>> NFS 102 V3 LOOKUP Reply (Call In 6045) Error: NFS3ERR_ACCES
>>
>> We've been able to verify that there is a _direct_ correlation between
>> the zfs recv command and these NFS errors. For every input/output error
>> we can find a log entry of a replication just finishing (zfs recv exiting).
>>
>> The receiving server is running 10.3-RELEASE
>>
>> I've read about a VFS/ZFS deadlock issue which is to be included/fixed
>> in Freebsd 11.0-BETA4.
>>
>> Could our issue be related?
>> Otherwise does anyone have any suggestions how to tackle this issue?
>>
>>
>> For the record, say we have two volumes:
>> tank/volumeA and tank/volumeB
>>
>> If there is a zfs recv busy for tank/volumeA then tank/volumeB can get
>> these NFS "io" errors, it does not have to be the same volume.
>>
>>
>> Has anyone else seen/experience this as well?
>>
>> Any insights are appreciated!
>>
>> With kind regards,
>>
>> Daniel
> Try adding mountd_flags="-S" to /etc/rc.conf.