Terrible NFS4 performance: FreeBSD 9.1 + UFS/ZFS + AWS EC2

[Rick Macklem]

Berend de Boer wrote:
> >>>>> "Rick" == Rick Macklem <rmacklem at uoguelph.ca> writes:
> 
>     Rick> If you could do some testing where you export a UFS volume,
>     Rick> the results might help to isolate the issue to ZFS vs nfsd.
> 
> Indeed! Have changed subject, as indeed ZFS is a red herring. Issue
> shows up with UFS as well. Very high cpu for nfds, about the same
> time
> to do the operation.
> 
> Tried it with these settings:
> 
>   vfs.nfsd.tcphighwater=5000
>   vfs.nfsd.tcpcachetimeo=300
>   nfs_server_flags="-u -t -n 256"
> 
> 
> On nfs3 + ufs everything back to normal. I.e. nfs4 was about 15
> minutes, same operation was 241s minutes with nfs3 and nfsd using no
> cpu at all basically.
> 
All I can suggest is capturing packets and then emailing be the captured
packet trace. You can use tcpdump to do the capture, since wireshark will
understand it:
# tcpdump -s 0 -w <file>.pcap host <client-host>
and then emailing me <file>.pcap.

I can take a look at the packet capture and maybe see what is going on.

I think you mentioned that you were using a Linux client, but not what
version. I'd suggest a recent kernel from kernel.org. (Fedora tracks
updates/fixes for NFSv4 pretty closely, so the newest Fedora release
should be pretty current.)


> Per other reply, tried this too:
> 
>   vfs.nfsd.issue_delegations=1
> 
> Locks up the client at first write access. Ctrl+C doesn't work, need
> to explicitly send a KILL signal from other terminal.
> 
> I think it locks up the server in some way as well. Doing an ls on an
> exported path locks up. Ctrl+C won't work anymore. Process doesn't
> react to any signal. In the end I rebooted the server to get rid of
> this.
> 
Obviously broken, but without a lot more information, I can's say anything.
(Assuming you can still run commands on the server, you could start with
 something like "ps axl" and "procstat -kk".)

rick

> --
> All the best,
> 
> Berend de Boer
> 
> 
>           ------------------------------------------------------
>           Awesome Drupal hosting: https://www.xplainhosting.com/
>