Re: FreeBSD 13.2 NFS client mount hangs

On Fri, Oct 6, 2023 at 10:48 AM J David <j.david.lists@gmail.com> wrote:
>
> On Mon, Oct 2, 2023 at 7:08 PM Rick Macklem <rick.macklem@gmail.com> wrote:
> > > The nfscbd daemon is not running on any of the clients.
> > >
> > > > If the Linux server still
> > > > issues delegations
> > >
> > > How would I determine that?
> > nfsstat -E -c
> > and then look at the number under "Delegs". It is a current count of
> > delegations, so if it remains 0 over time, no delegations are being issued.
>
> But if this is done from a client that is not running nfscbd, isn't it
> pretty well guaranteed to be zero?
>
> Checking all the clients I can find, "Deleg" is zero on all of them.
> On about half, "DelegRet" is nonzero but small (1-100), but I don't
> know what that is or if it's related.
>
> > I have attached a small patch which should make the NFS client handle
> > this error correctly.
>
> I will look for a way to try this patch, but the clients in this case
> are all managed with freebsd-update and don't have enough disk space
> to build a kernel locally, so it may be tricky.
>
> > > > # tcpdump -s 0 -w out.pcap host <nfs-server-name>
> > > > Let this run for a while and then pull out.pcap into wireshark and see what
> > > > traffic is going between the NFS client and server.
> > > > (Unlike tcpdump, wireshark does know how to decode NFS properly.)
> > >
> > > If/when the issue happens again, I will attempt to do this and report back.
>
> I am also working on getting access to Wireshark.
>
> In the interim, it did happen again, so the best I can do is put a
> little bit of tcpdump output here: https://pastebin.com/UDrphwr5 .
Maybe someone who is familiar with tcpdump output can look at this.
(I always use wireshark.)
It looks to me like the TCP checksum is failing on the client->server request,
but maybe I am not reading it correctly.

If the checksum is incorrect, then there is something badly broken in your
network fabric.

rick

>
> I can't vouch for "correct" but it does mostly seem to decode the NFS packets.
>
> It seems to loop the same couple of actions with long delays (15
> seconds) between retries:
>
> This sequence:
> +0.0000s: Client -> server xid 1205841201 getattr fh 0,7/2 ("Getattr"
> in packet body)
> +1.4106s: Client -> server xid 1205841202 getattr fh 0,5/2 ("Renew" in
> packet body)
> +0.0002s: Server -> client xid 1205841202 getattr LNK 12231267145 ids
> 1/53 sz 0 ("Renew" in packet body)
> +3.8001s: Server -> client xid 1205841201 getattr ERROR: Request
> couldn't be completed in time ("Getattr" in packet body)
>
> Repeats after 15 seconds:
> +15.0090s: client -> server 1205841203 getattr fh 0,7/2 ("Getattr" in
> packet body)
> ... etc
>
> The "fh 0,7/2" and "fh 0,5/2" seem to be consistent each time. The xid
> (transaction/request ID?) increments each time.
>
> Maybe that will provide a lucky flash of insight in the interim.
>
> Thanks!