Re: nfs server issues
- In reply to: Paul Barker : "Re: nfs server issues"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Tue, 07 Jul 2026 14:01:06 UTC
On Tue, Jul 7, 2026 at 5:30 AM Paul Barker <paul@pbarker.dev> wrote:
>
> On Tue, 2026-07-07 at 14:29 +0300, Konstantin Belousov wrote:
> > On Tue, Jul 07, 2026 at 12:18:21PM +0100, Paul Barker wrote:
> > > On Tue, 2026-07-07 at 09:52 +0100, Richard Purdie wrote:
> > > > On Sat, 2026-07-04 at 12:41 -0700, Rick Macklem wrote:
> > > > > On Sat, Jul 4, 2026 at 1:05 AM Richard Purdie
> > > > > <richard.purdie@linuxfoundation.org> wrote:
> > > > > >
> > > > > > nfsstat -E -s
> > > > > > Server Info:
> > > > > > Getattr Setattr Lookup Readlink Read Write
> > > > > > 17354100116 1674996998 5136769057 40544 18429670397 986443058
> > > > > > Create Remove Rename Link Symlink Mkdir
> > > > > > 0 98646613 119865835 17071853 0 0
> > > > > > Rmdir Readdir RdirPlus Access Mknod Fsstat
> > > > > > 0 1655544711 0 4129535844 0 0
> > > > > > FSinfo pathConf Commit LookupP SetClId SetClIdCf
> > > > > > 0 0 8723436 0 0 0
> > > > > One more thing to note. The # of Commits is much smaller than the # of Writes
> > > > > on the top line. This "hints" that a lot of the Writes are being done File_sync.
> > > > > (To check that, you'd need to capture packets and look at them in wireshark.)
> > > > >
> > > > > If a lot of the Writes are File_sync (which means the server must commit the
> > > > > data/metadata changes to stable storage before replying), doing the "mirrored
> > > > > pair of storage devices dedicated to the ZIL" could be what you need to get
> > > > > writes to work well. (As I mentioned, the "cheat" alternative is to
> > > > > set "sync=disabled",
> > > > > but that runs a risk of data loss when the NFS server crashes/reboots.)
> > > >
> > > > I was able to find a way to increase the Open/Lock counts on the
> > > > server. Most of the locking we do on the clients is from a tool called
> > > > Bitbake and the locking is isolated to specific function. I maintain
> > > > Bitbake and wrote/maintain the locking code over the <too many> years.
> > > > I copied and pasted the relevant code straight from bitbake into a test
> > > > script:
> > > >
> > > > https://valkyrie.yocto.io/pub/non-release/rptest/testlock.py
> > > >
> > > > (code is from
> > > > https://git.openembedded.org/bitbake/tree/lib/bb/utils.py)
> > > >
> > > > Obviously it could easily be simplified more but I wanted to test our
> > > > actual code. The lock file name/path is hardcoded at the end of the
> > > > script but easily changed.
> > > >
> > > > I ran that script on three of the NFS clients for around 5 minutes. On
> > > > the server, the counts went from:
> > > >
> > > > nfsdumpstate | egrep 'fd01:172:16::242:2157|fd01:172:16::12|fd01:172:16:1::11 '
> > > > CB 2 6307 55 55 0 0 fd01:172:16::242:2157 4c696e7578204e465376342e322064656269616e31322d766b2d31
> > > > CB 3 6543 77 77 0 0 fd01:172:16::12 4c696e7578204e465376342e322064656269616e31322d766b2d33
> > > > CB 2 6571 83 83 0 0 fd01:172:16:1::11 4c696e7578204e465376342e322064656269616e31322d766b2d32
> > > >
> > > > to:
> > > >
> > > > nfsdumpstate | egrep 'fd01:172:16::242:2157|fd01:172:16::12|fd01:172:16:1::11 '
> > > > CB 2 6462 65 65 0 0 fd01:172:16::242:2157 4c696e7578204e465376342e322064656269616e31322d766b2d31
> > > > CB 3 6679 83 83 0 0 fd01:172:16::12 4c696e7578204e465376342e322064656269616e31322d766b2d33
> > > > CB 2 6691 88 88 0 0 fd01:172:16:1::11 4c696e7578204e465376342e322064656269616e31322d766b2d32
> > > >
> > > > I've then stopped the scripts and waited and the counts just stayed
> > > > there.
> > > >
> > > > I'd be very interested if someone else could reproduce that and if so,
> > > > explain what is happening and if it is an issue?
> > > >
> > > > I can't prove the write hangs are related to the counts but it does
> > > > seem there is some issue there regardless as they don't make sense.
> > >
> > > Hi all,
> > >
> > > This is a long one...
> > >
> > > I've looked into this issue with the Yocto Project autobuilder cluster
> > > alongside Richard. I've used Claude to help me navigate the Linux and
> > > FreeBSD source trees, along with manually checking some of the key
> > > findings, so this is in the territory of a "working theory" as to what's
> > > going on. Some input from folks more familiar with the FreeBSD NFS
> > > server code would definitely be helpful to confirm I'm on the right
> > > track.
> > >
> > > The issue we're seeing is probably triggered by the following sequence
> > > of events:
> > >
> > > 1) Client A opens a file on the FreeBSD server via an NFS mount. Even if
> > > the client immediately closes the file, Linux's NFS client code will
> > > keep the file handle cached for a short while before sending a CLOSE
> > > to the server.
> > >
> > > 2) During this window, client B deletes or replaces the file. This sends
> > > a REMOVE or RENAME message to the server. The FreeBSD NFS server code
> > > unlinks the underlying file immediately.
> > >
> > > 3) Client A now releases the file, either explicitly or due to cache
> > > timing out. At this point it sends a PUTFH/CLOSE compound message to
> > > the NFS server, as CLOSE acts on the current file handle the PUTFH is
> > > needed to select which file is being closed. However, the file handle
> > > is now stale due to the removal from another client so the PUTFH
> > > operation fails with ESTALE. The CLOSE operation is never reached, so
> > > no resource cleanup occurs. The file handle has now leaked and will
> > > not be released unless the client session which owns it terminates.
> > >
> > > I've confirmed that the file becomes stale with a simple experiment:
> > >
> > > 1) Open a file via a Python shell on client A. Leave this Python shell
> > > running.
> > >
> > > >>> fh = open("/srv/autobuilder/valkyrie.yocto.io/pub/non-release/testfile.txt", "r")
> > >
> > > 2) Remove the file from client B:
> > >
> > > $ rm /srv/autobuilder/valkyrie.yocto.io/pub/non-release/testfile.txt
> > >
> > > 3) Now attempt to read the file via the open handle on client A:
> > >
> > > >>> fh.read()
> > > Traceback (most recent call last):
> > > File "<python-input-2>", line 1, in <module>
> > > fh.read()
> > > ~~~~~~~^^
> > > OSError: [Errno 116] Stale file handle
> > >
> > > I've also looked at the NFS server code myself on the freebsd-src
> > > stable/14 branch. In nfsrvd_compound(), case NFSV4OP_PUTFH calls
> > > nfsd_fhtovp(), which calls nfsvno_fhtovp() to resolve the file handle.
> > > This sets nd->nd_repstat to ESTALE if the file has been unlinked. This
> > > causes a break from the main loop in nfsrvd_compound() to return the
> > > error to the client, so the CLOSE operation is never processed. And if
> > > CLOSE is never processed, the open file handle leaks.
> > >
> > > This causes us two kinds of problems:
> > >
> > > 1) The hash table of open file handles grows. With >400k file handles in
> > > the hash table we see the CPU load over 5000%. There is one file
> > > handle table in the NFS server, with one lock, and all stateful
> > > operations seem to need to take this lock to lookup a file handle, so
> > > this leads to thrashing.
> > >
> > > Linux may interpret timeouts waiting for a server response as
> > > EREMOTEIO.
> > >
> > > This can be somewhat ameliorated by increasing vfs.nfsd.fhhashsize to
> > > spread the open handles across more buckets in the hash table, but
> > > that just delays the load issues, it doesn't resolve them.
> > >
> > > 2) When the total number of open file handles + open locks hits 500,000,
> > > we hit the default v4statelimit. This causes the server to reply to
> > > operations with NFSERR_RESOURCE which Linux also interprets as
> > > EREMOTEIO.
> > >
> > > The ~483,000 open file handles in Richard's original message is
> > > consistent with the 500k limit being reached before a few clients
> > > were disconnected (zeroing out their open counts and reducing the
> > > overall total).
> > >
> > > We could increase vfs.nfsd.v4statelimit, but again this is just
> > > delaying the inevitable issues. If we're leaking ~500k open file
> > > handles in less than a week then no limit will last very long.
> > >
> > > So, essentially we would need to avoid the leak in the first place.
> > > Should the error handling for PUTFH on a stale file handle free the
> > > associated state? Or should a REMOVE/RENAME operation leave the file
> > > handle intact until all consumers have released the file handle (this is
> > > what a Linux NFS server seems to do)?
> > >
> > > Thank you for bearing with that long explanation! If I've misunderstood
> > > anything then please let me know and I'll take another look.
> >
> > Does the FreeBSD NFS server still reference the vnode for the deleted file
> > in your situation? If yes, then c041b82c59e885d106f45454302bad3d9dcc58a7
> > might already help.
>
> Hi Konstantin, thanks for the info. I sadly don't think this patch will
> help, we're using ZFS as the backing store for these NFS shares and that
> commit only changes UFS/FFS code. I also don't think the NFS server
> holds a vnode reference for each open stateid but I may be wrong, I'm
> not really familiar with the code base.
You are correct. The Open/Lock structures hold file handles (no vnodes or
file descriptors).
--> This does imply that the code does not know if/when a file is deleted,
so the Opens/Locks can persist after the file is deleted. My understanding
is that the NFSv4 client is responsible for doing LockU/Close operations.
(You could argue that there should be some sort of garbage collection
running in background to get rid of Opens/Locks that exist after the
file has been deleted, but the current code does not do that. The
counter argument is that the client should be doing LockU/Close
and the structures are pretty harmless.)
Btw, I believe the Linux knfsd server does hold some sort of reference
to the file for Opens/Locks and since the Linux client mainly gets tested
against the Linux knfsd, they might have found that LockU/Close's are
not needed for the Linux knfsd. (This is just conjecture.)
rick
>
> Best regards,
>
> --
> Paul Barker
>