Re: nfs server issues
- Reply: Richard Purdie : "Re: nfs server issues"
- In reply to: Michael Halstead : "Re: nfs server issues"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Fri, 10 Jul 2026 23:33:33 UTC
On Fri, Jul 10, 2026 at 1:01 PM Michael Halstead
<mhalstead@linuxfoundation.org> wrote:
>
>
>
> On Wed, Jul 8, 2026 at 2:38 PM Rick Macklem <rick.macklem@gmail.com> wrote:
>>
>> On Wed, Jul 8, 2026 at 7:57 AM Rick Macklem <rick.macklem@gmail.com> wrote:
>> >
>> > On Wed, Jul 8, 2026 at 2:28 AM Richard Purdie
>> > <richard.purdie@linuxfoundation.org> wrote:
>> > >
>> > > On Tue, 2026-07-07 at 15:54 -0700, Michael Halstead wrote:
>> > > > On Tue, Jul 7, 2026 at 1:52 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
>> > > > > On Sat, 2026-07-04 at 12:41 -0700, Rick Macklem wrote:
>> > > > > > On Sat, Jul 4, 2026 at 1:05 AM Richard Purdie
>> > > > > > <richard.purdie@linuxfoundation.org> wrote:
>> > > > > > >
>> > > > > > > nfsstat -E -s
>> > > > > > > Server Info:
>> > > > > > > Getattr Setattr Lookup Readlink Read Write
>> > > > > > > 17354100116 1674996998 5136769057 40544 18429670397 986443058
>> > > > > > > Create Remove Rename Link Symlink Mkdir
>> > > > > > > 0 98646613 119865835 17071853 0 0
>> > > > > > > Rmdir Readdir RdirPlus Access Mknod Fsstat
>> > > > > > > 0 1655544711 0 4129535844 0 0
>> > > > > > > FSinfo pathConf Commit LookupP SetClId SetClIdCf
>> > > > > > > 0 0 8723436 0 0 0
>> > > > > > One more thing to note. The # of Commits is much smaller than the # of Writes
>> > > > > > on the top line. This "hints" that a lot of the Writes are being done File_sync.
>> > > > > > (To check that, you'd need to capture packets and look at them in wireshark.)
>> > > >
>> > > >
>> > > > I've captured data
>> > > > at https://valkyrie.yocto.io/pub/non-release/mtest/nfs_capture2.pcap.
>> > > > zst from a fresh reboot running the testlock.py for about 100MB of
>> > > > data and umounting the filesystems. Please let me know if this
>> > > > reveals the issue or what else I can offer.
>> > >
>> > > Thanks Michael. I think that didn't have enough parallel test scripts
>> > > running. This is one I just captured with more parallelism (4 clients),
>> > > just tracing one of them:
>> > >
>> > > https://valkyrie.yocto.io/pub/non-release/mtest/rp_capture.pcap.zst
>> > >
>> > > This one has a lot of NFS4ERR_STALE in it which I suspect is what we
>> > > were looking for...
>> > Yep. This one definitely shows RPCs that would do Close failing when
>> > PutFH replies NFS4ERR_STALE. (I looked at the previous one and couldn't
>> > see any problems in it, although the previous one does have the weird case
>> > where the same Lock is acquired twice. Both succeed because it is the
>> > same client and lock_owner. I am going to check to see what the RFCs
>> > say w.r.t. duplicate lock operations. Like Open, the Lock ops were modelled
>> > after what Windows wants, but were intended to handle POSIX lock semantics
>> > as well. Of course, no one implements mandatory locking and the protocol
>> > doesn't even allow the client to figure out which the server is doing, but
>> > that's another story.;-)
>> >
>> > I do think I can come up with a simple solution for this:
>> > - Since the operations that get rid of state all require a CFH, which
>> > must be set by PutFH.. I cannot see any reason to keep state once
>> > the PutFH is returning NFS4ERR_STALE for the file.
>> > --> So, I think I can just free all the state for that FH at that time.
>> > (The state is keyed on FH and not a vnode, so the PutFH has
>> > the required FH.)
>> > --> I'll post a patch here later to-day. Hopefully you guys can run a
>> > kernel from patched sources to test it.
>> I've attached a patch that I think fixes this (minimally tested).
>> If you can test this by applying it to your kernel sources and building a
>> kernel from sources, please do so and let us know how it goes.
>>
>> If you've never done this, the following should work (assuming the
>> source tree is /usr/src).
>> # cd /usr/src
>> # patch -p0 < freefh.patch
>> # make buildkernel
>> # make installkernel
>> - reboot
>>
>> rick
>>
>
> I updated to FreeBSD 15.1p1 and applied this patch. Running our testlock.py script on 40+ clients at once worked fine. Open counts stayed at 0 or 1 for the duration of the test. Our build cluster now has 22 hours uptime with normal operation and all the stats look great.
>
> nfsstat -E -s
> Server Info:
> Getattr Setattr Lookup Readlink Read Write
> 247019884 21674278 77085915 2 145835919 9351171
> Create Remove Rename Link Symlink Mkdir
> 0 1663599 1990692 278837 0 0
> Rmdir Readdir RdirPlus Access Mknod Fsstat
> 0 9782666 0 65790267 0 0
> FSinfo pathConf Commit LookupP SetClId SetClIdCf
> 0 0 93580 0 0 0
> Open OpenAttr OpenDwnGr OpenCfrm DelePurge DelRet
> 26716944 0 0 0 0 0
> GetFH Lock LockT LockU Close Verify
> 47284768 15945046 0 1514379 23213889 0
> NVerify PutFH PutPubFH PutRootFH Renew RestoreFH
> 0 458678547 0 85 0 278837
> SaveFH Secinfo RelLockOwn V4Create
> 2269529 0 0 8443
> BackChannelCtBindConnToSes ExchangeID CreateSess DestroySess FreeStateID
> 0 0 85 47 3 1473548
> GetDirDeleg GetDevInfo GetDevList layoutCommit LayoutGet LayoutReturn
> 26 0 0 0 0 0
> SecInfNoName Sequence SetSSV TestStateID WantDeleg DestroyClId
> 41 457883368 0 41 0 0
> ReclaimCompl
> 44
> Allocate Copy CopyNotify Deallocate IOAdvise LayoutError
> 0 0 0 0 0 0
> LayoutStats OffloadCncl OffloadStat ReadPlus Seek WriteSame
> 0 0 0 11 615 0
> Clone GetExtattr SetExtattr ListExtattr RmExtattr
> 0 0 0 685 0
> Server:
> Clients OpenOwner Opens LockOwner Locks Delegs
> 44 0 0 0 0 0
> Layouts
> 0
> Server Cache Stats:
> Inprog Non-idem Misses CacheSize TCPPeak
> 0 0 0 0 0
>
> When the cluster is idle, `nfsdumpstate` shows zeros across the board as expected.
>
> I also added to /boot/loader.conf:
> vfs.nfsd.clienthashsize=2048
> vfs.nfsd.sessionhashsize=2048
> vfs.nfsd.statehashsize=4096
> vfs.nfsd.fhhashsize=16384
>
> Are there any other useful detail we can provide?
Not that I can think of.
I'll assume you don't mind being listed as "Tested by:"
for a commit of the patch. (If you do mind, just email me.)
I am still wondering if you will see your write problems,
but I assume it's wait and see for now.
Thanks for testing it, rick
ps: I will probably commit this little patch to main soon. It should be in
stable/15 and stable/14 for the next releases.
>
>
>>
>> >
>> > Thanks for the packet trace and help tracking this down, rick
>> >
>> > >
>> > > Cheers,
>> > >
>> > > Richard
>
>
>
> --
> Michael Halstead (he/him)
> Linux Foundation / Yocto Project
> Staff IT Engineer