[Bug 300581] NFS client hang under heavy load

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=300581

            Bug ID: 300581
           Summary: NFS client hang under heavy load
           Product: Base System
           Version: 15.1-RELEASE
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: shurd@FreeBSD.org

I've hit this twice in the last few months, under heavy load when doing large
number of NFS iops, the NFS client can hang. I had my LLM take a look and what
it says looks plausible... the rest of this report is LLM generated:

A worker-startup race stands out, and heavy load would make it more plausible.
This is a credible hypothesis, not a confirmed diagnosis.

In nfs_clnfsiod.c:180 (/usr/src/sys/fs/nfsclient/nfs_clnfsiod.c:180), the
creator:

Releases the NFS worker mutex.
Starts a worker that can immediately execute.
Reacquires the mutex and unconditionally marks that worker AVAILABLE.

Meanwhile, the worker can mark itself available at line 263
(/usr/src/sys/fs/nfsclient/nfs_clnfsiod.c:263). If it occupies a reused,
low-numbered slot, another I/O request can assign it work before step 3. The
creator could then overwrite its busy state with AVAILABLE, allowing the same
worker to be assigned again.

That could corrupt the per-mount count of assigned workers and leave a mount
believing it has a worker after all actual workers have exited.

The resulting failure matches our observations particularly well. At
nfs_clbio.c:1466 (/usr/src/sys/fs/nfsclient/nfs_clbio.c:1466), that stale count
would let a writer reach:

while (nmp->nm_bufqlen >= 2*ncl_numasync)

With zero workers, even an empty queue satisfies 0 >= 0. The writer sleeps
indefinitely in nfsaio. Newly created workers aren’t assigned the mount, so
they sit idle. Fresh I/O can assign one, drain the queue and wake the
writer—exactly how we recovered.

The startup ordering is worth investigating first. We’d need instrumentation or
a controlled reproduction to establish that this particular interleaving caused
your hang. I haven’t modified the kernel source.

-- 
You are receiving this mail because:
You are the assignee for the bug.