Problems Re-Starting mountd

[Andrey Simonenko]

On Tue, Jan 08, 2013 at 10:22:16AM -0500, Jason Keltz wrote:
> On 01/08/2013 10:05 AM, Andrey Simonenko wrote:
> > I created 2000 file systems on ZFS file system backed by vnode md(4)
> > device.  The /etc/exports file contains 4000 entries like your example.
> >
> > On 9.1-STABLE mountd spends ~70 seconds in flushing current NFS exports
> > in the NFS server, parsing data from /etc/exports and loading parsed
> > data into the NFS server.  ~70 seconds is not several minutes.  Most of
> > time mountd spends in nmount() system call in "zio->io_cv" lock.
> >
> > Can you show the output of "truss -fc -o /tmp/output.txt mountd"
> > (wait wchan "select" state of mountd and terminate it by a signal).
> > If everything is correct you should see N statfs() calls, N+M nmount()
> > calls and something*N lstat() calls, where N is the number of /etc/exports
> > lines, M is the number of mounted file systems.  Number of lstat() calls
> > depends on number of components in pathnames.
> 
> Andrey,
> 
> Would that still be an ~70 second period in which new mounts would not 
> be allowed? In the system I'm preparing, I'll have at least 4000 entries 
> in /etc/exports, probably even more, so I know I'll be dealing with the 
> same issue that Tim is dealing with when I get there.  However, I don't 
> see how to avoid the issue ... If I want new users to be able to login 
> shortly after their account is created, and each user has a ZFS 
> filesystem as a home directory, then at least at some interval, after 
> adding a user to the system, I need to update the exports file on the 
> file server, and re-export everything.  Yet,  even a >1 minute delay 
> where users who are logging in won't get their home directory mounted on 
> the system they are logging into - well, that's not so good... accounts 
> can be added all the time and this would create random chaos.  Isn't 
> there some way to make it so that when you re-export everything, the 
> existing exports are still served until the new exports are ready?

When mountd starts it flushes NFS export settings for all file systems,
for each mount point it calls nmount(), even if /etc/exports is empty
it will call nmount() for all currently mounted file systems.

When mountd loads export settings into NFS server it calls statfs() and
lstat() for each pathname from /etc/exports (number of lstat() calls depends
on number of '/' in each pathname), then it calls nmount() for each
address specification for each pathname from /etc/exports.  It uses
nmount() interface for communication with kern/vfs_export.c that is
responsible for NFS export settings for file systems.  For the NFSv4
root directory mountd uses nfssvc() to update its settings, that
calls kern/vfs_export.c:vfs_export().

When mountd receives SIGHUP it flushes everything and loads /etc/exports.
This signal is sent by mount(8) when it mounts any file system.

This delay in above described example came from ZFS kernel code, since
the same configuration for 2000 nullfs(5) file systems takes ~0.20 second
(less than second) by mountd in nmount() system calls.  At least on
9.1-STABLE I do not see that this delay came from mountd code, it came
from nmount() used by mountd.

> Would this be the same for NFSv3 versus NFSv4?  I suspect yes.

NFS export settings are loaded into NFS server in the same for all NFS
versions.