how to fix an interesting issue with mountd?

[Rick Macklem]

Rodney Grimes wrote:
>> Hi,
>> 
>> I'm posting this one to freebsd-net@ since it seems vaguely similar
>> to a network congestion problem and thought that network types
>> might have some ideas w.r.t. fixing it?
>> 
>> PR#246597 - Reports a problem (which if I understand it is) where a sighup
>>    is posted to mountd and then another sighup is posted to mountd while
>>    it is reloading exports and the exports are not reloaded again.
>>    --> The simple patch in the PR fixes the above problem, but I think will
>>           aggravate another one.
>> For some NFS servers, it can take minutes to reload the exports file(s).
>> (I believe Peter Erriksonn has a server with 80000+ file systems exported.)
>> r348590 reduced the time taken, but it is still minutes, if I recall correctly.
Actually, my recollection w.r.t. the times was way off.
I just looked at the old PR#237860 and, without r348590, it was 16seconds
(aka seconds, not minutes) and with r348590 that went down to a fraction
of a second (there was no exact number in the PR, but I noted milliseconds in
the commit log entry.

I still think there is a risk of doing the reloads repeatedly.

>> --> If you apply the patch in the PR and sighups are posted to mountd as
>>        often as it takes to reload the exports file(s), it will simply reload the
>>        exports file(s) over and over and over again, instead of processing
>>        Mount RPC requests.
>> 
>> So, finally to the interesting part...
>> - It seems that the code needs to be changed so that it won't "forget"
>>   sighup(s) posted to it, but it should not reload the exports file(s) too
>>   frequently.
>> --> My thoughts are something like:
>>   - Note that sighup(s) were posted while reloading the exports file(s) and
>>     do the reload again, after some minimum delay.
>>     --> The minimum delay might only need to be 1second to allow some
>>            RPCs to be processed before reload happens again.
>>      Or
>>     --> The minimum delay could be some fraction of how long a reload takes.
>>           (The code could time the reload and use that to calculate how long to
>>            delay before doing the reload again.)
>> 
>> Any ideas or suggestions? rick
>> ps: I've actually known about this for some time, but since I didn't have a good
>>      solution...
>
>Build a system that allows adding and removing entries from the
>in mountd exports data so that you do not have to do a full
>reload every time one is added or removed?
>
>Build a system that used 2 exports tables, the active one, and the
>one that was being loaded, so that you can process RPC's and reloads
>at the same time.
Well, r348590 modified mountd so that it built a new set of linked list
structures from the modified exports file(s) and then compared them with
the old ones, only doing updates to the kernel exports for changes.

It still processes the entire exports file each time, to produce the in mountd
memory linked lists (using hash tables and a binary tree).

Peter did send me a patch to use a db frontend, but he felt the only
performance improvements would be related to ZFS.
Since ZFS is something I avoid like the plague I never pursued it.
(If anyone willing to ZFS stuff wants to pursue this,
just email and I can send you the patch.)
Here's a snippet of what he said about it.
>  It looks like a very simple patch to create and even though it wouldn’t really        >  improve the speed for the work that mountd does it would make possible really >  drastic speed improvements in the zfs commands. They (zfs commands) currently >  reads the thru text-based exports file multiple times when you do work with zfs  >  filesystems (mounting/sharing/changing share options etc). Using a db based  
>  exports file for the zfs exports (b-tree based probably) would allow the zfs code > to be much faster.

At this point, I am just interested in fixing the problem in the PR, rick