[patch] giant-less quotas for UFS

[Eric Anderson]

Nicolas KOWALSKI wrote:
> Eric Anderson <anderson at centtech.com> writes:
> 
>> Nicolas KOWALSKI wrote:
>>> Eric Anderson <anderson at centtech.com> writes:
>>>
>>>> Nicolas KOWALSKI wrote:
>>>>> Eric Anderson <anderson at centtech.com> writes:
>>>>>
>>>>>> Nicolas KOWALSKI wrote:
>>>>>>> Eric Anderson <anderson at centtech.com> writes:
>>>>>>>
>>>>>>>> Nicolas KOWALSKI wrote:
>>>>>>>>> Yes, this is exactly what is happening. To add some precision, some
>>>>>>>>> students here use calculation applications
>>>>>>>>> that allocate a lot of disk space, ususally more than their allowed
>>>>>>>>> home quotas; when by error they launch these apps in their home
>>>>>>>>> directories, instead of their workstation dedicated space, it makes
>>>>>>>>> the server go to its knees on the NFS client side.
>>>>>>>> When you say 'to it's knees' - what do you mean exactly?  How many
>>>>>>>> clients do you have, how much memory is on the server, and how many
>>>>>>>> nfsd threads are you using?  What kind of load average do you see
>>>>>>>> during this (on the server)?
>>>>>>> Sorry for the imprecision.
>>>>>>> The server is a Dual-Xeon 2.8Ghz, 2GB of RAM, using SCSI3 Ultra320
>>>>>>> 76GB disks and controller. It is accessed by NFS from ~100 Unix
>>>>>>> (Linux, Solaris) clients, and by Samba from ~15 Windows XP. The
>>>>>>> network connection is GB ethernet.
>>>>>>> During slowdowns, it's only from a NFS client view that the server
>>>>>>> does not respond. For example, a simple 'ls' in my home directory is
>>>>>>> almost immediate, but when it slows down, it can take up to 2 minutes.
>>>>>>> On the server, the load average goes to 0.5, compared to a default
>>>>>>> maximum of 0.15-0.20. The nfsd processus shows them in the state
>>>>>>> "biowr" in top, but nothing is really written, because the quotas
>>>>>>> system block any further writes to the user exceeding her/his quotas.
>>>>>>>
>>>>>> In this case (which is what I suspected), try bumping up your nfsd
>>>>>> threads to 128.  I set mine very high (I have around 1000 clients),
>>>>>> and I can say there aren't really ill-effects besides a bit of memory
>>>>>> usage (which you have plenty of).  I suspect increasing the threads
>>>>>> will neutralize this problem for you.
>>>>> Using 128 nfsd threads, I stressed the server, by running on a NFS
>>>>> client a small C program, writting continuously in a file, so that the
>>>>> user "biguser" (account stored on /export/home2) exceeds his quota.
>>>>> It half-works: during the test, users working on another disk
>>>>> (/export/home) did not see any difference, but users working on the
>>>>> same disk that "biguser" (/export/home2) where almost halted.
>>>>> So, this is better, because before everybody was halted, but there is
>>>>> still a problem.
>>>>> Any other tips ?
>>>> Watch gstat during the testing, and see if the disk that holds the
>>>> full partition is really busy.  I'm betting it's thrashing the disk
>>>> continually checking for free space.  I don't think there's any way to
>>>> avoid that.
>>> Mh, I did not find this "gstat" tool on the system or in the ports;
>>> perhaps is it in >= 5.x ? (the server is running 4.11-p15).
>>> It is sad I can not do anything about it: such a server pulled down
>>> by
>>> a single NFS-client. :-(
>>
>> *sigh* - I should really pay more attention to the beginning of the
>> thread.  I thought you were on 5.x, so my mistake.  You'll need to use
>> iostat to see what's busy with your disks.
> 
> Ok, I'll check with that. Thanks.
> 
>> I strongly recommend going to 6.x if you can..
> 
> Yep, it's planned, but only in a few months, when all
> students/teachers will go to the beach. ;-)
> 
> Just for my knowledge, what will improve the situation ? Better
> Locking ? If I read the start of the thread, it looks like the actual
> quotas system is still under the giant lock, so...
> 

There are a lot of file system improvements, from locking to just plain 
bug fixing.


Eric




-- 
------------------------------------------------------------------------
Eric Anderson        Sr. Systems Administrator        Centaur Technology
Anything that works is better than anything that doesn't.
------------------------------------------------------------------------