Long Format Directory Listing 15x Slower in FreeBSD 5.x

[Nikolai Schupbach]

Brooks Davis wrote:

>On Mon, Sep 12, 2005 at 09:37:27PM -0400, Kris Kennaway wrote:
>  
>
>>On Mon, Sep 12, 2005 at 06:33:53PM -0700, Brooks Davis wrote:
>>    
>>
>>>On Mon, Sep 12, 2005 at 08:51:44PM -0400, Kris Kennaway wrote:
>>>      
>>>
>>>>On Tue, Sep 13, 2005 at 11:55:55AM +1200, Nikolai Schupbach wrote:
>>>>        
>>>>
>>>>>Hello,
>>>>>
>>>>>We have been trying to migrate to FreeBSD 5.4 from an older 4.x release 
>>>>>for one of our busy mail servers. But we have encountered problems with 
>>>>>directory listings on 5.4.
>>>>>
>>>>>Our /var/mail directory contains approximately 8,000 files doing a long 
>>>>>directory listing (ls -l) takes approximately 5 min and during this time 
>>>>>the CPU is running near 100%, on a FSBD 4 box the same directory 
>>>>>contents takes less than 20 seconds to list. Even on a directory with 
>>>>>200 files, each file with a different owner, it will still take at least 
>>>>>4-5 seconds to list. The problems only seems to occur when the directory 
>>>>>contains files from many different users. (as in /var/mail). If I chown 
>>>>>all the files in the /var/mail directory to a single user the directory 
>>>>>listing is near instant.
>>>>>
>>>>>It appears it has something to do with 'ls' looking up the id's in the 
>>>>>password database, because if I instruct ls to display numeric IDs (ls 
>>>>>-ln), rather than converting to user and group names, the directory with 
>>>>>8,000 files, with 8,000 different owners will list instantly.
>>>>>
>>>>>The reason this concerns me so much is we are also having a problem with 
>>>>>our Washington IMAP / POP3 server on FBSD 5 using a lot of CPU while 
>>>>>operating on small and even 0 byte mailboxes, when there are approx five 
>>>>>or more concurrent POP3 or IMAP sessions. And I can't help thinking that 
>>>>>the two problems are related.
>>>>>
>>>>>Does anyone have any ideas? Has anyone else noticed this problem? If I 
>>>>>can't resolve it I'm most likely going to revert to using 4.11, but I'd 
>>>>>really like to know what is going on. (and yes I'm using UFS_DIRHASH)
>>>>>          
>>>>>
>>>>Sounds like the real problem is that you have >8000 users on your
>>>>machine and lookups are taking a long time.  There has been discussion
>>>>of this problem and how to solve it on freebsd-questions@ several
>>>>times..I think it involves making a minor change to pwd_mkdb and
>>>>rebuilding.  Please search the archives for that mailing list.
>>>>        
>>>>
>>>If you are using NIS and have any compat options in /etc/nsswitch.conf
>>>your performance will really suck in situations like this.  IIRC, the
>>>compat code is worse than O(n^2) if you look up each user and the
>>>non-compat code is close to O(n).  I'd really like to stop generating
>>>nsswitch.conf entries that use compat in 7.0.
>>>      
>>>
>>I'm pretty sure there's a problem with the database hash parameters
>>used by pwd_mkdb on systems with large numbers of users..this is what
>>I was alluding to above.
>>    
>>
>
>That wouldn't suprised me.  For that matter it could be both problems.
>
>-- Brook
>
Looks like exact problem is described here:
http://www.freebsd.org/cgi/query-pr.cgi?pr=75855

I'm not using NIS, so I will try the change /etc/nsswitch.conf  group 
and passwd lines from 'compat' to 'files' and report back.

-- Nikolai.