kern/75855: [libc] getpwent(3) functions on 5.3 with large password file extremely slow

[Scott Lambert]

The following reply was made to PR kern/75855; it has been noted by GNATS.

From: Scott Lambert <lambert at lambertfam.org>
To: bug-followup at FreeBSD.org
Cc: adrian at FreeBSD.org
Subject: kern/75855: [libc] getpwent(3) functions on 5.3 with large password file extremely slow
Date: Thu, 16 Dec 2010 17:42:01 -0700

 We had someone run into this issue on the postfix mailing list
 recently with FreeBSD 8.2-PRERELEASE.
 
 Here is the link to the thread on the Postfix list:
 
 http://tech.groups.yahoo.com/group/postfix-users/message/271899
 
 I was about to open another PR for nsswitch.conf for lack of any
 better idea on where to file this when someone mentioned this PR.
 
 I have no idea what could be done to make for a better experience
 out of the box.
 
 Here is the thread where I first asked about this in 2007:
 http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2007-04/msg00193.html
 
 Here is some current testing from the same box 3+ years later with
 a 90 line passwd file on the host and all the users in a jail on
 the same machine:
 
 $ uname -a          
 FreeBSD web1.hosting.tcworks.net 7.4-PRERELEASE FreeBSD 7.4-PRERELEASE #0: Wed Dec  1 11:02:50 CST 2010     root at web1.hosting.tcworks.net:/usr/obj/usr/src/sys/GENERIC  i386
 
 $ wc -l /etc/passwd 
       90 /etc/passwd
 
 lambert at web1.hosting.tcworks.net /home/ezjails/users.tcworks.net/home
 18:14:16 Thu Dec 16 $ egrep '^(group|passwd):' /etc/nsswitch.conf
 group: files
 passwd: files
 
 lambert at web1.hosting.tcworks.net /home/ezjails/users.tcworks.net/home
 18:14:31 Thu Dec 16 $ time ls -l | wc -l
     3917
 
 real    0m0.124s
 user    0m0.095s
 sys     0m0.040s
 
 lambert at web1.hosting.tcworks.net /home/ezjails/users.tcworks.net/home
 18:14:34 Thu Dec 16 $ time ls -l | wc -l
     3917
 
 real    0m0.113s
 user    0m0.053s
 sys     0m0.071s
 
 lambert at web1.hosting.tcworks.net /home/ezjails/users.tcworks.net/home
 18:14:35 Thu Dec 16 $ time ls -l | wc -l
     3917
 
 real    0m0.124s
 user    0m0.078s
 sys     0m0.056s
 
 $ sudo vi /etc/nsswitch.conf
 
 lambert at web1.hosting.tcworks.net /home/ezjails/users.tcworks.net/home
 18:15:37 Thu Dec 16 $ egrep '^(group|passwd):' /etc/nsswitch.conf
 group: compat
 passwd: compat
 
 lambert at web1.hosting.tcworks.net /home/ezjails/users.tcworks.net/home
 18:15:47 Thu Dec 16 $ time ls -l | wc -l
     3917
 
 real    0m0.748s
 user    0m0.384s
 sys     0m0.355s
 
 lambert at web1.hosting.tcworks.net /home/ezjails/users.tcworks.net/home
 18:15:51 Thu Dec 16 $ time ls -l | wc -l
     3917
 
 real    0m0.743s
 user    0m0.425s
 sys     0m0.317s
 
 lambert at web1.hosting.tcworks.net /home/ezjails/users.tcworks.net/home
 18:15:54 Thu Dec 16 $ time ls -l | wc -l
     3917
 
 real    0m0.734s
 user    0m0.381s
 sys     0m0.351s
 
 
 Then from inside the jail:
 
 $ wc -l /etc/passwd 
     3935 /etc/passwd
 
 lambert at users.tcworks.net /home
 18:04:11 Thu Dec 16 $ egrep '^(group|passwd):' /etc/nsswitch.conf
 group: compat
 passwd: compat
 
 lambert at users.tcworks.net /home
 18:16:32 Thu Dec 16 $ time ls -l | wc -l
     3917
 
 real    1m0.092s
 user    0m11.030s
 sys     0m47.301s
 
 I ran top outside the jail while running ls the first time and got:
 
 68759    997           1 116    0  4492K  2140K CPU0    0   0:25 88.96% ls
 later during the same run ...
 68759    997           1 118    0  4492K  2240K CPU0    0   0:57 98.29% ls
 
 lambert at users.tcworks.net /home
 18:17:35 Thu Dec 16 $ time ls -l | wc -l
     3917
 
 real    1m1.616s
 user    0m11.464s
 sys     0m48.228s
 
 lambert at users.tcworks.net /home
 18:18:57 Thu Dec 16 $ time ls -l | wc -l
     3917
 
 real    1m0.295s
 user    0m12.076s
 sys     0m46.384s
 
 lambert at users.tcworks.net /home
 18:20:08 Thu Dec 16 $ 
 
 $ sudo vi /etc/nsswitch.conf
 
 lambert at users.tcworks.net /home
 18:20:49 Thu Dec 16 $ egrep '^(group|passwd):' /etc/nsswitch.conf
 group: files
 passwd: files
 
 lambert at users.tcworks.net /home
 18:20:53 Thu Dec 16 $ time ls -l | wc -l
     3917
 
 real    0m0.148s
 user    0m0.086s
 sys     0m0.074s
 
 lambert at users.tcworks.net /home
 18:20:59 Thu Dec 16 $ time ls -l | wc -l
     3917
 
 real    0m0.158s
 user    0m0.078s
 sys     0m0.089s
 
 lambert at users.tcworks.net /home
 18:21:00 Thu Dec 16 $ time ls -l | wc -l
     3917
 
 real    0m0.157s
 user    0m0.035s
 sys     0m0.134s
 
 -- 
 Scott Lambert                    KC5MLE                       Unix SysAdmin
 lambert at lambertfam.org