FreeBSD 5.x with > 70k users?

[Ted Mittelstaedt]

You need to try it and find out.  Since pwd_mkdb is what is failing the
db build, and it pays attention to the PW_SCAN_BIG_IDS variable, that
is going to be the very first question any of the developers ask.

While it is true that someone could setup UID's greater than 65535 with
less than 65535 actual password entries (for sheer idiocy if nothing
else)
it seems obvious that in a normal situation that you would only have
UID's greater than 65535 if you actually had more than 65535 entries in
the password file - hence the need for the variable.

Originally FreeBSD didn't use db files for passwords, that mechanism was
added because - you guessed it - the need to support very large password
files.  Linear scans of password files greater than 3-5K entries were
getting
noticeably slower.  The fact that the db mechanism exists at all, not to
mention that variable, most definitely indicates that someone at the time
wanted to support very large password files.  And I'm talking in the
hundreds of thousands of users.  The fact that it's broke now probably
means
someone accidentally used the wrong data type somewhere in the code
during one of the many revisions of FreeBSD.  Thus I would recommend
strongly
that as soon as you get the preliminary testing out of the way that you
file a PR since this really looks like an introduced bug IF IT BLOWS UP
on
the tests I outlined below.  And that bug might
be in a library that pwd_mkdb is using, not in the actual program, so the
core definitely needs to be looking at this, not one of us.

The FreeBSD password structure has already been fundamentally modified
from
the original ATT format to support 16 character usernames and md5
passwords,
there is no reason that the 65535 limit needs to be present either.  If a
few
apps break so what, bitch to those developers or fix them yourself.

I'm assuming this is a mailserver and if so, for <150K users there is no
reason that the current FreeBSD system authentication mechanism couldn't
handle
it with a fast CPU.  Sure, if you got 200-300K users then you want to
bypass the system authentication and use a SQL server - patches for that
for the major open source mail apps like /bin/mail, imapd, popper,
procmail
and such are floating around the Internet - but in your case the only
thing
you should need to do for 70K users is to patch the delivery agent and
the pop/imap server to use a heirarchial directory structure for the
mailboxes - once again, patches for that are also floating around the
Internet.

What you need to do for testing IMHO is do the following:

1) Generate a test password file of 70K users with a script as such:

user1::1:20::0:0:User1:/usr/home/user1:/bin/sh
user2::2:20::0:0:User2:/usr/home/user2:/bin/sh
user3::3:20::0:0:User3:/usr/home/user3:/bin/sh
.
.
.
user79997::79997:20::0:0:User79997:/usr/home/user79997:/bin/sh
user79998::79998:20::0:0:User79998:/usr/home/user79998:/bin/sh
user79999::79999:20::0:0:User79999:/usr/home/user79999:/bin/sh

and feed this to the pwd_mkdb program.  The reason why is that
with your production password file you have NO assurance that
every line in it is correct - it is possible that pwd_mkdb is
blowing up because of something wrong in your password file.

Remember pwd_mkdb is intended to be called by adduser, vipw and
the like, and so it has few provisions for error checking since it
assumes that whatever is fed to it is absolutely correctly formatted.

2)  If this blows chunks then start truncating it down by halves
until you find the exact number of lines that pwd_mkdb will build,
and the number that it blows chunks at.  The developers will need
to know this to help them know what to look for.

Good luck!

Ted

>-----Original Message-----
>From: Ben Hockenhull [mailto:benh at jpj.net]
>Sent: Monday, June 06, 2005 9:33 PM
>To: Ted Mittelstaedt
>Subject: RE: FreeBSD 5.x with > 70k users?
>
>
>I didn't, but I'm less concerned about pwd_mkdb complaining about UIDs >
>65535 than I am about the less than enlightening db build failure I get
>with the entire 71k+ file.
>
>/etc/master.passwd rebuilds just fine with UIDs > 65535 present (but
>considerably fewer than 65535 actual entries), but just
>complains about it.
>
>
>Will setting that variable fix the build failure?  I got the
>sense that it
>just suppresses the complaining.
>
>Ben
>
>>Did you set the PW_SCAN_BIG_IDS as documented in the man page for
>>pwd_mkdb?
>>
>>Ted
>>
>>>-----Original Message-----
>>>From: owner-freebsd-questions at freebsd.org
>>>[mailto:owner-freebsd-questions at freebsd.org]On Behalf Of Ben
>Hockenhull
>>>Sent: Monday, June 06, 2005 6:36 AM
>>>To: freebsd-questions at freebsd.org
>>>Subject: FreeBSD 5.x with > 70k users?
>>>
>>>
>>>Hi there,
>>>
>>>I'm attempting to migrate an old BSDI system to FreeBSD.  The
>system in
>>>question has about 71k users, with UIDs from about 2000 up to
>>>about 70000.
>>>When I import the master.passwd file (formats are the same) to the new
>>>system and try to rebuild the .db files, it fails with a
>pwd_mkdb: put:
>>>Unknown error: 0 .
>>>
>>>As best I can tell, if I import a small subset of the users
>(about 5k),
>>>things work fine.  From what I understand, FreeBSD can have
>>>massive UIDs,
>>>with the caveat that some applications may not like UIDs > 65535.
>>>
>>>I did some research and found someone reporting an identical
>>>problem, but
>>>didn't see that he'd found a solution.  Any input, pointers, solutions
>>>greatly appreciated.
>>>
>>>Ben
>>>
>>>
>>>_______________________________________________
>>>freebsd-questions at freebsd.org mailing list
>>>http://lists.freebsd.org/mailman/listinfo/freebsd-questions
>>>To unsubscribe, send any mail to
>>>"freebsd-questions-unsubscribe at freebsd.org"
>>>
>
>
>
>