Huge (100k+) number of processes

[Charles Swiger]

On Apr 26, 2006, at 10:50 AM, Dan Nelson wrote:
>> The problem is that kern.maxproc=200000 line in /boot/loader.conf
>> only raised the real value to 21576. Is that authoritative, or should
>> I just try and run 200k processes? I only have 512Mb of RAM, and I
>> wonder if it's enough to run 100k+ of, say, /bin/sleep?
>
> Make sure you have enough RAM to cover those 100k processes.  There's
> code that limits maxproc to physpages/12 ("Limit maxproc so that kmap
> entries cannot be exhausted by processes").  You can try removing that
> code, but each process does require a couple of private pages.
> Assuming 5 pages, you'll need 2 GB of RAM just for that overhead,
> before you take into account any memory your application will  
> allocate.

And then add at least three, maybe four pages-- one for TEXT, one for  
BSS, one for the heap, and one for the stack) for the most minimal  
reasonable process, and anything which uses libc is going to run  
dozens to ~100 pages-- /bin/sleep wants 410K resident, for example.

The pure text pages are not duplicated for each process, but even so,  
you'd need on the order of 20GB or RAM or swap to run 100,000  
instances of sleep.

FreeBSD scales into the thousands of processes OK, and might do 10K  
or 20K under extreme circumstances, but that's almost certainly not  
the right architecture to head towards; a smaller number of  
persistent server processes along the lines of apache's preforked  
children or a multithreaded or poll()/select() based concurrent  
design are going to be better....

-- 
-Chuck