Max. number of opened files, efficiency

[Bill Moran]

In response to Laszlo Nagy <gandalf at shopzeus.com>:

> How many files can I open under FreeBSD, at the same time?
> 
> Problem: I'm making a pivot table, and when I drill down the facts, I 
> would like to create a new temporary file for each possible dimension 
> value. In most cases, there will be less than 1000 dimension values. I 
> tried to open 1000 temporary files and I could do so within one second.
> 
> But how efficient is that? What happens when I open 1000 temporary 
> files, and write data into them randomly, 10 million times. (avg. 10 000 
> write operations per file) Will this be handled efficiently by the OS? 
> Is efficiency affected by the underlying filesystem?
> 
> I also tried to create 10 000 temporary files, but performance dropped down.
> 
> Example in Python:
> 
> import tempfile
> import time
> N = 10000
> start = time.time()
> files = [ tempfile.TemporaryFile() for i in range(N)]
> stop = time.time()
> print "created %s files/second" % ( int(N/(stop-start)) )
> 
> On my computer this program prints "3814 files/second" for N=1000, and  
> "1561 files/second" for N=10000.

Directories generally start to perform poorly when you put too many files
in them (i.e. the time required to add a new directory entry or find
an existing name in the entry goes up)

If you're going to be making 10s of 1000s of files, I'd recommend making
a tree of directories.  I.e., make directories 1 - 10, then put files
0-999 in directory 1 and files 1000-1999 in directory 2, etc

-- 
Bill Moran
http://www.potentialtech.com