Millions of small files: best filesystem / best options

[Jason Hellenthal]

On Mon, May 28, 2012 at 07:01:08PM +0200, Alessio Focardi wrote:
> > FreeBSD handles this wonderfully thanks to all the people that have
> > put in time and effort over the years.
> 
> That's a great news and I recognize the effort of the community!
>  
> > I wasn't aware that FreeBSD was unique in this respect, but yes,
> > FreeBSD has a block size and a fragment size. While formatting a UFS
> > filesystem you can specify these sizes with the "-b SIZE" and "-f
> > SIZE" arguments to newfs(8), for example:
> > 
> > 	newfs -b 16384 -f 2048 /dev/da0s1a
> 
> -b block-size
> 	     The block size of the file system, in bytes.  It must be a power
> 	     of 2.  The default size is 16384 bytes, and the smallest allow-
> 	     able size is 4096 bytes.
> 
> -f frag-size
> 	     The fragment size of the file system in bytes.  It must be a
> 	     power of two ranging in value between blocksize/8 and blocksize.
> 	     The default is 2048 bytes.
> 
> So in my case I would have to use -b 4096 -f 512
> 
> It's an improvement, but still is not ideal: still a big waste with 200 bytes files!
> 
> ZFS with compression, maybe?
> 

ZFSv28 with dedup turned on...

Be careful of new instances of SPA working its way into the source tree
though. An upgrade could possibly make you incompatible with Solaris ZFS
systems if for some reason you ever wanted to change from FreeBSD.

-- 

 - (2^(N-1))