Millions of small files: best filesystem / best options

[Marcelo Araujo]

2012/5/30 Julian Elischer <julian at freebsd.org>

> On 5/29/12 2:15 AM, Alessio Focardi wrote:
>
>> I ran a Usenet server this way for quite a while with fairly good
>>>> results, though the average file size was a bit bigger, about 2K or
>>>> so.
>>>> I found that if I didn't use "-o space" that space optimization
>>>> wouldn't
>>>> kick in soon enough and I'd tend to run out of full blocks that
>>>> would be
>>>> needed for larger files.
>>>>
>>> Fragmentation is not a problem for me, mostly I will have a write
>> once-read many situation, still is not clear to me if "-o space" works in
>> the constraints of the block/fragment ratio, that in my case it would still
>> mean that I will have to use a 512 bytes subblock for every 200 byte files.
>>
>> ps
>>
>> really thank you for all of your help!
>>
>>  Maybe try use something other than a filesystem. The problem you will
> have is that the physical media is going to be formatted to 512 bytes or
> larger so every operation will be a read/modify/write if you try do to much
> packing..  As others have said it makes a big difference if you are talking
> about 20MB of data or 200GB of data too..  If it fits in ram, you could use
> 'mailbox format' and pack them all in one file and then just load it into
> ram when you need to access it.
>
> You could always buy one of our flash cards in key-value-store mode if you
> need 1.3 TB of 512 byte values at 400,000 values per second(read/write),
> but the pricetag might be a bit scary.. :-) (10k-ish)
>
>
>

Maybe you could take a look on Fusion-IO[1].

[1] http://www.fusionio.com/platforms/

-- 
Marcelo Araujo
araujo at FreeBSD.org