Poor ZFS performance

[Jordan Hubbard]

> On Jan 21, 2017, at 12:49 AM, Alex Povolotsky <tarkhil at webmail.sub.ru <mailto:tarkhil at webmail.sub.ru>> wrote:
> 
> I'm writing lots of (tens of millions) relatively small files, hashing them out in three-level directory, 100 entries per level.

You’re never going to get good performance doing that.   The constraints placed on ZFS’ design for directories (UNIX API, POSIX compliance, etc) and small file representation formats means it’ll never be a “database” - the filesystem's design, to say nothing of UNIX’s directory iteration / lookup APIs, just aren't optimized for millions of small files because it was never the intention that any UNIX filesystem be a low-cost KVS or database analog.  Things will get quickly pathological from a performance perspective and fixing the pathologies would require such a significant redesign of a number of different pieces of the puzzle here that it’s never likely to happen.

Your application would be far better served by using an actual database.  I’m not just suggesting this as a hypothetical, either.  I’ve dealt with several folks who went down this path, storing millions aornd even billions of small files into ZFS, and the results have never been pretty, nor have there been any easy options or “simple tunables” that were going to make those scenarios significantly prettier.  The advice was the same:  This needs to be a database, and there are plenty of those to choose from with all kinds of performance / consistency / redundancy characteristics to pick and choose between.

- Jordan