getting to 4K disk blocks in ZFS

[Don Lewis]

On 13 Oct, Lyndon Nerenberg wrote:
> 
> On Oct 13, 2014, at 1:47 PM, Don Lewis <truckman at FreeBSD.org> wrote:
> 
>> Combine that with raidz and now the
>> overhead is about 40%.  Ouch!
> 
> But ~33 of that 40% is generic RAID overhead, and has nothing to do
> with the 4K block size issue.  (I.e., you would have 33% hit even if
> you were running UFS on a three disk RAID5.)

Actually, I meant this as just the fragmentation overhead.  I was
incorrectly thinking that there was one parity block per stripe, and with
a three disk raidz1, I'd get two data sectors and one parity sector per
stripe, and that even a tiny file would get two data sectors and one
parity sector.  The 40% wasted space was calcualted based on an
effective 8K sector size.  Parity would then add another 50% to the
space.

It turns out that this isn't the way it works, at least according to my
inderstanding from the previously posted link.  A tiny file only gets
allocated a single data sector, but also gets its own dedicated parity
sector. This helps reduce the waste from fragmentation, but increases
the parity overhead. With ashift=12, if you filled the filessytem with
4K files, half the space would be consumed by parity blocks instead of
the expected 1/3rd.  If you reduce the file size to 512b, you still
can't fit in any more files because each file would still require a 4K
data sector and a 4K parity sector, so only 6.25% of the space consumed
would hold useful data.

It looks like my mail spool would grow by a factor of about 1.9 because
of fragmentation and parity overhead.  Actually, not as bad as I feared
it could be, but not much more efficient than a mirror.
 
> On any real-world system where you're running ZFS, it's unlikely the
> 4K block overhead is really going to be an issue.  And the underlying
> disk hardware is moving to 4K physical sectors, anyway.  Sooner or
> later you're just going to have to suck it up.

If you're storing enough small files to make it worthwhile to optimize
for that case, you might be better off with older technology 512b sector
drives than newer 4K sector drives that have to be 8x larger to store
the same amount of data.  Drive capacities aren't growing nearly as fast
anymore.  How long does it take drive capacity to go up by a factor of
8x?