getting to 4K disk blocks in ZFS

[Don Lewis]

On 13 Oct, Steven Hartland wrote:
> ----- Original Message ----- 
> From: "Matthew D. Fuller" <fullermd at over-yonder.net>
> 
> 
>> On Mon, Oct 13, 2014 at 11:48:27AM -0700 I heard the voice of
>> Darren Pilgrim, and lo! it spake thus:
>>> 
>>> If the default is 4k and (for the limited time they're still common)
>>> you use true 512b disks, you can waste space.  Sure, but how much
>>> space?
>> 
>> The median file in /usr/ports is 408 bytes.  Over 90% of the files are
>> under 2k, which means the wastage for them is over 100% (before
>> counting what gain compression might get).  A little offhand mathery
>> says it's about 78% extra overhead on the whole.
>> 
>> And that includes the almost hundred megs (over 22% of the total size
>> of the FS) for the INDEX.db, plus the ~90 megs of the flat INDEX files
>> (another 20%).  If you pull those out, the overhead is 130%.
>> 
>> 
>> (To be sure, relatively few people have ports trees eating most of
>> their space, but still; it's pretty pathological.  I for one did
>> decide some years back to always force 4k on any new FSen to make
>> future life simpler, accepting the bloat, but it's there.)
> 
> And thats before you add the overhead if your running RAIDZ...
> 
> A good read on this is
> http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/

This is a timely subject.  I'm planning on moving my Cyrus imap mail
spool from a 4K/1K UFS filesystem to a three drive raidz1.  It looks
like the UFS fragmentation overhead is about 2.4%.  ZFS ashift=12
increases that to about 17%.  Combine that with raidz and now the
overhead is about 40%.  Ouch!