Slow resilvering with mirrored ZIL

[Warren Block]

On Wed, 17 Jul 2013, Gezeala M. Bacuño II wrote:

> If ZFS goes on a bare drive, it will be aligned by default.  If ZFS is going in a partition, yes, align that partition to 4K boundaries or larger multiples of 4K, like 1M.
> 
> Your statement is enlightening and concise, exactly what I need. Thanks.
>
>       The gnop/ashift workaround is just to get ZFS to use the right block size.  So if you don't take care to get partition alignment right, you might end up using the right
>       block size but misaligned.
>
>       And yes, it will be nice to be able to just explicitly tell ZFS the block size to use.
> 
> 
> We do add the entire drive (no partitions) to ZFS, perform gnop/ashift and other necessary steps and then verify ashift=12 through zdb.
> 
> The gpart/gnop/ashift steps, if I understand correctly (do correct me if I'm stating this incorrectly), is needed for further SSD performance tuning. Taking into consideration leaving a
> certain chunk for wear leveling and also if the SSD has a size that may be too big for L2ARC.

Well, there are several things going on.

Partitions can be used for a couple of things.  Limiting the size of 
space available to ZFS, leaving an unallocated part of the drive for 
wear leveling.  Note that ZFS on FreeBSD now has TRIM, which should make 
leaving unused space on SSDs unnecessary.

Aligning partitions preserves performance.  If a partition is 
misaligned, writes can slow down to half speed.  For example, a 4K 
filesystem block written to an aligned partition writes a single block. 
If the partition is misaligned, that 4K write is split over two disk 
blocks.  Each block has to be read, partly modified, then written, 
taking roughly twice as long.

Finally, ZFS's ashift controls the minimum size of block ZFS uses. 
ashift=12 (12 bits) sets that to 4K blocks (2^12=4096).  Again, a 
performance thing, matching the filesystem block size to device block 
size.

It would be interesting to see a benchmark of ZFS on a 4K drive with 
different ashift values.