ZFS guidelines - preparing for future storage expansion

[James R. Van Artsdalen]

Andrew Snow wrote:
> Currently there is no "read-ahead" for scrubbing and resilvering, so
> it only talks to one disk and at a time and proceeds using only about
> half the I/O capacity of your disks (or less). Read-ahead is one of
> the planned features for ZFS next year.
gstat sometimes shows multiple outstanding I/O requests to a drive
during a scrub. All of the disks are lit up at the same time: there's no
one-disk-at-a-time.

I see roughly 500 MB/sec during a scrub, which is around 50% of the
theoretical bandwidth of both the disk-to-HBA links and the
HBA-to-system slot in my case.  I hope to be able to fix both this
spring and see if I can reach gigabyte-per-second levels, especially for
userland reads (I've seen 420 MB/s so far).

(each vdev in my case is a 2-way mirror so 500 MB/s of disk is 250 MB/s
of user data)

> Also, when your disks are 98% or more full and you are doing any
> writes at all ZFS spends a long time looking for free blocks with an
> inefficient algorithm.  An improved "disk full" algorithm is also
> planned for next year.

As the disk approaches 100% capacity the free space list(s) become
shorter, not longer.  It's fragmentation, or the need to search a long
time for a large block in the right area, that is likely the problem. 
If you can accept the block at the head of the list there is no search
at all.

A  quick snapshot during a scrub:

dT: 1.006s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0    238    238  30525    4.4      0      0    0.0   34.8| ada2
    0    235    235  30016    3.1      0      0    0.0   26.3| ada3
    0    274    274  35104    3.7      0      0    0.0   35.4| ada4
    0    277    277  35485    4.0      0      0    0.0   40.5| ada5
    0    273    273  34976    2.9      0      0    0.0   29.4| ada6
    4    270    270  34474    7.2      0      0    0.0   53.4| ada7
    0    271    271  34722    3.2      0      0    0.0   32.4| ada8
    5    270    270  34410    3.4      0      0    0.0   34.0| ada9
    7    268    268  34277    5.8      0      0    0.0   43.6| ada10
    0    267    267  34213    4.3      0      0    0.0   32.1| ada11
    4    269    269  34468    5.7      0      0    0.0   41.6| ada12
    7    268    268  34277    4.7      0      0    0.0   33.4| ada13
    0    277    277  35421    5.1      0      0    0.0   36.5| ada14
    4    270    270  34595    5.4      0      0    0.0   37.5| ada15
    0    269    269  34468    6.3      0      0    0.0   43.9| ada16
    0    275    275  35167    6.2      0      0    0.0   44.9| ada17