Uneven load on drives in ZFS RAIDZ1

[Garrett Cooper]

On Mon, Dec 19, 2011 at 6:22 AM, Stefan Esser <se at freebsd.org> wrote:
> Hi ZFS users,
>
> for quite some time I have observed an uneven distribution of load
> between drives in a 4 * 2TB RAIDZ1 pool. The following is an excerpt of
> a longer log of 10 second averages logged with gstat:
>
> dT: 10.001s  w: 10.000s  filter: ^a?da?.$
>  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
>    0    130    106   4134    4.5     23   1033    5.2   48.8| ada0
>    0    131    111   3784    4.2     19   1007    4.0   47.6| ada1
>    0     90     66   2219    4.5     24   1031    5.1   31.7| ada2
>    1     81     58   2007    4.6     22   1023    2.3   28.1| ada3
>
>  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
>    1    132    104   4036    4.2     27   1129    5.3   45.2| ada0
>    0    129    103   3679    4.5     26   1115    6.8   47.6| ada1
>    1     91     61   2133    4.6     30   1129    1.9   29.6| ada2
>    0     81     56   1985    4.8     24   1102    6.0   29.4| ada3
>
>  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
>    1    148    108   4084    5.3     39   2511    7.2   55.5| ada0
>    1    141    104   3693    5.1     36   2505   10.4   54.4| ada1
>    1    102     62   2112    5.6     39   2508    5.5   35.4| ada2
>    0     99     60   2064    6.0     39   2483    3.7   36.1| ada3

This suggests (note that I said suggests) that there might be a slight
difference in the data path speeds or physical media as someone else
suggested; look at zpool iostat -v <interval> though before making a
firm statement as to whether or not a drive is truly not performing to
your assumed spec. gstat and zpool iostat -v suggest performance
though -- they aren't the end-all-be-all for determining drive
performance.

If the latency numbers were high enough, I would suggest dd'ing out to
the individual drives (i.e. remove the drive from the RAIDZ) to see if
there's a noticeable discrepancy, as this can indicate a bad cable,
backplane, or drive; from there I would start doing the physical swap
routine and see if the issue moves with the drive or stays static with
the controller channel and/or chassis slot.

Cheers,
-Garrett