Terrible disk performance with LSI / FreeBSD 9.2-RC1

[J David]

To follow up on this issue, at one point the stats were down to this:

                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
da0        0.0   0.0     0.0     0.0    0   0.0   0
da1        0.0   0.0     0.0     0.0    0   0.0   0
da2      127.9   0.0   202.3     0.0    1  47.5 100
da3      125.9   0.0   189.3     0.0    1  43.1  97
da4      127.9   0.0   189.8     0.0    1  45.8 100
da5      128.9   0.0   206.3     0.0    0  42.5  99
da6      127.9   0.0   202.3     0.0    1  46.2  98
da7        0.0 249.7     0.0   334.2   10  39.5 100

At some point, I figured out that 125 random iops is pretty much the
limit for 7200 RPM SATA drives.  So mostly what we're looking at here
is the resilver of a raidz2 is the pathological worst case.  Lesson
learned; raidz2 is just really not viable without some kind of sort on
the resilver operations.  Wish I understood ZFS well enough to do
something about that, but research suggests the problem is
non-trivial. :(

There also seems to be a separate ZFS issue related to having a very
large number of snapshots (e.g. hourly for several months on a couple
of filesystems).  Some combination of the OS updates we've been doing
trying to get this machine to 9.2-RC1 and deleting a ton of snapshots.
 It would be nice to know which it was; I guess we'll find out in a
few months.

So it seems like the combination of these two issues is mostly what
is/was plaguing us.

Thanks!