ZFS (or something) is absurdly slow

[Thomas Steen Rasmussen]

On 24-11-2013 04:15, Eitan Adler wrote:
>
>> vfsstat.d https://forums.freebsd.org/showpost.php?p=182070&postcount=6
> I can run this script, what output should I be looking for?

Check the sample output on the page: It shows two lists, "Number
of operations" and "Bytes read or write". The lists are ordered
with the busiest at the bottom and seperated by filesystem
location. While things are slow, try running it to see if some
location on the filesystem is being hammered. One case I had was
where /home/pgsql/data was the culprit, due to the default sync
setting in postgres, along with a malfunctioning webapp that kept
making a lot of queries. This activity wasn't shown with
"top -m io -o total" for some reason.

Another thing you should probably do is run a SMART check on the
disk to see if something is wrong with it. I had another case with
a zfs mirror that performed appalingly, turned out it was because
one of the disks was dodgy, not in a way that made zfs show
checksum errors, but enough to make it really really slow. Since a
ZFS vdev only performs as good as the slowest disk in a vdev,
which in turn will slow the whole pool down, replacing the disk
made everything much better.

What does diskinfo -ct /dev/whatever say about the seek times on the
bad disk ? Are the results the same if you boot off of an usb stick and
test the disk when it is completely idle and independent of the running OS ?

/Thomas