Terrible ix performance

[Garrett Wollman]

In article <CAGE5yCpojnENZW+6SN9WyeE5RUzpUickE8dB8r0zGrJGBJ2Wqg at mail.gmail.com>,
Peter Wemm quotes some advice about ZFS filesystem vdev layout:
>"1. Virtual Devices Determine IOPS
>IOPS (I/O per second) are mostly a factor of the number of virtual
>devices (vdevs) in a zpool. They are not a factor of the raw number of
>disks in the zpool. This is probably the single most important thing
>to realize and understand, and is commonly not.
>
>ZFS stripes writes across vdevs (not individual disks). A vdev is
>typically IOPS bound to the speed of the slowest disk within it. So if
>you have one vdev of 100 disks, your zpool's raw IOPS potential is
>effectively only a single disk, not 100.
>" -- end quote
>
>I made this mistake myself a number of times before I found out.

Getting a bit off-topic for this mailing-list, but I'll note that the
fileservers that I built earlier this year (which also use the Intel
10G NIC) were able to get nearly good-as-local performance over NFS
when connected through a single layer-2 switch. 

This hardware platform can do (local) about 5 GB/s from ARC, or 1.6
GB/s from disk, both reading single-threaded with compression
disabled; writes are about 1.2 GB/s; a single-threaded FreeBSD NFS
client could do between 750 MB/s and 1 GB/s, or about 80% of full link
speed.  The disk configuration here is 11 x 8-disk RAID-Z2 with
separate ZIL and L2ARC, but reading is no faster on a 44 x 2-disk
mirror.  Both of these are on 7200-RPM drives on two 16-port LSI eSAS
HBAs with gmultipath interposed for labeling and fault-tolerance.
(Both layouts are largely determined by the hardware: we have four
24-drive shelves per server, one SSD and one hot spare per shelf,
leaving 88 drives, which only factorizes as 44x2, 22x4, or 11x8, and
22x4 RAID-Z2 has the same capacity as 44x2 mirrored but with higher
overhead.)

A simple four-disk, two-way mirror configuration (i.e., two mirror
vdevs of two disks each) on the same machine can do about 120 MB/s in
the same tests.

-GAWollman