PCIe SATA HBA for ZFS on -STABLE

[Matt Thyer]

On 8 June 2011 20:55, Matt Thyer <matt.thyer at gmail.com> wrote:

> On 7 June 2011 12:03, Matthew Dillon <dillon at apollo.backplane.com> wrote:
>
>>    The absolute cheapest solution is to buy a Sil-3132 PCIe card
>>    (providing 2 E-SATA ports), and then connect an external port
>> multiplier
>>    to each port.  External port multiplier enclosures typically support
>>    5 drives each so that would give you your 10 drives.
>>
>
> I've decided to avoid issues with port multiplication by going for a
> Supermicro AOC-USAS2-L8i and then to flash it to IT (as opposed to IR) mode
> to make it run as a standard non-RAID HBA.
>
> As I've got 8 x 2 TB drives for the ZFS raidz2 I'll put them all on the
> AOC-USAS2-L8i and save my onboard SATA-II ports for my 2 x 1TB drives for
> the FreeBSD O.S. and any eSATA use.
>
> Now the only remaining issue is whether to go with the Supermicro firmware
> or the generic Lsi Logic firmware as some have reported better performance
> with the version 9 Lsi Logic firmware.
>
> I'll report on my experiences (as I keep a record of the revision of my
> -STABLE build this should actually be useful!).
>
>
I could not be happier with the result.

As planned, I've put all 8 2TB Western Digital WD20EARS drives on the 8 lane
PCIe 2.0 AOC-USAS2-L8i (flashed with the latest SuperMicro firmware to
behave as an AOC-USAS2-L8e).

Many of you are now screaming "Why is he using the dreaded WD20EARS ?".  The
answer is that I bought the first 4 drives before I knew their issues and
later decided to continue with them once I knew how to mitigate their issues
by:


   - Using the DOS based WDIDLE3.EXE utility to change the default "park the
   heads after 8 seconds of idle time" to the maximum of 5 minutes
   - Avoiding alignment problems by putting ZFS on the whole disks (with no
   GPT or MBR partition table)
   - Convincing ZFS to use 4KiB transfers by creating the pool on top of
   4KiB block sized devices created with "gnop -S 4096"


And the resulting performance...

Previously I could sustain about 45 MiB/s writing from high powered Windows
7 machines (via Samba using asynchronous I/O) on to my old arrangement of 4
drives in raidz1 (using only the Intel on-board SATA ports on the H55
chipset motherboard).

I can now sustain 90 MiB/s over the network with the 8 drive raidz2 but
that's only because the Windows 7 machines can't feed the data fast enough.

I can see from "zpool iostat pool 1" that ZFS is idle for about 5-7 seconds
and then it will write at around 390 - 410 MB/s for about 3 seconds.

dd tests are:
> sudo dd if=/dev/zero of=/export/1G bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 3.218604 secs (333604841 bytes/sec)
> sudo dd if=/dev/random of=/export/1G bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 15.531615 secs (69132658 bytes/sec)

So I'm very happy that I can keep my home network users happy with the
limits now being due to my gigabit ethernet network (and I'm not going 10
GbE any time soon!).

This is on 8-STABLE at r220359 (~ 5th of April 2011).