8.0-RELEASE/amd64 - full ZFS install - low read and write disk performance

[Dan Naumov]

On Mon, Jan 25, 2010 at 9:34 AM, Dan Naumov <dan.naumov at gmail.com> wrote:
> On Mon, Jan 25, 2010 at 7:33 AM, Bob Friesenhahn
> <bfriesen at simple.dallas.tx.us> wrote:
>> On Mon, 25 Jan 2010, Dan Naumov wrote:
>>>
>>> I've checked with the manufacturer and it seems that the Sil3124 in
>>> this NAS is indeed a PCI card. More info on the card in question is
>>> available at
>>> http://green-pcs.co.uk/2009/01/28/tranquil-bbs2-those-pci-cards/
>>> I have the card described later on the page, the one with 4 SATA ports
>>> and no eSATA. Alright, so it being PCI is probably a bottleneck in
>>> some ways, but that still doesn't explain the performance THAT bad,
>>> considering that same hardware, same disks, same disk controller push
>>> over 65mb/s in both reads and writes in Win2008. And agian, I am
>>> pretty sure that I've had "close to expected" results when I was
>>
>> The slow PCI bus and this card look like the bottleneck to me. Remember that
>> your Win2008 tests were with just one disk, your zfs performance with just
>> one disk was similar to Win2008, and your zfs performance with a mirror was
>> just under 1/2 that.
>>
>> I don't think that your performance results are necessarily out of line for
>> the hardware you are using.
>>
>> On an old Sun SPARC workstation with retrofitted 15K RPM drives on Ultra-160
>> SCSI channel, I see a zfs mirror write performance of 67,317KB/second and a
>> read performance of 124,347KB/second.  The drives themselves are capable of
>> 100MB/second range performance. Similar to yourself, I see 1/2 the write
>> performance due to bandwidth limitations.
>>
>> Bob
>
> There is lots of very sweet irony in my particular situiation.
> Initially I was planning to use a single X25-M 80gb SSD in the
> motherboard sata port for the actual OS installation as well as to
> dedicate 50gb of it to a become a designaed L2ARC vdev for my ZFS
> mirrors. The SSD attached to the motherboard port would be recognized
> only as a SATA150 device for some reason, but I was still seeing
> 150mb/s throughput and sub 0.1 ms latencies on that disk simply
> because of how crazy good the X25-M's are. However I ended up having
> very bad issues with the Icydock 2,5" to 3,5" converter jacket I was
> using to keep/fit the SSD in the system and it would randomly drop
> write IO on heavy load due to bad connectors. Having finally figured
> out the cause of my OS installations to the SSD going belly up during
> applying updates, I decided to move the SSD to my desktop and use it
> there instead, additionally thinking that my perhaps my idea of the
> SSD was crazy overkill for what I need the system to do. Ironically
> now that I am seeing how horrible the performance is when I am
> operating on the mirror through this PCI card, I realize that
> actually, my idea was pretty bloody brilliant, I just didn't really
> know why at the time.
>
> An L2ARC device on the motherboard port would really help me with
> random read IO, but to work around the utterly poor write performance,
> I would also need a dedicaled SLOG ZIL device. The catch is that while
> L2ARC devices and be removed from the pool at will (should the device
> up and die all of a sudden), the dedicated ZILs cannot and currently a
> "missing" ZIL device will render the pool it's included in be unable
> to import and become inaccessible. There is some work happening in
> Solaris to implement removing SLOGs from a pool, but that work hasn't
> yet found it's way in FreeBSD yet.
>
>
> - Sincerely,
> Dan Naumov

OK final question: if/when I go about adding more disks to the system
and want redundancy, am I right in thinking that: ZFS pool of
disk1+disk2 mirror + disk3+disk4 mirror (a la RAID10) would completely
murder my write and read performance even way below the current 28mb/s
/ 50mb/s I am seeing with 2 disks on that PCI controller and that in
order to have the least negative impact, I should simply have 2
independent mirrors in 2 independent pools (with the 5th disk slot in
the NAS given to a non-redundant single disk running off the one
available SATA port on the motherboard)?

- Sincerely,
Dan Naumov