8.0-RELEASE/amd64 - full ZFS install - low read and write disk performance

[Dan Naumov]

On Sun, Jan 24, 2010 at 11:53 PM, Alexander Motin <mav at freebsd.org> wrote:
> Dan Naumov wrote:
>> This works out to 1GB in 36,2 seconds / 28,2mb/s in the first test and
>> 4GB in 143.8 seconds / 28,4mb/s and somewhat consistent with the
>> bonnie results. It also sadly seems to confirm the very slow speed :(
>> The disks are attached to a 4-port Sil3124 controller and again, my
>> Windows benchmarks showing 65mb/s+ were done on exact same machine,
>> with same disks attached to the same controller. Only difference was
>> that in Windows the disks weren't in a mirror configuration but were
>> tested individually. I do understand that a mirror setup offers
>> roughly the same write speed as individual disk, while the read speed
>> usually varies from "equal to individual disk speed" to "nearly the
>> throughput of both disks combined" depending on the implementation,
>> but there is no obvious reason I am seeing why my setup offers both
>> read and write speeds roughly 1/3 to 1/2 of what the individual disks
>> are capable of. Dmesg shows:
>>
>> atapci0: <SiI 3124 SATA300 controller> port 0x1000-0x100f mem
>> 0x90108000-0x9010807f,0x90100000-0x90107fff irq 21 at device 0.0 on
>> pci4
>> ad8: 1907729MB <WDC WD20EADS-32R6B0 01.00A01> at ata4-master SATA300
>> ad10: 1907729MB <WDC WD20EADS-00R6B0 01.00A01> at ata5-master SATA300
>
> 8.0-RELEASE, and especially 8-STABLE provide alternative, much more
> functional driver for this controller, named siis(4). If your SiI3124
> card installed into proper bus (PCI-X or PCIe x4/x8), it can be really
> fast (up to 1GB/s was measured).
>
> --
> Alexander Motin

Sadly, it seems that utilizing the new siis driver doesn't do much good:

Before utilizing siis:

iozone -s 4096M -r 512 -i0 -i1
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         4194304     512   28796   28766    51610    50695
		
After enabling siis in loader.conf (and ensuring the disks show up as ada):

iozone -s 4096M -r 512 -i0 -i1

                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         4194304     512   28781   28897    47214    50540

I've checked with the manufacturer and it seems that the Sil3124 in
this NAS is indeed a PCI card. More info on the card in question is
available at http://green-pcs.co.uk/2009/01/28/tranquil-bbs2-those-pci-cards/
I have the card described later on the page, the one with 4 SATA ports
and no eSATA. Alright, so it being PCI is probably a bottleneck in
some ways, but that still doesn't explain the performance THAT bad,
considering that same hardware, same disks, same disk controller push
over 65mb/s in both reads and writes in Win2008. And agian, I am
pretty sure that I've had "close to expected" results when I was
booting an UFS FreeBSD installation off an SSD (attached directly to
SATA port on the motherboard) while running the same kinds of
benchmarks with Bonnie and DD on a ZFS mirror made directly on top of
2 raw disks.


- Sincerely,
Dan Naumov