ZFS I/O Throughput question..

[Chris Watson]

I have been testing ZFS on a home box now for a few days and I have a  
question that is perplexing me. Everything I have read on ZFS says in  
almost every case mirroring is faster than raidz. So I initially setup  
a 2x2 Raid 10 striped mirror. Like so:

priyanka# zpool status
   pool: tank
  state: ONLINE
  scrub: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	tank        ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    ada2    ONLINE       0     0     0
	    ada3    ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    ada4    ONLINE       0     0     0
	    ada5    ONLINE       0     0     0

errors: No known data errors
priyanka#

With this configuration I am getting the following throughput for reads:

priyanka# dd if=/dev/zero of=/tank/Aperture/test01 bs=1m count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 98.533820 secs (106417878 bytes/sec)
priyanka#

And for reads:

priyanka# dd if=/tank/Aperture/test01 of=/dev/null bs=1m
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 50.309988 secs (208423027 bytes/sec)
priyanka#

So basically 100MB/writes, 200MB/reads.

I thought the disks I have would do a little better than that assuming  
from much of the zfs literature proclaiming mirroring to be fastest  
with more I/O and more OPS/sec. Well I decided to blow away the mirror  
and instead do a 4 disk raidz to see just how much faster mirroring  
was with ZFS vs raidz. This is where I was blown away and more than a  
little confused.

priyanka# zpool status
   pool: tank
  state: ONLINE
  scrub: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	tank        ONLINE       0     0     0
	  raidz1    ONLINE       0     0     0
	    ada2    ONLINE       0     0     0
	    ada3    ONLINE       0     0     0
	    ada4    ONLINE       0     0     0
	    ada5    ONLINE       0     0     0

errors: No known data errors
priyanka#

Write performance:

priyanka# dd if=/dev/zero of=/tank/test.001 bs=1m count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 34.310930 secs (305609903 bytes/sec)
priyanka#

Read performance:

priyanka# dd if=/tank/test.001 of=/dev/null bs=1m count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 31.463025 secs (333272467 bytes/sec)
priyanka#

Say whaaaaaat?! Perhaps I am completely misunderstanding every zfs  
admin guide, FAQ and paper on ZFS. But everything I have read says  
mirroring should be much faster than a raidz and should almost always  
be preferred. Which clearly from above is not the case. The only thing  
I can think of is that the dd "benchmark" is not accurate because it  
is writing data sequentially? Which is the place raidz has an edge  
over mirroring, again from what I have read. But the above is not so  
much an 'edge' in performance as much as a complete and total data  
rape. So my question is, is everything i've read about ZFS and  
mirroring vs raidz wrong? Is the benchmark horribly flawed? Is raidz  
actually faster versus mirroring? Does FreeBSD perform some kind of  
voodoo h0h0magic that makes raidz perform much better than mirroring  
in ZFS than other platforms? Or am I just having a really weird dream  
and none of this is real.

Thank you for any comments or light anyone cares to share on this.

Chris

The pertinent specs for the machine are:

Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-STABLE #8: Sun Sep 12 01:00:49 CDT 2010
     root at priyanka.open-systems.net:/usr/obj/usr/src/sys/PRIYANKA amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Phenom(tm) II X4 965 Processor (3415.13-MHz K8-class CPU)
   Origin = "AuthenticAMD"  Id = 0x100f43  Family = 10  Model = 4   
Stepping = 3
    
Features 
= 
0x178bfbff 
< 
FPU 
,VME 
,DE 
,PSE 
,TSC 
,MSR 
,PAE 
,MCE 
,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
   Features2=0x802009<SSE3,MON,CX16,POPCNT>
   AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM, 
3DNow!+,3DNow!>
   AMD  
Features2 
= 
0x37ff 
<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT>
   TSC: P-state invariant
real memory  = 8589934592 (8192 MB)
avail memory = 8259407872 (7876 MB)
ACPI APIC Table: <GBT    GBTUACPI>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  1
  cpu2 (AP): APIC ID:  2
  cpu3 (AP): APIC ID:  3
ioapic0: Changing APIC ID to 2
ioapic0 <Version 2.1> irqs 0-23 on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <GBT GBTUACPI> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, cfce0000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff  
on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xee00-0xeeff mem  
0xd0000000-0xdfffffff,0xfdde0000-0xfddeffff irq 18 at device 0.0 on pci1
hdac0: <ATI RV770 High Definition Audio Controller> mem  
0xfddfc000-0xfddfffff irq 19 at device 0.1 on pci1
hdac0: HDA Driver Revision: 20100226_0142
hdac0: [ITHREAD]
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pci2: <serial bus, USB> at device 0.0 (no driver attached)
pcib3: <ACPI PCI-PCI bridge> irq 17 at device 5.0 on pci0
pci3: <ACPI PCI bus> on pcib3
ahci0: <AHCI SATA controller> port  
0xcf00-0xcf07,0xce00-0xce03,0xcd00-0xcd07,0xcc00-0xcc03,0xcb00-0xcb0f  
mem 0xfdaff000-0xfdaff7ff irq 17 at device 0.0 on pci3
ahci0: [ITHREAD]
ahci0: AHCI v1.20 with 8 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich0: [ITHREAD]
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich1: [ITHREAD]
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich2: [ITHREAD]
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich3: [ITHREAD]
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich4: [ITHREAD]
ahcich5: <AHCI channel> at channel 5 on ahci0
ahcich5: [ITHREAD]
ahcich6: <AHCI channel> at channel 6 on ahci0
ahcich6: [ITHREAD]
ahcich7: <AHCI channel> at channel 7 on ahci0
ahcich7: [ITHREAD]
pcib4: <ACPI PCI-PCI bridge> irq 19 at device 7.0 on pci0
pci4: <ACPI PCI bus> on pcib4
re0: <RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet> port  
0xbe00-0xbeff mem 0xfd7ff000-0xfd7fffff,0xfd7f8000-0xfd7fbfff irq 19  
at device 0.0 on pci4
re0: Using 1 MSI messages
re0: Chip rev. 0x28000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
rgephy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,  
1000baseT-FDX, auto
re0: Ethernet address: 6c:f0:49:5e:70:dd
re0: [FILTER]
pcib5: <ACPI PCI-PCI bridge> irq 17 at device 9.0 on pci0
pci5: <ACPI PCI bus> on pcib5
re1: <RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet> port  
0xae00-0xaeff mem 0xfd3ff000-0xfd3fffff,0xfd3f8000-0xfd3fbfff irq 17  
at device 0.0 on pci5
re1: Using 1 MSI messages
re1: Chip rev. 0x28000000
re1: MAC rev. 0x00000000
miibus1: <MII bus> on re1
rgephy1: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus1
rgephy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,  
1000baseT-FDX, auto
re1: Ethernet address: 6c:f0:49:5e:70:cd
re1: [FILTER]
pcib6: <ACPI PCI-PCI bridge> irq 18 at device 10.0 on pci0
pci6: <ACPI PCI bus> on pcib6
pci6: <mass storage, SATA> at device 0.0 (no driver attached)
ahci1: <ATI IXP700 AHCI SATA controller> port  
0xff00-0xff07,0xfe00-0xfe03,0xfd00-0xfd07,0xfc00-0xfc03,0xfb00-0xfb0f  
mem 0xfe02f000-0xfe02f3ff irq 22 at device 17.0 on pci0
ahci1: [ITHREAD]
ahci1: AHCI v1.10 with 6 3Gbps ports, Port Multiplier supported
ahcich8: <AHCI channel> at channel 0 on ahci1
ahcich8: [ITHREAD]
ahcich9: <AHCI channel> at channel 1 on ahci1
ahcich9: [ITHREAD]
ahcich10: <AHCI channel> at channel 2 on ahci1
ahcich10: [ITHREAD]
ahcich11: <AHCI channel> at channel 3 on ahci1
ahcich11: [ITHREAD]
ahcich12: <AHCI channel> at channel 4 on ahci1
ahcich12: [ITHREAD]
ahcich13: <AHCI channel> at channel 5 on ahci1
ahcich13: [ITHREAD]
ohci0: <OHCI (generic) USB controller> mem 0xfe02e000-0xfe02efff irq  
16 at device 18.0 on pci0
ohci0: [ITHREAD]
usbus0: <OHCI (generic) USB controller> on ohci0
ohci1: <OHCI (generic) USB controller> mem 0xfe02d000-0xfe02dfff irq  
16 at device 18.1 on pci0
ohci1: [ITHREAD]
usbus1: <OHCI (generic) USB controller> on ohci1
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfe02c000-0xfe02c0ff  
irq 17 at device 18.2 on pci0
ehci0: [ITHREAD]
usbus2: EHCI version 1.0
usbus2: <EHCI (generic) USB 2.0 controller> on ehci0
ohci2: <OHCI (generic) USB controller> mem 0xfe02b000-0xfe02bfff irq  
18 at device 19.0 on pci0
ohci2: [ITHREAD]
usbus3: <OHCI (generic) USB controller> on ohci2
ohci3: <OHCI (generic) USB controller> mem 0xfe02a000-0xfe02afff irq  
18 at device 19.1 on pci0
ohci3: [ITHREAD]
usbus4: <OHCI (generic) USB controller> on ohci3
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfe029000-0xfe0290ff  
irq 19 at device 19.2 on pci0
ehci1: [ITHREAD]
usbus5: EHCI version 1.0
usbus5: <EHCI (generic) USB 2.0 controller> on ehci1
pci0: <serial bus, SMBus> at device 20.0 (no driver attached)
pci0: <mass storage, ATA> at device 20.1 (no driver attached)
hdac1: <ATI SB600 High Definition Audio Controller> mem  
0xfe024000-0xfe027fff irq 16 at device 20.2 on pci0
hdac1: HDA Driver Revision: 20100226_0142
hdac1: [ITHREAD]
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
pcib7: <ACPI PCI-PCI bridge> at device 20.4 on pci0
pci7: <ACPI PCI bus> on pcib7
fwohci0: <Texas Instruments TSB43AB23> mem 0xfd5ff000-0xfd5ff7ff, 
0xfd5f8000-0xfd5fbfff irq 22 at device 14.0 on pci7
fwohci0: [ITHREAD]
fwohci0: OHCI version 1.10 (ROM=0)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:5b:e0:78:00:6c:f0:49
fwohci0: Phy 1394a available S400, 3 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
fwe0: <Ethernet over FireWire> on firewire0
if_fwe0: Fake Ethernet address: 02:5b:e0:6c:f0:49
fwe0: Ethernet address: 02:5b:e0:6c:f0:49
fwip0: <IP over FireWire> on firewire0
fwip0: Firewire address: 00:5b:e0:78:00:6c:f0:49 @ 0xfffe00000000,  
S400, maxrec 2048
dcons_crom0: <dcons configuration ROM> on firewire0
dcons_crom0: bus_addr 0xcfd2c000
fwohci0: Initiate bus reset
fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1,  
CYCLEMASTER mode
ohci4: <OHCI (generic) USB controller> mem 0xfe028000-0xfe028fff irq  
18 at device 20.5 on pci0
ohci4: [ITHREAD]
usbus6: <OHCI (generic) USB controller> on ohci4
amdtemp0: <AMD K8 Thermal Sensors> on hostb4
atrtc0: <AT realtime clock> port 0x70-0x73 on acpi0
orm0: <ISA Option ROM> at iomem 0xd0000-0xd2fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on  
isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model Generic PS/2 mouse, device ID 0
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Timecounters tick every 1.000 msec
firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0)  (me)
firewire0: bus manager 0
hdac0: HDA Codec #0: ATI R6xx HDMI
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 480Mbps High Speed USB v2.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 480Mbps High Speed USB v2.0
usbus6: 12Mbps Full Speed USB v1.0
pcm0: <HDA ATI R6xx HDMI PCM #0 HDMI> at cad 0 nid 1 on hdac0
hdac1: HDA Codec #0: Realtek ALC889
pcm1: <HDA Realtek ALC889 PCM #0 Analog> at cad 0 nid 1 on hdac1
pcm2: <HDA Realtek ALC889 PCM #1 Analog> at cad 0 nid 1 on hdac1
pcm3: <HDA Realtek ALC889 PCM #2 Digital> at cad 0 nid 1 on hdac1
pcm4: <HDA Realtek ALC889 PCM #3 Digital> at cad 0 nid 1 on hdac1
ugen0.1: <ATI> at usbus0
uhub0: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <ATI> at usbus1
uhub1: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <ATI> at usbus2
uhub2: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
ugen3.1: <ATI> at usbus3
uhub3: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <ATI> at usbus4
uhub4: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <ATI> at usbus5
uhub5: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
ugen6.1: <ATI> at usbus6
uhub6: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
uhub6: 2 ports with 2 removable, self powered
uhub0: 3 ports with 3 removable, self powered
uhub1: 3 ports with 3 removable, self powered
uhub3: 3 ports with 3 removable, self powered
uhub4: 3 ports with 3 removable, self powered
ada0 at ahcich8 bus 0 scbus8 target 0 lun 0
ada0: <SAMSUNG HD103SJ 1AJ10001> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada1 at ahcich9 bus 0 scbus9 target 0 lun 0
ada1: <SAMSUNG HD103SJ 1AJ10001> ATA-8 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada2 at ahcich10 bus 0 scbus10 target 0 lun 0
ada2: <SAMSUNG HD103SJ 1AJ10001> ATA-8 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada3 at ahcich11 bus 0 scbus11 target 0 lun 0
ada3: <SAMSUNG HD103SJ 1AJ10001> ATA-8 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada4 at ahcich12 bus 0 scbus12 target 0 lun 0
ada4: <SAMSUNG HD103SJ 1AJ10001> ATA-8 SATA 2.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada5 at ahcich13 bus 0 scbus13 target 0 lun 0
ada5: <SAMSUNG HD103SJ 1AJ10001> ATA-8 SATA 2.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
GEOM_MIRROR: Device mirror/gm0 launched (2/2).
GEOM: mirror/gm0s1: geometry does not match label (16h,63s != 255h,63s).
Root mount waiting for: usbus5 usbus2
Root mount waiting for: usbus5 usbus2
uhub2: 6 ports with 6 removable, self powered
uhub5: 6 ports with 6 removable, self powered
Trying to mount root from ufs:/dev/mirror/gm0s1a
ugen1.2: <Logitech> at usbus1
ums0: <Logitech USB RECEIVER, class 0/0, rev 1.10/25.10, addr 2> on  
usbus1
ums0: 16 buttons and [XYZT] coordinates ID=0
GEOM_ELI: Device mirror/gm0s1b.eli created.
GEOM_ELI: Encryption: AES-CBC 256
GEOM_ELI:     Crypto: software
WARNING: /usr/src was not properly dismounted
WARNING: /var was not properly dismounted
ugen1.3: <Logitech> at usbus1
ukbd0: <Logitech Logitech USB Keyboard, class 0/0, rev 1.10/28.00,  
addr 3> on usbus1
kbd1 at ukbd0
ZFS filesystem version 3
ZFS storage pool version 14