Reading from ZFS mirror 2x slower than expected?

[Urmas Lett]

Steve Polyack wrote:
> I can also vouch that the mirror performance is less than stellar.  
> Writes are also about 50% slower than single-disk performance, although 
> this may be expected due to checksums / other ZFS overhead.  I can try 
> to provide more details when I get time.

I'm luckily observing only 50% slowness when reading from mirror. 
Writing seems to be OK.

dd if=/dev/zero of=test.bin bs=1m count=8192
8589934592 bytes transferred in 143.145040 secs (60008608 bytes/sec)

systat -vm shows 400+ tps per disk

dd if=test.bin of=/dev/null bs=1m
8589934592 bytes transferred in 155.068721 secs (55394373 bytes/sec)

Disks  ada0  ada1  ada2 pass0 pass1 pass2
KB/t   0.00   128   128  0.00  0.00  0.00
tps       0   226   224     0     0     0
MB/s   0.00 28.30 28.06  0.00  0.00  0.00
%busy     0    54    64     0     0     0

systat -vm shows only 200+ tps per disk

After detaching one disk from mirror:

dd if=test.bin of=/dev/null bs=1m
8589934592 bytes transferred in 149.991512 secs (57269471 bytes/sec)

Disks  ada0  ada1  ada2 pass0 pass1 pass2
KB/t   0.00   128  0.00  0.00  0.00  0.00
tps       0   444     0     0     0     0
MB/s   0.00 55.55  0.00  0.00  0.00  0.00
%busy     0    98     0     0     0     0

systat -vm shows 400+ tps

Now I did simple test on the same hardware with Belenix 7.1 liveCD:

jack at belenix:/zpool0# dd if=/dev/zero of=8gb.test bs=1024k count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 167.512 s, 51.3 MB/s

jack at belenix:/zpool0# dd if=8gb.test of=/dev/null bs=1024k
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 82.4957 s, 104 MB/s

read speed from mirror is 2x better and Solaris iostat shows 400+ tps 
and 50+ MB/s per mirrored disk.

-- 
Urmas Lett

Tel:    +(372) 7 302 110
Fax:    +(372) 7 302 111
E-Mail: urmas.lett at eenet.ee