gvinum raid5 performance seems slow
Steve Peterson
stevep-hv at zpfe.com
Sun Oct 29 17:20:26 UTC 2006
Petri -- thanks for the idea.
I ran 2 dds in parallel; they took roughly twice as long in clock
time, and had about 1/2 the throughput of the single dd. On my
system it doesn't look like how the work is offered to the disk
subsystem matters.
# time dd if=/dev/zero of=blort1 bs=1m count=1000 & ; time dd
if=/dev/zero of=blort2 bs=1m count=1000&
[1] 69356
[2] 69357
#
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 382.511089 secs (2741296 bytes/sec)
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 385.677897 secs (2718787 bytes/sec)
[2] Done dd if=/dev/zero of=blort2 bs=1m count=1000
0.023u 16.913s 6:25.75 4.3% 20+2813k 3+8772io 2pf+0w
[1] + Done dd if=/dev/zero of=blort1 bs=1m count=1000
0.023u 16.880s 6:22.70 4.4% 20+2810k 1+8765io 0pf+0w
vmstat output during this run:
procs memory page disks
faults cpu
r b w avm fre flt re pi po fr sr ad0 ad4 ad6 ad8
ad10 in sy cs us sy id
1 1 0 101692 13132 0 0 0 0 5872 1857 0 54 62 69 59
2293 321 3914 2 62 35
3 1 0 101692 15576 0 4 0 0 5840 1947 0 52 54 60 57
2023 308 3379 2 64 35
2 1 0 101692 10072 0 0 0 0 5344 0 0 61 54 55 56
2133 368 3496 0 58 42
2 2 0 101692 11496 0 0 0 0 5984 1860 0 61 65 57 61
2377 429 4118 1 69 30
3 0 0 101692 13124 0 0 0 0 5568 1799 0 57 62 62 65
2440 466 4252 1 64 36
1 2 0 101692 12404 0 0 0 0 6176 1868 0 61 70 67 59
1923 482 3337 1 64 36
0 3 0 101692 15080 0 0 0 0 6576 1805 0 67 71 65 68
2423 468 4300 1 68 31
0 2 0 101692 17448 0 0 0 0 5760 1792 0 56 57 57 60
2463 399 4155 4 65 31
That got me thinking about whether there was a bottleneck to the disk
somewhere else. I ran a dd reading 1GB simultaneous from each of the
4 disks. Each disk could be read at about 25MB/sec for a combined
read transfer rate of 100MB/sec. A snipped of iostat output:
tty ad0 ad4 ad6
ad8 ad10 cpu
tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t
tps MB/s KB/t tps MB/s us ni sy in id
0 334 0.00 0 0.00 127.36 201 24.97 126.73 201
24.85 127.35 198 24.60 126.71 199 24.60 2 0 11 12 75
0 115 0.00 0 0.00 128.00 191 23.86 126.98 188
23.30 128.00 196 24.48 127.36 199 24.73 2 0 15 9 73
0 115 0.00 0 0.00 127.37 203 25.23 127.08 208
25.79 126.75 205 25.35 127.27 175 21.73 2 0 9 9 80
So I'm definitely not getting the full 60MB/sec performance out of
the disk with my current controller and mobo.
I guess the fundamental question is this -- if I have a 4 disk
subsystem that supports an aggregate ~100MB/sec transfer raw to the
underlying disks, is it reasonable to expect a ~5MB/sec transfer rate
for a RAID5 hosted on that subsystem -- a 95% overhead.
Steve
At 01:19 PM 10/28/2006, Petri Helenius wrote:
>According to my understanding vinum does not overlap requests to
>multiple disks when running in raid5 configuration so you're not
>going to achieve good numbers with just "single stream" tests.
>
>Pete
>
>
>Steve Peterson wrote:
>>Eric -- thanks for looking at my issue. Here's a dd reading from
>>one of the disks underlying the array (the others have basically
>>the same performance):
>>
>>$ time dd if=/dev/ad10 of=/dev/null bs=1m count=1000
>>1000+0 records in
>>1000+0 records out
>>1048576000 bytes transferred in 15.322421 secs (68434095 bytes/sec)
>>0.008u 0.506s 0:15.33 3.2% 20+2715k 0+0io 0pf+0w
>>
>>and here's a dd reading from the raw gvinum device /dev/gvinum/vol1:
>>
>>$ time dd if=/dev/gvinum/vol1 of=/dev/null bs=1m count=1000
>>1000+0 records in
>>1000+0 records out
>>1048576000 bytes transferred in 25.870684 secs (40531437 bytes/sec)
>>0.006u 0.552s 0:25.88 2.1% 23+3145k 0+0io 0pf+0w
>>
>>Is there a way to nondestructively write to the raw disk or gvinum device?
>>
>>For comparison, here's a read against the raw PATA device on the machine:
>>
>>$ time dd if=/dev/ad0 of=/dev/null bs=1m count=1000
>>1000+0 records in
>>1000+0 records out
>>1048576000 bytes transferred in 26.096070 secs (40181376 bytes/sec)
>>0.013u 0.538s 0:26.10 2.0% 24+3322k 0+0io 0pf+0w
>>
>>Steve
>>
>>
>>At 11:14 PM 10/27/2006, Eric Anderson wrote:
>>>On 10/27/06 18:03, Steve Peterson wrote:
>>>>I recently set up a media server for home use and decided to try
>>>>the gvinum raid5 support to learn about it and see how it
>>>>performs. It seems slower than I'd expect -- a little under
>>>>6MB/second, with about 50 IOs/drive/second -- and I'm trying to
>>>>understand why. Any assistance/pointers would be appreciated.
>>>>The disk system consists of 4 Seagate NL35 SATA ST3250623NS
>>>>drives connected to a Promise TX4300 (PDC40719) controller,
>>>>organized as a RAID5 volume via gvinum using this configuration:
>>>>drive drive01 device /dev/ad10
>>>>drive drive02 device /dev/ad6
>>>>drive drive03 device /dev/ad4
>>>>drive drive04 device /dev/ad8
>>>>volume vol1
>>>> plex org raid5 256k
>>>> sd length 200001m drive drive01
>>>> sd length 200001m drive drive02
>>>> sd length 200001m drive drive03
>>>> sd length 200001m drive drive04
>>>>dd reports the following performance on a 1G file write to the
>>>>RAID5 hosted volume:
>>>>$ time dd if=/dev/zero of=big.file bs=1m count=1000
>>>>1000+0 records in
>>>>1000+0 records out
>>>>1048576000 bytes transferred in 179.717742 secs (5834571 bytes/sec)
>>>> 179.76 real 0.02 user 16.60 sys
>>>>By comparison, creating the same file on the system disk (an old
>>>>ATA ST380021A connected via a SIS 730 on the motherboard):
>>>>time dd if=/dev/zero of=big.file bs=1m count=1000
>>>>1000+0 records in
>>>>1000+0 records out
>>>>1048576000 bytes transferred in 28.264056 secs (37099275 bytes/sec)
>>>> 28.32 real 0.01 user 19.13 sys
>>>>and vmstat reports about 280-300 IOs/second to that drive.
>>>>The CPU is pretty weak -- an Athlon 750. Is that the source of
>>>>my problem? If you look at the vmstat output below the machine
>>>>is busy but not pegged.
>>>
>>>
>>>Try the dd to the raw gvinum device instead of through a
>>>filesystem, and also to the individual disks. That will at least
>>>tell us where to look.
>>>
>>>
>>>Eric
>>>
>>>
>>>
>>>--
>>>------------------------------------------------------------------------
>>>Eric Anderson Sr. Systems Administrator Centaur Technology
>>>Anything that works is better than anything that doesn't.
>>>------------------------------------------------------------------------
>>>_______________________________________________
>>>freebsd-performance at freebsd.org mailing list
>>>http://lists.freebsd.org/mailman/listinfo/freebsd-performance
>>>To unsubscribe, send any mail to
>>>"freebsd-performance-unsubscribe at freebsd.org"
>>
>>
>>_______________________________________________
>>freebsd-performance at freebsd.org mailing list
>>http://lists.freebsd.org/mailman/listinfo/freebsd-performance
>>To unsubscribe, send any mail to
>>"freebsd-performance-unsubscribe at freebsd.org"
>
More information about the freebsd-performance
mailing list