raid3 is slow

Thu Mar 29 08:41:26 UTC 2007

> On Wed, Mar 28, 2007 at 06:21:34AM -0700, R. B. Riddick wrote:
> > > # atacontrol mode ad0
> > > current mode = UDMA100
> > > # atacontrol mode ad1
> > > current mode = UDMA100
> > > # atacontrol mode ad2
> > > current mode = UDMA100
> > > # atacontrol mode ad3
> > > current mode = UDMA100
> > > # atacontrol mode ad6
> > > current mode = SATA150
> > >
> > Looks good...
>
> Does it? It seems there are two disks per ata channel, which is very bad
> for performance.
>
> I suggesting trying this:
>
>  # apply "dd if=/dev/ad%1 if=/dev/null bs=1m count=250 &" 0 1 2 3 6
>
> and once it completes try this:

>  # dd if=/dev/raid3/<name> of=/dev/null bs=1m count=1000

> and send the results here.

A SATA cable was wrong, and the 5th disk of the raid was not connected. Now 
i fix it.

Geom name: nmivol
State: COMPLETE
Components: 5
Flags: NONE
GenID: 0
SyncID: 77
ID: 2396346217
Zone64kFailed: 0
Zone64kRequested: 19808642
Zone16kFailed: 0
Zone16kRequested: 4629259
Zone4kFailed: 28270
Zone4kRequested: 2027159

# apply "dd if=/dev/ad%1 of=/dev/null bs=1m count=250 &" 0 1 2 3 4
250+0 records out
262144000 bytes transferred in 4.324466 secs (60618814 bytes/sec)
250+0 records in
250+0 records out
262144000 bytes transferred in 6.289833 secs (41677418 bytes/sec)
250+0 records in
250+0 records out
262144000 bytes transferred in 6.262564 secs (41858894 bytes/sec)
250+0 records in
250+0 records out
262144000 bytes transferred in 7.871292 secs (33303808 bytes/sec)
250+0 records in
250+0 records out
262144000 bytes transferred in 7.857484 secs (33362333 bytes/sec)

I forgot to tell, one label of the raid is a gbde encoded partition. If i do 
a read test from the raid, the load seems normal?!?! I don't know how high 
is normal.

# dd if=/dev/raid3/nmivol of=/dev/null bs=1m count=30000
30000+0 records in
30000+0 records out
31457280000 bytes transferred in 336.545001 secs (93471244 bytes/sec)

# top
last pid: 16191;  load averages:  0.32,  0.36,  0.49 
up 0+15:16:03  10:14:12
58 processes:  1 running, 57 sleeping
CPU states:  0.0% user,  0.0% nice, 35.9% system,  3.5% interrupt, 60.5% 
idle
Mem: 50M Active, 306M Inact, 165M Wired, 20K Cache, 111M Buf, 475M Free

# systat -vmstat
    7 users    Load  0.45  0.39  0.50                  Mar 29 10:13

Mem:KB    REAL            VIRTUAL                     VN PAGER  SWAP PAGER
        Tot   Share      Tot    Share    Free         in  out     in  out
Act   60900    8108   143488     9684  486056 count
All  533396   12740622626264    15932         pages
                                                          zfod   Interrupts
Proc:r  p  d  s  w    Csw  Trp  Sys  Int  Sof  Flt        cow    3986 total
           1 56     14559    2  418 4171   10    9 169268 wire   1000 0: clk
                                                    51560 act     128 8: rtc
38.9%Sys   3.3%Intr  0.0%User  0.0%Nice 57.8%Idl   313632 inact     9 10: 
fxp
|    |    |    |    |    |    |    |    |    |         20 cache     1 11: 
ata
===================++                              486036 free   1424 14: 
ata
                                                          daefr  1424 15: 
ata
Namei         Name-cache    Dir-cache                     prcfr
    Calls     hits    %     hits    %                     react
                                                          pdwake
                                                          pdpgs
Disks   ad0   ad1   ad2   ad3   ad4                       intrn
KB/t  31.97 31.97 31.97 31.97  3.12                113904 buf
tps     712   712   712   712     1                     3 dirtybuf
MB/s  22.22 22.22 22.22 22.22  0.00                 69984 desiredvnodes
% busy   33    71    32    59     0                 59556 numvnodes
                                                    17488 freevnodes

The same test from the encoded partition:

# dd if=/dev/raid3/nmivole.bde of=/dev/null bs=1m count=30000
30000+0 records in
30000+0 records out
31457280000 bytes transferred in 1282.804938 secs (24522263 bytes/sec)

# top
last pid: 16322;  load averages:  1.07,  0.66,  0.55 
up 0+15:22:28  10:20:37
61 processes:  1 running, 60 sleeping
CPU states:  0.4% user,  0.0% nice, 90.3% system,  0.4% interrupt,  8.9% 
idle
Mem: 58M Active, 306M Inact, 166M Wired, 20K Cache, 111M Buf, 466M Free

# systat -vmstat
    7 users    Load  1.04  0.70  0.57                  Mar 29 10:21

Mem:KB    REAL            VIRTUAL                     VN PAGER  SWAP PAGER
        Tot   Share      Tot    Share    Free         in  out     in  out
Act   60912    8108   143504     9684  486044 count
All  533408   12740622626280    15932         pages
                                                          zfod   Interrupts
Proc:r  p  d  s  w    Csw  Trp  Sys  Int  Sof  Flt        cow    3014 total
           1 56      8575    5  275 3181   20   13 169268 wire    999 0: clk
                                                    51572 act     128 8: rtc
81.8%Sys   2.5%Intr  0.2%User  0.0%Nice 15.5%Idl   313632 inact     8 10: 
fxp
|    |    |    |    |    |    |    |    |    |         20 cache     1 11: 
ata
=========================================+         486024 free    939 14: 
ata
                                                          daefr   939 15: 
ata
Namei         Name-cache    Dir-cache                     prcfr
    Calls     hits    %     hits    %                     react
                                                          pdwake
                                                          pdpgs
Disks   ad0   ad1   ad2   ad3   ad4                       intrn
KB/t  12.88 12.88 12.88 12.88  0.50                113904 buf
tps     470   470   470   470     1                    22 dirtybuf
MB/s   5.92  5.91  5.92  5.92  0.00                 69984 desiredvnodes
% busy   24    40    20    35     0                 59556 numvnodes
                                                    17489 freevnodes

Maybe is what the hardware can do :( , isn't it?

Matyee 