Very inconsistent (read) speed on UFS2

[Lev Serebryakov]

Hello, Jeremy.
You wrote 31 августа 2011 г., 4:42:51:

> Furthermore, why are these benchmarks not providing speed data
> per-device (e.g. gstat or iostat -x data)?  There is a possibility that
> one of your drives could be performing at less-than-ideal rates (yes,
> intermittently) and therefore impacts (intermittently) your overall I/O
> throughput.
  Ok. I've run my benchamrk when `iostat -x -d -c 999999' is running.
  Results are like this:

device     r/s   w/s    kr/s    kw/s wait svc_t  %b
ada1     340.9 292.9 43138.8   146.5    0   1.2  42
ada2     340.9 293.9 43138.8   147.0    0   1.9  63
ada3     340.9 292.9 43044.7   146.5    0   1.5  57
ada4     341.9 292.9 43232.9   146.5    0   1.3  42
ada5     341.9 292.0 43138.8   146.0    2   1.3  40

  Yes, NUMBERS are different from sample to sample and oscillate from
16MB/s to 80Mb/s, but they VERY consistent among disks in question.
Slow read? All disks work slowly. Fast read? All disks work fast.
  I don't like this low-level speed oscillation too. I understand,
that something higher on stack cause it. And want to understand --
WHAT.

  What additionally surprise me:

1) benchmark induce some writing. atime modification? No, I've turned
   this one off, but it doesn't help. I afraid, that this read-write
   interleaving could be cause of "problems", but I don't understand,
   WHY here is some writing (1 writing per 2 reads in average) when
   read-only benchmark runs. It doesn't write any logs, etc. Yes,
   writing speed is very low, every write transaction is about 2Kb,
   but WHY they are here?! If I stop benchmark, here will be less than
   1 write transaction per second.

2) without `-x' it shows, that typical read transaction size is
   about 50Kb. It is very strange, as geom_raid5 shows (I have
   diagnostics in it), that almost all file access is aligned and is
   128Kb-sized...

P.S. Several samples for example of consistency in ONE sample and
inconsistency BETWEEN samples. Random pick from output, no editing,
they were in exact this order:

                        extended device statistics
device     r/s   w/s    kr/s    kw/s wait svc_t  %b
ada1     165.3  87.0 10515.9    43.5    2   5.0  50
ada2     165.3  87.0 10547.2    43.5    2   7.7  61
ada3     167.2  87.0 10703.7    43.5    1   6.1  55
ada4     165.3  87.0 10484.6    43.5    3   4.9  44
ada5     160.4  87.0 10265.5    43.5    5   5.1  48
                        extended device statistics
device     r/s   w/s    kr/s    kw/s wait svc_t  %b
ada1     884.1 350.9 56583.1   175.4    0   1.0  49
ada2     886.1 350.9 56677.2   175.4    0   1.3  58
ada3     882.2 349.9 56489.0   175.0    2   1.7  63
ada4     885.1 350.9 56614.5   175.4    0   1.4  64
ada5     887.1 350.9 56739.9   175.4    0   1.5  63
                        extended device statistics
device     r/s   w/s    kr/s    kw/s wait svc_t  %b
ada1     640.6 261.5 41001.3   130.8    0   0.9  40
ada2     639.7 261.5 40969.9   130.8    0   0.9  35
ada3     637.7 262.5 40844.5   131.3    0   1.5  46
ada4     640.6 260.6 41001.3   130.3    1   1.3  65
ada5     638.7 261.5 40875.9   130.8    0   1.3  46
                        extended device statistics
device     r/s   w/s    kr/s    kw/s wait svc_t  %b
ada1     243.7 102.8 15660.2    51.4    2   1.9  36
ada2     240.8 102.8 15503.6    51.4    3   1.9  43
ada3     242.7 103.7 15566.2    51.9    0   1.9  30
ada4     244.7 103.7 15785.5    51.9    2   2.4  56
ada5     243.7 102.8 15566.2    51.4    2   1.8  30

-- 
// Black Lion AKA Lev Serebryakov <lev at FreeBSD.org>