Very inconsistent (read) speed on UFS2

[Daniel Kalchev]

On 31.08.11 11:36, Lev Serebryakov wrote:
> device     r/s   w/s    kr/s    kw/s wait svc_t  %b
> ada1     340.9 292.9 43138.8   146.5    0   1.2  42
> ada2     340.9 293.9 43138.8   147.0    0   1.9  63
> ada3     340.9 292.9 43044.7   146.5    0   1.5  57
> ada4     341.9 292.9 43232.9   146.5    0   1.3  42
> ada5     341.9 292.0 43138.8   146.0    2   1.3  40
>
Very interesting, this writes. You need to find out what is causing these.

Just some random thoughts:

This flapping may have something to do with the drives' internal caches. 
What are the drives?

SATA drives, unlike SAS have simplex communication with the host, that 
is, the drive cannot simultaneously read and write data and commands 
(from/to host). There might be some, perhaps locking contention in 
there? It is not contention for bandwidth obviously.

Most consumer drives have rather low IOPS performance. This is 
especially pronounced when there are both reads and writes. Your IOPS 
rate here is relatively high for such drive, although the busy 
percentage is low -- but then, it may not be accurate.

In any case, you cannot measure read performance as long as it 
intermixes with writes, especially as you noted that your RAID5 code has 
some non-obvious write characteristics/optimizations.

Daniel