GEOM profiling - how to?

[Lev Serebryakov]

Hello, Ivan.
You wrote 27 ноября 2010 г., 19:04:42:

>>    I'm  doing  some simple benchmarking of geom_raid5 in preparation of
>>    putting it into ports. And I notice strange results.
>>
>>    It is array of 5 disks, stripsize=128k. All disks are SATA2 disks on
>>    ICH10R, AHCI driver (8.1-STABLE).
>>
>>    Reading  from device itself (dd with bs=512K) gives speed of one HDD
>>    exactly.  gstat  shows  100% load of RAID geom and 1/5 of this speed
>>    (and 18-22% load) on all disk GEOMs.
> This "100% load of RAID geom" is an approximation of disk load, not CPU
   I understand this.
> load. I don't know how graid5 module works but if it's like most GEOM 
> modules, you will probably need to use a very small stripe size, 
> basically 128 / number_of_disks so that one request can span multiple 
> drives. In your case, try 32 KiB stripe size or 16 KiB stripe size.
  It  doesn't  work,  really.  Now  I  can  not  reconfigure array (it
  contains  a  lot  of  valuable  data),  but  I  did  very  extensive
  benchmarking  before  (about  2  years ago), and speed was best with
  128Kb stripe. Stripe of 32 or 16Kb gave about half of this speed.

  New round of benchamrking was insipired by upgrade to 8.x (and new
  AHCI  driver)  and  ampount  of  spare time which I want to spend on
  maintaince and improving RAID5, which is abadoned by author.

>>    Reading big file  from   FS   (dd   with   bs=512k,   FS   block  size  32K,
>>    vfs.read_max=32)  gives  about  twice  speed  and every disk GEOM is
>>    loaded  38-42%.  CPU time is about 8% system, 0.5% interrupt, so CPU
>>    is not a bottle neck.
> With big readahead (btw try larger read_max values, like 128) you get
> parallelism on the drive hardware level, not GEOM, this is why it works.
   Aagain,   going  above  "32"  (which  means  32*32  = 1024Kb, twice
  array-wide  stripe size) doesn't work better for one thread and work
  much  worse  in mult-thread loads accroding to my benchmarsk (it was
  multi-demensional  multi-day  run with varying stripe size, FS block
  size and pre-read setting).

     So,   these   numbers   are   local optimum   with   this   RAID
  implementation.   My   interest   is,   how  could  I  improve  RAID
  implementation  to  get better speeds. For this I need to understand
  what is bottle neck in it.

>>    How could I profile I/O and GEOM?
> There is no specific answer to this question; basically you can use
> gstat to observe performance of every GEOM device individually, and use
> "top" and similar to observe CPU usage. If you turn on GEOM logging,
   Yes,  I  know  these tools, but what I see here, that in both cases
"top"  GEOM (raid array) loaded 100%, but in one case underlying disks
loaded   for  20% (with individual read speeds about 20Mb/s, 80-90Mb/s
combined)  and  in  other  numbers are 100% and 40% (40Mb/s individual
speed and 150-160Mb/s combined). It doesn't help a lot :(

> your logs will be swamped by a huge number of messages which you can, in
> theory, create a tool to analyze them with.
   Thnaks, I'll try it.

-- 
// Black Lion AKA Lev Serebryakov <lev at FreeBSD.org>