Reading via mmap stinks (Re: weird bugs with mmap-ing via NFS)

[Peter Jeremy]

On Fri, 2006-Mar-24 15:18:00 -0500, Mikhail Teterin wrote:
>On the machine, where both mzip and the disk run at only 50%, the disk is a 
>plain SATA drive (mzip's state goes from "RUN" to "vnread" and back).
...
>   18 users    Load  0.46  0.53  0.60                  24 ??? 15:15
>
>Mem:KB    REAL            VIRTUAL                     VN PAGER  SWAP PAGER
>        Tot   Share      Tot    Share    Free         in  out     in  out
>Act 1833864    5880 27758552    45268   92216 count  240
>All 1881188    5992 1432466k    52864         pages 3413
>                                                                 Interrupts
>Proc:r  p  d  s  w    Csw  Trp  Sys  Int  Sof  Flt        cow    2252 total
>     1     2101      1605 2025  197  422    2 2018 251432 wire        irq1: atkb
>                                                   506156 act         irq6: fdc0
> 3.0%Sys   0.0%Intr 45.2%User  0.0%Nice 51.9%Idl  1038216 inact       irq15: ata
>|    |    |    |    |    |    |    |    |    |      89252 cache       irq17: fwo
>=>>>>>>>>>>>>>>>>>>>>>>>                             2964 free        irq20: nve
>                                                          daefr       irq21: ohc
>Namei         Name-cache    Dir-cache                     prcfr   241 irq22: ehc
>    Calls     hits    %     hits    %                 951 react    11 irq25: em0
>                                                          pdwak       irq29: amr
>                                      618 zfod            pdpgs  2000 cpu0: time
>Disks   ad4 amrd0                         ofod            intrn
>KB/t  56.79  0.00                         %slo-z   200816 buf
>tps     241     0                    5143 tfree         8 dirtybuf
>MB/s  13.38  0.00                                  100000 desiredvnodes
>% busy   47     0                                   34717 numvnodes
>                                                    24991 freevnodes

OK.  I _can_ see something like this when I try to compress a big file using
either your program or gzip.  In my case, both the disk % busy and system idle
vary widely but there's typicaly 50-60% disk utilisation and 30-40% CPU idle.
However, systat is reporting 23-25MB/sec (whereas dd peaks at ~30MB/sec) so the
time to gzip the datafile isn't that much different to the time to just read it.

My guess is that the read-ahead algorithms are working but aren't doing enough
re-ahead to cope with "read a bit, do some cpu-intensive processing and repeat"
at 25MB/sec so you're winding up with a degree of serialisation where the I/O
and compressing aren't overlapped.  I'm not sure how tunable the read-ahead is.

-- 
Peter Jeremy