Reading via mmap stinks (Re: weird bugs with mmap-ing via NFS)

[Peter Jeremy]

On Fri, 2006-Mar-24 10:00:20 -0800, Matthew Dillon wrote:
>    Ok.  The next test is to NOT do umount/remount and then use a data set
>    that is ~2x system memory (but can still be mmap'd by grep).  Rerun
>    the data set multiple times using grep and grep --mmap.

The results here are weird.  With 1GB RAM and a 2GB dataset, the
timings seem to depend on the sequence of operations: reading is
significantly faster, but only when the data was mmap'd previously
There's one outlier that I can't easily explain.

hw.physmem: 932249600
hw.usermem: 815050752
+ ls -l /6_i386/var/tmp/test
-rw-r--r--  1 peter  wheel  2052167894 Mar 25 05:44 /6_i386/var/tmp/test
+ /usr/bin/time -l grep dfhfhdsfhjdsfl /6_i386/var/tmp/test
+ /usr/bin/time -l grep --mmap dfhfhdsfhjdsfl /6_i386/var/tmp/test

This was done in multi-user on a VTY using a script.  X was running
(and I forgot to kill an xclock) but there shouldn't have been anything
else happening.

grep --mmap followed by grep --mmap:
mm     77.94 real         1.65 user         2.08 sys
mm     78.22 real         1.53 user         2.21 sys
mm     78.34 real         1.55 user         2.21 sys
mm     79.33 real         1.48 user         2.37 sys

grep --mmap followed by grep/read
mr     56.64 real         0.77 user         2.45 sys
mr     56.73 real         0.67 user         2.53 sys
mr     56.86 real         0.68 user         2.60 sys
mr     57.64 real         0.64 user         2.63 sys
mr     57.71 real         0.62 user         2.68 sys
mr     58.04 real         0.63 user         2.59 sys
mr     58.83 real         0.78 user         2.50 sys
mr     59.15 real         0.74 user         2.50 sys

grep/read followed by grep --mmap
rm     75.98 real         1.56 user         2.19 sys
rm     76.06 real         1.50 user         2.29 sys
rm     76.50 real         1.40 user         2.38 sys
rm     77.35 real         1.47 user         2.30 sys
rm     77.49 real         1.39 user         2.44 sys
rm     79.14 real         1.56 user         2.19 sys
rm     88.88 real         1.57 user         2.27 sys

grep/read followed by grep/read
rr     78.00 real         0.69 user         2.74 sys
rr     78.34 real         0.67 user         2.74 sys
rr     79.64 real         0.69 user         2.71 sys
rr     79.69 real         0.73 user         2.75 sys

>    free and cache pages.  The system would only be allocating ~60MB/s
>    (or whatever your disk can do), so the pageout thread ought to be able
>    to keep up.

This is a laptop so the disk can only manage a bit over 25 MB/sec.

-- 
Peter Jeremy