problems with mmap() and disk caching

[Andrey Zonov]

On 04.04.2012 11:17, Konstantin Belousov wrote:
>
> Calling madvise(MADV_RANDOM) fixes the issue, because the code to
> deactivate/cache the pages is turned off. On the other hand, it also
> turns of read-ahead for faulting, and the first loop becomes eternally
> long.

Now it takes 5 times longer.  Anyway, thanks for explanation.

>
> Doing MADV_WILLNEED does not fix the problem indeed, since willneed
> reactivates the pages of the object at the time of call. To use
> MADV_WILLNEED, you would need to call it between faults/memcpy.
>

I played with it, but no luck so far.

>>
>> I've also never seen super pages, how to make them work?
> They just work, at least for me. Look at the output of procstat -v
> after enough loops finished to not cause disk activity.
>

The problem was in my test program.  I fixed it, now I see super pages 
but I'm still not satisfied.  There are several tests below:

1. With madvise(MADV_RANDOM) I see almost all super pages:
$ ./mmap /mnt/random-1024 5
mmap:  1 pass took:  26.438535 (none:      0; res: 262144; super: 
511; other:      0)
mmap:  2 pass took:   0.187311 (none:      0; res: 262144; super: 
511; other:      0)
mmap:  3 pass took:   0.184953 (none:      0; res: 262144; super: 
511; other:      0)
mmap:  4 pass took:   0.186007 (none:      0; res: 262144; super: 
511; other:      0)
mmap:  5 pass took:   0.185790 (none:      0; res: 262144; super: 
511; other:      0)

Should it be 512?

2. Without madvise(MADV_RANDOM):
$ ./mmap /mnt/random-1024 50
mmap:  1 pass took:   7.629745 (none: 262112; res:     32; super: 
0; other:      0)
mmap:  2 pass took:   7.301720 (none: 261202; res:    942; super: 
0; other:      0)
mmap:  3 pass took:   7.261416 (none: 260226; res:   1918; super: 
1; other:      0)
[skip]
mmap: 49 pass took:   0.155368 (none:      0; res: 262144; super: 
323; other:      0)
mmap: 50 pass took:   0.155438 (none:      0; res: 262144; super: 
323; other:      0)

Only 323 pages.

3. If I just re-run test I don't see super pages with any size of "block".

$ ./mmap /mnt/random-1024 5 $((1<<30))
mmap:  1 pass took:   1.013939 (none:      0; res: 262144; super: 
0; other:      0)
mmap:  2 pass took:   0.267082 (none:      0; res: 262144; super: 
0; other:      0)
mmap:  3 pass took:   0.270711 (none:      0; res: 262144; super: 
0; other:      0)
mmap:  4 pass took:   0.268940 (none:      0; res: 262144; super: 
0; other:      0)
mmap:  5 pass took:   0.269634 (none:      0; res: 262144; super: 
0; other:      0)

4. If I activate madvise(MADV_WILLNEDD) in the copy loop and re-run test 
then I see super pages only if I use "block" greater than 2Mb.

$ ./mmap /mnt/random-1024 1 $((1<<21))
mmap:  1 pass took:   0.299722 (none:      0; res: 262144; super: 
0; other:      0)
$ ./mmap /mnt/random-1024 1 $((1<<22))
mmap:  1 pass took:   0.271828 (none:      0; res: 262144; super: 
170; other:      0)
$ ./mmap /mnt/random-1024 1 $((1<<23))
mmap:  1 pass took:   0.333188 (none:      0; res: 262144; super: 
258; other:      0)
$ ./mmap /mnt/random-1024 1 $((1<<24))
mmap:  1 pass took:   0.339250 (none:      0; res: 262144; super: 
303; other:      0)
$ ./mmap /mnt/random-1024 1 $((1<<25))
mmap:  1 pass took:   0.418812 (none:      0; res: 262144; super: 
324; other:      0)
$ ./mmap /mnt/random-1024 1 $((1<<26))
mmap:  1 pass took:   0.360892 (none:      0; res: 262144; super: 
335; other:      0)
$ ./mmap /mnt/random-1024 1 $((1<<27))
mmap:  1 pass took:   0.401122 (none:      0; res: 262144; super: 
342; other:      0)
$ ./mmap /mnt/random-1024 1 $((1<<28))
mmap:  1 pass took:   0.478764 (none:      0; res: 262144; super: 
345; other:      0)
$ ./mmap /mnt/random-1024 1 $((1<<29))
mmap:  1 pass took:   0.607266 (none:      0; res: 262144; super: 
346; other:      0)
$ ./mmap /mnt/random-1024 1 $((1<<30))
mmap:  1 pass took:   0.901269 (none:      0; res: 262144; super: 
347; other:      0)

5. If I activate madvise(MADV_WILLNEED) immediately after mmap() then I 
see some number of super pages (the number from test #2).

$ ./mmap /mnt/random-1024 5
mmap:  1 pass took:   0.178666 (none:      0; res: 262144; super: 
323; other:      0)
mmap:  2 pass took:   0.158889 (none:      0; res: 262144; super: 
323; other:      0)
mmap:  3 pass took:   0.157229 (none:      0; res: 262144; super: 
323; other:      0)
mmap:  4 pass took:   0.156895 (none:      0; res: 262144; super: 
323; other:      0)
mmap:  5 pass took:   0.162938 (none:      0; res: 262144; super: 
323; other:      0)

6. If I read file manually before test then I don't see super pages with 
any size of "block" and madvise(MADV_WILLNEED) doesn't help.

$ ./mmap /mnt/random-1024 5 $((1<<30))
mmap:  1 pass took:   0.996767 (none:      0; res: 262144; super: 
0; other:      0)
mmap:  2 pass took:   0.311129 (none:      0; res: 262144; super: 
0; other:      0)
mmap:  3 pass took:   0.317430 (none:      0; res: 262144; super: 
0; other:      0)
mmap:  4 pass took:   0.314437 (none:      0; res: 262144; super: 
0; other:      0)
mmap:  5 pass took:   0.310757 (none:      0; res: 262144; super: 
0; other:      0)


-- 
Andrey Zonov