Performance of SheevaPlug on 8-stable

[Ian Lepore]

On Tue, 2012-02-07 at 17:10 -0600, Mark Tinguely wrote:
> On Sun, Feb 5, 2012 at 11:48 PM, Ian Lepore
> <deletions>
> > Okay, I've just confirmed that the problem as I described it above still
> > exists in 9.0 ...
> >
> >        # uname -a
> >        FreeBSD  9.0-RELEASE FreeBSD 9.0-RELEASE #7: Mon Feb  6 04:38:59
> >        UTC 2012
> >        root at revolution.hippie.lan:/usr/obj/arm.arm/usr/src/sys/TFLEX
> >        arm
> >
> >        # /usr/tsc/bin/testsimple
> >        Elapsed 0.782453
> >        # /usr/tsc/bin/testsimple
> >        Elapsed 0.782539
> >
> >        # cat /usr/tsc/bin/testsimple >/dev/null
> >
> >        # /usr/tsc/bin/testsimple
> >        Elapsed 10.090336
> >        # /usr/tsc/bin/testsimple
> >        Elapsed 10.090611
> >
> > The testsimple program is just a little loop that repeatedly assigns a
> > volatile variable (had to fool the optimizer into generating some code).
> > I statically linked it to avoid any variability based on the race
> > between paging and readahead when loading shared libs.  I'm just showing
> > here that the base problem still exists:  when executable pages get into
> > the vfs buffer cache they (semi-)permanently lose their i-cache bit.
> >
> > I also applied my patches from the start of this thread and re-tested
> > and they appear to still be a usable workaround for the problem in 9.
> > But they are just a workaround, we need to figure out a real fix for
> > this.  (I had to enable the #if in ffs_read() as well as the one in
> > ffs_write() for this test, since my quick 'cat' test is reading the file
> > to get it into the cache.)
> >
> > -- Ian
> 
> 
> I dug out my files from that era.
> 
> I was looking at the cases where the page is marked executable and writable:
> 
>  1) were they shared or single mappings?
>  2) are all the shared mappings executable?
>  3) was the page mapped several time in one process?
>  4) was the all the writers a kernel process?
>  5) was there many kernel mappings
> 
> I was think something like (pardon the bad formatting)
> 
>  if (exec && (exec == entries + kentries) &&
> #ifndef test1
>    (writable == 0 || (writable == kwritable)) &&
> #else
>    (writable == 0 || (writable == 1 && kwritable == 1)) &&
> #endif
>    (ncaches == 0)) {
>       return;  /* bypass the cache fix routine */
> }
> 
>  this test is just before the comment that says
> 
>                 /*
>                  * check if the user duplicate mapping has
>                  * been removed.
>                  */
> 
> I never felt brave enough with this test though.
> 
> --Mark.

There are always one or more mappings which are read+exec which are the
actual executable mappings established by rtld-elf/map_object.c for a
shared lib or by code in kern/kern_exec.c for an app, plus one kernel
mapping that is read+exec+write which was established by the code path
which begins with ffs_read() or ffs_write() and ends with a copy of the
disk blocks in the buffer cache with write access still enabled in the
mapping after the IO is done.

The patch to map_object.c in my original posting fixes most of the
problem, but more as a workaround than a proper fix.  It avoids calling
ffs_read() as part of loading a shared lib, so the pages never end up in
the buffer cache and a writable mapping of the pages doesn't linger.
But there are other legitimate things you can do with an app or shared
lib that invoke the ffs_read()/ffs_write() code, such as copy it or
read/write it for other reasons, and that results in the lingering
writable mapping.  

I believe the arm pmap code is correct; it is managing the cachability
of the pages properly according to the permissions that exist in the
mappings.  The problem is leaving write permissions in place for the
buffer cache mapping of the page after the IO is done and there's no
longer a need to write to the page.  Trying to infer that situation from
the pmap code and make the page cacheable because it appears that the
only writable mapping might be the buffer cache mapping feels like a
very dangerous thing to do.  

The right fix, I believe, is to not leave writable mappings for pages in
the buffer cache when there is no IO in progress.  If changing the
permissions after the IO is done is an expensive operation, then we may
need some sort of compile-time flag that indicates a VIVT architecture
so we can avoid the extra expense on platforms that don't need it.

I'd like to prove my point in that last paragraph with a patch that I
could claim works, but I can't figure out without help from a VM guru
where is the right place to drop the write permission on the pages.

-- Ian