Page Coloring Defines in vm_page.h

Terry Lambert tlambert2 at mindspring.com
Wed Jun 25 00:42:43 PDT 2003


Bruce M Simpson wrote:
> Yes, that was what I was really getting at; if you look at how PQ_L2_SIZE
> is computed from PQ_CACHESIZE, it implies 4-way set associative is
> the default optimization. This is fine for Intel chips but not for AMD ones.

FWIW, I'd like to see this automatically optimized, if possible;
when the code was originally written, you couldn't easily get the
cache size for either cache, except by having a human tell the code.


> The names of the definitions as they stand are perhaps slightly misleading
> in this respect. PQ_L2_SIZE might be better renamed PQ_SETSIZE and defined
> in terms of PQ_CACHESIZE/PQ_NSETS.

Per before: the L1 code was ripped out; the ripping out left all
sorts of little pieces of gore like this lying around.  Personally,
I'd like to see the L1 code put back in, now that L1 caches are
getting huge, and the L2 caches aren't so much larger than the L1,
on some processors.


> The page queue structures are sized according to the number of cache
> colors in use. These structures are allocated once for the lifetime
> of the kernel. So auto-tuning for an architecture's L2 cache
> organization at boot-time is not out of the question.

Yes, if you can get the information on the cache sizes from the
hardware... this may pessimize (even worse) older hardware, so
you may even want to consider turning it off, in those cases.


> Whilst it might be said that people who want to diddle with these defines
> will generally know what they're doing anyway, it would be nice to make these
> optimizations a wee bit more accessible; also, as we grow to support more
> Tier 1 architectures, having page coloring work 'out-of-the-box' across them
> might be seen as a desirable feature.
> 
> I'd like to know people's feelings on this.

I would really, really like to see this happen.  There's some
original research ahppening on FreeBSD, but not enough (IMO).


> > really want to color for the L2 cache, though; the L1 instruction
> > and data caches are usually small and hard to color for, and
> > occasional false sharing there doesn't cost much if you have an
> > L2.  There used to be L1 coloring support, but it was removed some
> > time ago.
> 
> I concur.

So does the paper I referenced earlier.  I'm not so sure that it's
not dated.  The cost of accessing L1 is down, but the realtive cost
of reloading it from L2 is getting up there; there's about a factor
of 7, between the fastest L2 and the fastest P4 it could be attached
to, for example.  Worst case, it'd be nice to diddle the code to get
benchmark numbers from the Novell "Standard SMP workload", just to
see what the effects are.  The second paper I referenced implied from
their numbers that the benefits for kernel would approach 10%, and
user space 15%, assuming you did everything right (which is what their
"cache preloading on context switch" simulated, in the limit).


> > Good question.  I know why they're prime numbers slightly smaller
> > than a power of two, but I can't figure out how that power of two
> > was chosen.
> 
> I'd need to work this out before being confident enough to even begin
> adding the above feature.

They are for perfect hashing to avoid cache line collisions.  The best
reference on getting the numbers for this is the Sun Micrososystems
paper from (I think) 1993 or so.  John Dyson's posts are also relevent
(sorry to repeat myself here, but at the time his posts really helped
me understand the Sun paper much better).

-- Terry


More information about the freebsd-hackers mailing list