svn commit: r208589 - head/sys/mips/mips

[Jayachandran C.]

On Wed, Jun 9, 2010 at 11:20 AM, Jayachandran C.
<c.jayachandran at gmail.com> wrote:
> On Wed, Jun 9, 2010 at 3:01 AM, Jayachandran C.
> <c.jayachandran at gmail.com> wrote:
>> On Tue, Jun 8, 2010 at 12:03 PM, Alan Cox <alc at cs.rice.edu> wrote:
>>>
>>> VM_FREEPOOL_DIRECT is used by at least amd64 and ia64 for page table pages
>>> and small kernel memory allocations.  Unlike mips, these machines don't have
>>> MMU support for a direct map.  Their direct maps are just a range of
>>> mappings in the regular (kernel) page table.  So, unlike mips, accesses
>>> through their direct map may still miss in the TLB and require a page table
>>> walk.  VM_FREEPOOL_* is a way to increase the physical locality (or
>>> clustering) of page allocations, so that, for example, page table page
>>> accesses by the pmap on amd64 are less likely to miss in the TLB.  However,
>>> it doesn't place a hard restriction on the range of physical addresses that
>>> will be used, which you need for mips.
>>>
>>> The impact of this clustering can be significant.  For example, on amd64 we
>>> use 2MB page mappings to implement the direct map.  However, old Opterons
>>> only had 8 data TLB entries for 2MB page mappings.  For a uniprocessor
>>> kernel running on such an Opteron, I measured an 18% reduction in system
>>> time during a buildworld with the introduction of VM_FREEPOOL_DIRECT.  (See
>>> the commit logs for vm/vm_phys.c and the comment that precedes the
>>> VM_NFREEORDER definition on amd64.)
>>>
>>> Until such time as superpage support is ported to mips from the amd64/i386
>>> pmaps, I don't think there is a point in having more than one VM_FREEPOOL_*
>>> on mips.  And then, the point would be to reduce fragmentation of the
>>> physical memory that could be caused by small allocations, such as page
>>> table pages.
>>
>> Thanks for the detailed explanation.
>>
>> Also, after looking at the code again,  I think vm_phys_alloc_contig()
>> can optimized not to look into segments which lie outside the area of
>> interest. The patch is:
> [...]
>> This change, along with the vmparam.h changes for HIGHMEM, I think we
>> should be able to use  vm_phys_alloc_contig() for page table pages (or
>> have I again missed something fundamental?).
>
> That patch was obviously wrong - many segments can map to a freelist
> as the comment right above my change noted.
>
> But the idea of eliminating freelists for which all the segments are
> outside (low,high) may still be useful, will look at this some more.

I have attached a patch (also at
http://people.freebsd.org/~jchandra/pmap-with-HIGHMEM-freelist.patch)
which reverts most of the changes I did to convert the page table page
allocation to use UMA zone, and replaces it with an implementation
using vm_phys_alloc_contig() and vm_contig_grow_cache(). This creates
a new HIGHMEM freelist for mips for memory outside the KSEG0 area, and
makes a few changes in vm_phys_alloc_contig() to skip freelists for
which all the segments fall outside the address range requested.

With this the buildworld perf on MIPS is similar to what I got with
the older code with zones.

If this approach is okay, I will do another round of
testing(buildworld passes, but I haven't really tested the case where
grow_cache is called).  If the changes are not okay, I will add
another page allocator which takes freelist as argument as you had
suggested earlier, instead of the vm_phys_alloc_contig() changes.

Thanks,
JC.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pmap-with-HIGHMEM-freelist.patch
Type: application/octet-stream
Size: 9034 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-mips/attachments/20100609/a4266397/pmap-with-HIGHMEM-freelist.obj