pmap and mtx scalability problem

[K. Macy]

Known problem. There is an open disagreement about how to improve the
granularity of locking in pmap.

-Kip

On Tue, Apr 24, 2012 at 9:14 PM, Slawa Olhovchenkov <slw at zxy.spb.ru> wrote:
> I treid make -j 30 build{world,kernel} (latest -CURRENT) on 24-core machine and see poor
> scalability of pmap/mtx -- more then 50% cpu spend on system time.
>
> pmcstat:
>
> @ CPU_CLK_UNHALTED_CORE [194841 samples]
>
> 42.65%  [83102]    _mtx_lock_sleep @ /boot/kernel/kernel
>  40.97%  [34051]     pmap_enter
>  100.0%  [34051]      vm_fault_hold
>   100.0%  [34051]       trap_pfault
>  30.40%  [25262]     vm_page_activate
>  100.0%  [25262]      vm_fault_hold
>   100.0%  [25262]       trap_pfault
>  18.41%  [15300]     vm_pageq_remove
>  73.63%  [11266]      vm_page_free_toq
>   70.69%  [7964]        vm_object_terminate
>   29.31%  [3302]        vm_object_page_remove
>  26.37%  [4034]       vm_fault_hold
>   100.0%  [4034]        trap_pfault
>
> make -j 8:
>
> 15.44%  [10740]    _mtx_lock_sleep @ /boot/kernel/kernel
>  38.77%  [4164]      pmap_enter
>  99.93%  [4161]       vm_fault_hold
>   100.0%  [4161]        trap_pfault
>  00.07%  [3]          kmem_back
>   100.0%  [3]           kmem_malloc
>  27.98%  [3005]      vm_page_activate
>  100.0%  [3005]       vm_fault_hold
>   100.0%  [3005]        trap_pfault
>  20.64%  [2217]      vm_pageq_remove
>  64.73%  [1435]       vm_page_free_toq
>   63.41%  [910]         vm_object_terminate
>
> make -j 4
>
> 06.86%  [4222]     pagezero @ /boot/kernel/kernel
>  98.39%  [4154]      trap_pfault
>  100.0%  [4154]       trap
>  01.11%  [47]        vm_fault_hold
>
> _______________________________________________
> freebsd-performance at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "freebsd-performance-unsubscribe at freebsd.org"



-- 
   “The real damage is done by those millions who want to 'get by.'
The ordinary men who just want to be left in peace. Those who don’t
want their little lives disturbed by anything bigger than themselves.
Those with no sides and no causes. Those who won’t take measure of
their own strength, for fear of antagonizing their own weakness. Those
who don’t like to make waves—or enemies.

   Those for whom freedom, honour, truth, and principles are only
literature. Those who live small, love small, die small. It’s the
reductionist approach to life: if you keep it small, you’ll keep it
under control. If you don’t make any noise, the bogeyman won’t find
you.

   But it’s all an illusion, because they die too, those people who
roll up their spirits into tiny little balls so as to be safe. Safe?!
>From what? Life is always on the edge of death; narrow streets lead to
the same place as wide avenues, and a little candle burns itself out
just like a flaming torch does.

   I choose my own way to burn.”

   Sophie Scholl