HEADS UP: Superpages support for ARMv6/v7

[Zbyszek Bodek]

Hello Everyone.

I'm happy to announce that Superpages support for ARM has just been
integrated to the FreeBSD HEAD:
http://svnweb.freebsd.org/changeset/base/254918

This project was sponsored by The FreeBSD Foundation and Semihalf.
It was developed with great support of Alan Cox (alc) who was also the
technical reviewer of the code. Thank you very much Alan for all your help!
I would also like to thank Grzegorz Bernacki (gber) and Rafal Jaworowski
(raj) for mentoring and help with the code integration and all the
people involved in testing of the patches and review.

The code was tested on a quad-core, ARMv7, Marvell Armada XP SoC in SMP
environment.

Superpages is a feature that can increase TLB coverage and allow for
efficient use of page table entries. Current implementation for ARM
supports two page sizes: 4KB small pages (used as base pages) and 1MB
sections (used as superpages).
Superpages are created either directly by 1MB section insertion or as a
result of promotion of 256 4KB pages. In both cases superpages creation
and utilization depends on *sp_enabled* sysctl variable.

By default, superpages support is disabled.
In order to use this functionality one needs to set
*vm.pmap.sp_enabled* tunable to non-zero value. This can be done either
in loader.conf or by modifying *sp_enabled* variable in
sys/arm/arm/pmap-v6.c . Statistics regarding superpages usage are
available through: sysctl vm.pmap.section

All ARMv6/v7-based platforms can take advantage from superpages, so
please enable this feature on your ARM kernels.
We will appreciate all your feedback regarding performance impact and
general system behavior.

Performance improvement should be visible in all tasks where intensive
memory utilization is involved. GUPS (Giga Updates Per Second) benchmark
can be used to show the difference in memory utilization efficiency with
superpages enabled and disabled. GUPS src can be downloaded from here:
http://people.freebsd.org/~raj/patches/arm/superpages/GUPS.tar.gz

Exemplary GUPS results:
--------------------------------------------------------------------
*superpages enabled*
vm.pmap.section.promotions: 1024
vm.pmap.section.p_failures: 58
vm.pmap.section.mappings: 0
vm.pmap.section.demotions: 0

# ./gups
Main table size   = 2^27 = 134217728 words
Number of updates = 536870912
CPU time used  = 97.085938 seconds
Real time used = 97.082504 seconds
0.005530048 Billion(10^9) Updates    per second [GUP/s]

vm.pmap.section.promotions: 2048
vm.pmap.section.p_failures: 58
vm.pmap.section.mappings: 0
vm.pmap.section.demotions: 0

* superpages disabled *
Main table size   = 2^27 = 134217728 words
Number of updates = 536870912
CPU time used  = 145.679688 seconds
Real time used = 145.680798 seconds
0.003685255 Billion(10^9) Updates    per second [GUP/s]
--------------------------------------------------------------------

*Self host buildworld*
World build time on Armada XP has shortened from 6h 36min to
5h 14min with superpages enabled.

*Stress tests*
stress --cpu 4 --io 4 --vm 2 --vm-bytes 800M
Survived long time runs, large superpages creation ratio has been observed.

*Swapping*
No problems with swapping or system running under heavy load with
shortage of memory have been observed.


Please feel free to send your results.

Best regards
Zbigniew Bodek