Working on NUMA support

[Andrew Bates]

Hey all,

My name is Andrew Bates, and I would like to take a bit of your time to
talk about NUMA support.

Supporting Non-Uniform Memory Access in FreeBSD is something that has been
brought up in the past <
http://freebsd.1045724.n5.nabble.com/NUMA-Support-is-there-in-FreeBSD-td4865200.html>.
This is becoming increasingly important now that multiprocessor
systems
are an expanding technology, thus performance is scaling in terms of cpu
count, rather than just clock rate.

There is a great opportunity here to optimize performance.  After being
asked to look into this by the EMC Isilon Storage Division, myself and a
few colleagues advised by Andrew Pilloud and Jeff Roberson would like to
propose APIs to handle basic memory allocation/management to specific NUMA
domains.

What we have devised so far consists of two levels.  First there are the
KPIs, to expose NUMA functionality at a thread level of domain affinity.
 Secondly, there would be a userspace/interface to take advantage of the
proposed APIs, thus giving users the capability to make their applications
NUMA-aware.

We took the time to look into how many other systems (Linux, Macintosh,
Solaris, Windows) already approach this problem, so there are some aspects
of our solution that are similar to how Linux and Solaris handle NUMA.
Unlike Linux libnuma, we are only proposing a few additions and a minimal
library that can easily be expanded later to suit users’ needs.


KISS in mind, we came up with the following KPI prototypes (freebsdnuma.h)
to uncover NUMA in a usable fashion:


   -

   cpuset_get_memory_affinity()
   -

   cpuset_set_memory_affinity()
   -

   move_pages()
   -

   migrate_pages()
   -

   get_numa_cpus()
   -

   get_numa_weights()


Then to the second part, we have the following userspace API prototypes
(numanor.h) for our interface and testing purposes:


   -

   is_numa_available()
   -

   set_thread_on_domain()
   -

   set_memory_policy()
   -

   move_thread()


In much much more detail, you can learn more about these prototypes, this
project, view our progress, track along, and give input on our github repo
< https://github.com/andrewbates09/freebsd-numa > or simply via email. This
repo currently includes fully commented prototypes (like a mini man page)
and will later include additions to the project.

If anyone has any comments, suggestions, concerns, quandaries, or just
general thoughts please feel free to contact us, as we would love to hear
your input!

The Leaders: Sakire Arslan Ay, Andrew Pilloud, Jeff Roberson
The Team: Andrew Bates, Joshua Clark, Alex Schuldberg, Dustin Walker

-- 
V/Respectfully,
Andrew M Bates