Working on NUMA support

[Chad J. Milios]

I'm very interested and excited to watch the progress you are making. Many thank yous for grappling this task on FreeBSD!

Are you designing/implementing a fully hierarchy-aware approach (nested domains) or is each memory domain separate (not nested, simply "mine" or "not mine" in respect to a particular thread context)?

> On Jan 8, 2014, at 3:23 AM, Andrew Bates <andrewbates09 at gmail.com> wrote:
> 
> Hey Adrian,
> 
> We spent the last few months on research/design and plan on spending the
> next few months preparing tests, building the physical server(s) to test
> on, and finishing the prototype'd functions.
> 
> There is some white-board and pen/paper design that hasn't yet made it to
> the github repo, but the prototypes online are hopefully a solid base for
> what we intend to complete. In the meantime, we are excited to hear who
> else is interested in NUMA and if anyone has suggestions or concerns about
> our approach.
> 
> 
>> On Wed, Jan 8, 2014 at 12:03 AM, Adrian Chadd <adrian at freebsd.org> wrote:
>> 
>> Cool! Do you have any working code to implement the API, or is this
>> just in the design phase right now?
>> 
>> 
>> -a
>> 
>> 
>>> On 6 January 2014 12:11, Andrew Bates <andrewbates09 at gmail.com> wrote:
>>> Hey all,
>>> 
>>> My name is Andrew Bates, and I would like to take a bit of your time to
>>> talk about NUMA support.
>>> 
>>> Supporting Non-Uniform Memory Access in FreeBSD is something that has
>> been
>>> brought up in the past <
>> http://freebsd.1045724.n5.nabble.com/NUMA-Support-is-there-in-FreeBSD-td4865200.html
>>> .
>>> This is becoming increasingly important now that multiprocessor
>>> systems
>>> are an expanding technology, thus performance is scaling in terms of cpu
>>> count, rather than just clock rate.
>>> 
>>> There is a great opportunity here to optimize performance.  After being
>>> asked to look into this by the EMC Isilon Storage Division, myself and a
>>> few colleagues advised by Andrew Pilloud and Jeff Roberson would like to
>>> propose APIs to handle basic memory allocation/management to specific
>> NUMA
>>> domains.
>>> 
>>> What we have devised so far consists of two levels.  First there are the
>>> KPIs, to expose NUMA functionality at a thread level of domain affinity.
>>> Secondly, there would be a userspace/interface to take advantage of the
>>> proposed APIs, thus giving users the capability to make their
>> applications
>>> NUMA-aware.
>>> 
>>> We took the time to look into how many other systems (Linux, Macintosh,
>>> Solaris, Windows) already approach this problem, so there are some
>> aspects
>>> of our solution that are similar to how Linux and Solaris handle NUMA.
>>> Unlike Linux libnuma, we are only proposing a few additions and a minimal
>>> library that can easily be expanded later to suit users’ needs.
>>> 
>>> 
>>> KISS in mind, we came up with the following KPI prototypes
>> (freebsdnuma.h)
>>> to uncover NUMA in a usable fashion:
>>> 
>>> 
>>>   -
>>> 
>>>   cpuset_get_memory_affinity()
>>>   -
>>> 
>>>   cpuset_set_memory_affinity()
>>>   -
>>> 
>>>   move_pages()
>>>   -
>>> 
>>>   migrate_pages()
>>>   -
>>> 
>>>   get_numa_cpus()
>>>   -
>>> 
>>>   get_numa_weights()
>>> 
>>> 
>>> Then to the second part, we have the following userspace API prototypes
>>> (numanor.h) for our interface and testing purposes:
>>> 
>>> 
>>>   -
>>> 
>>>   is_numa_available()
>>>   -
>>> 
>>>   set_thread_on_domain()
>>>   -
>>> 
>>>   set_memory_policy()
>>>   -
>>> 
>>>   move_thread()
>>> 
>>> 
>>> In much much more detail, you can learn more about these prototypes, this
>>> project, view our progress, track along, and give input on our github
>> repo
>>> < https://github.com/andrewbates09/freebsd-numa > or simply via email.
>> This
>>> repo currently includes fully commented prototypes (like a mini man page)
>>> and will later include additions to the project.
>>> 
>>> If anyone has any comments, suggestions, concerns, quandaries, or just
>>> general thoughts please feel free to contact us, as we would love to hear
>>> your input!
>>> 
>>> The Leaders: Sakire Arslan Ay, Andrew Pilloud, Jeff Roberson
>>> The Team: Andrew Bates, Joshua Clark, Alex Schuldberg, Dustin Walker
>>> 
>>> --
>>> V/Respectfully,
>>> Andrew M Bates
>>> _______________________________________________
>>> freebsd-hackers at freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
>>> To unsubscribe, send any mail to "
>> freebsd-hackers-unsubscribe at freebsd.org"
> 
> 
> 
> -- 
> V/Respectfully,
> Andrew M Bates
> _______________________________________________
> freebsd-hackers at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org"