NUMA?

[Julian Elischer]

Ivan Voras wrote:
> Hi,

I did the AMD course a few weeks ago so I'm also very interested in this..

> 
> As even Intel's new CPUs have integrated memory controllers and thus 
> become NUMA, I'm interested in what is, in theory (I'm not proposing to 
> do it, I'm just curious), necessary to change in an OS to support NUMA. 
> My guess is:
> 
> 1) node topology detection - something similar to what ULE does but also 
> recording which memory ranges are "close" to which CPU and the 
> "distance" between nodes/CPUs

at a minimum, this is needed before anything else can really work.

> 2) on new image load (exec), pick a node for it, among "least used" 
> nodes and record the choice per-proc; on fork, keep the new process on 
> the same node

In some cases it may be worth having multiple copies of teh read-only 
text segments.
For example, it may eventually be worth having a /bin/sh text segment
in each CPU's memory space.


> 3) schedule threads on a CPU from the proc's node if at all possible 
> (e.g, when a 6-core CPU is still 1 node), then on a "near" node from a 
> list of distances sorted in order of cost

this is where it really starts getting hairy.. when do you migrate a 
process? and what if there are as many threads runnable as processors?

> 4) allocate new pages for a proc from its node's memory range(s) if at 
> all possible.
> 
> Is this all?

There are other interesting effects too..

assigning network interrupts to processors that have good access to 
the hardware AND the destination if you can..


> 
> On the other hand, did someone do a study of performance increase for 
> todays "consumer" NUMA systems (e.g. 2-4 sockets/nodes x86/x64 systems) 
> - is it worth it?

caches hide a multitude of sins..

> 
> _______________________________________________
> freebsd-smp at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-smp
> To unsubscribe, send any mail to "freebsd-smp-unsubscribe at freebsd.org"