[rfc] enumerating device / bus domain information

[Adrian Chadd]

On 8 October 2014 12:07, Warner Losh <imp at bsdimp.com> wrote:
>
> On Oct 7, 2014, at 7:37 PM, Adrian Chadd <adrian at FreeBSD.org> wrote:
>
>> Hi,
>>
>> Right now we're not enumerating any NUMA domain information about devices.
>>
>> The more recent intel NUMA stuff has some extra affinity information
>> for devices that (eventually) will allow us to bind kernel/user
>> threads and/or memory allocation to devices to keep access local.
>> There's a penalty for DMAing in/out of remote memory, so we'll want to
>> figure out what counts as "Local" for memory allocation and perhaps
>> constrain the CPU set that worker threads for a device run on.
>>
>> This patch adds a few things:
>>
>> * it adds a bus_if.m method for fetching the VM domain ID of a given
>> device; or ENOENT if it's not in a VM domain;
>
> Maybe a default VM domain. All devices are in VM domains :) By default
> today, we have only one VM domain, and that’s the model that most of the
> code expects…

Right, and that doesn't change until you compile in with num domains > 1.

Then, CPUs and memory have VM domains, but devices may or may not have
a VM domain. There's no "default" VM domain defined if num domains >
1.

The devices themselves don't know about VM domains right now, so
there's nothing constraining things like IRQ routing, CPU set, memory
allocation, etc. The isilon team is working on extending the cpuset
and allocators to "know" about numa and I'm sure this stuff will fall
out of whatever they're working on.

So when I go to add sysctl and other tree knowledge for device -> vm
domain mapping I'm going to make them return -1 for "no domain."

(Things will get pretty hilarious later on if we have devices that are
"local" to two or more VM domains ..)


-a