Strategy for PCI resource management (for supporting hot-plug)

[Rajat Jain]

Hi,

I'm trying to add PCI-E hotplug support to the FreeBSD. As a first step
for the PCI-E hotplug support, I'm trying to decide on a resource
management / allocation strategy for the PCI memory / IO and the bus
numbers. Can you please comment on the following approach that I am
considering for resource allocation:

PROBLEM STATEMENT:
------------------
Given a memory range [A->B], IO range [C->D], and limited (256) bus
numbers, enumerate the PCI tree of a system, leaving enough "holes" in
between to allow addition of future devices.

PROPOSED STRATEGY:
------------------
1) When booting, start enumerating in a depth-first-search order. While
enumeration, always keep track of:

 * The next bus number (x) that can be allocated

 * The next Memory space pointer (A + y) starting which allocation can
be 
   done. ("y" is the memory already allocated).

 * The next IO Space pointer (C + z) starting which allocation can be
done.
   ("z" is the IO space already allocated).

Keep incrementing the above as the resources are allocated.

2) Allocate bus numbers sequentially while traversing down from root to
a leaf node (end point). When going down traversing a bridge:

 * Allocate the next available bus number (x) to the secondary bus of 
   bridge.

 * Temporarily mark the subordinate bridge as 0xFF (to allow discovery
of 
   maximum buses).

 * Temporarily assign all the remaining available memory space to bridge

   [(A+x) -> B]. Ditto for IO space.

3) When a leaf node (End point) is reached, allocate the memory / IO
resource requested by the device, and increment the pointers. 

4) While passing a bridge in the upward direction, tweak the bridge
registers such that its resources are ONLY ENOUGH to address the needs
of all the PCI tree below it, and if it has its own internal memory
mapped registers, some memory for it as well.

The above is the standard depth-first algorithm for resource allocation.
Here is the addition to support hot-plug:

At each bridge that supports hot-plug, in addition to the resources that
would have normally been allocated to this bridge, additionally
pre-allocate and assign to bridge (in anticipation of any new devices
that may be added later):

a) "RSRVE_NUM_BUS" number of busses, to cater to any bridges, PCI trees 
   present on the device plugged.

b) "RSRVE_MEM" amount of memory space, to cater to all the PCI devices
that 
   may be attached later on.

c) "RESRVE_IO" amount of IO space, to cater to all PCI devices that may
be 
   attached later on.

Please note that the above RSRVE* are constants defining the amount of
resources to be set aside for /below each HOT-PLUGGABLE bridge; their
values may be tweaked via a compile time option or via a sysctl. 

FEW COMMENTS
------------
 
1) The strategy is fairly generic and tweak-able since it does not waste
a lot of resources (The developer neds to pick up a smart bvalue for
howmuch resources to reserve at each hot-pluggable slot):

   * The reservations shall be done only for hot-pluggable bridges

   * The developer can tweak the values (even disable it) for how much 
     Resources shall be allocated for each hot-pluggable bridge.
   
2) One point of debate is what happens if there are too much resource
demands in the system (too many devices or the developer configures too
many resources to be allocated for each hot-pluggable devices). For e.g.
consider that while enumeration we find that all the resources are
already allocated, while there are more devices that need resources. So
do we simply do not enumerate them? Etc...

Overall, how does the above look?

Thanks & Best Regards,

Rajat Jain