Strategy for PCI resource management (for supporting hot-plug)

[John Baldwin]

On Tuesday 23 February 2010 2:16:40 am Rajat Jain wrote:
> 
> Hi,
> 
> I'm trying to add PCI-E hotplug support to the FreeBSD. As a first step
> for the PCI-E hotplug support, I'm trying to decide on a resource
> management / allocation strategy for the PCI memory / IO and the bus
> numbers. Can you please comment on the following approach that I am
> considering for resource allocation:
> 
> PROBLEM STATEMENT:
> ------------------
> Given a memory range [A->B], IO range [C->D], and limited (256) bus
> numbers, enumerate the PCI tree of a system, leaving enough "holes" in
> between to allow addition of future devices.
> 
> PROPOSED STRATEGY:
> ------------------
> 1) When booting, start enumerating in a depth-first-search order. While
> enumeration, always keep track of:
> 
>  * The next bus number (x) that can be allocated
> 
>  * The next Memory space pointer (A + y) starting which allocation can
> be 
>    done. ("y" is the memory already allocated).
> 
>  * The next IO Space pointer (C + z) starting which allocation can be
> done.
>    ("z" is the IO space already allocated).
> 
> Keep incrementing the above as the resources are allocated.
> 
> 2) Allocate bus numbers sequentially while traversing down from root to
> a leaf node (end point). When going down traversing a bridge:
> 
>  * Allocate the next available bus number (x) to the secondary bus of 
>    bridge.
> 
>  * Temporarily mark the subordinate bridge as 0xFF (to allow discovery
> of 
>    maximum buses).
> 
>  * Temporarily assign all the remaining available memory space to bridge
> 
>    [(A+x) -> B]. Ditto for IO space.
> 
> 3) When a leaf node (End point) is reached, allocate the memory / IO
> resource requested by the device, and increment the pointers. 
> 
> 4) While passing a bridge in the upward direction, tweak the bridge
> registers such that its resources are ONLY ENOUGH to address the needs
> of all the PCI tree below it, and if it has its own internal memory
> mapped registers, some memory for it as well.
> 
> The above is the standard depth-first algorithm for resource allocation.
> Here is the addition to support hot-plug:
> 
> At each bridge that supports hot-plug, in addition to the resources that
> would have normally been allocated to this bridge, additionally
> pre-allocate and assign to bridge (in anticipation of any new devices
> that may be added later):
> 
> a) "RSRVE_NUM_BUS" number of busses, to cater to any bridges, PCI trees 
>    present on the device plugged.
> 
> b) "RSRVE_MEM" amount of memory space, to cater to all the PCI devices
> that 
>    may be attached later on.
> 
> c) "RESRVE_IO" amount of IO space, to cater to all PCI devices that may
> be 
>    attached later on.
> 
> Please note that the above RSRVE* are constants defining the amount of
> resources to be set aside for /below each HOT-PLUGGABLE bridge; their
> values may be tweaked via a compile time option or via a sysctl. 
> 
> FEW COMMENTS
> ------------
>  
> 1) The strategy is fairly generic and tweak-able since it does not waste
> a lot of resources (The developer neds to pick up a smart bvalue for
> howmuch resources to reserve at each hot-pluggable slot):
> 
>    * The reservations shall be done only for hot-pluggable bridges
> 
>    * The developer can tweak the values (even disable it) for how much 
>      Resources shall be allocated for each hot-pluggable bridge.
>    
> 2) One point of debate is what happens if there are too much resource
> demands in the system (too many devices or the developer configures too
> many resources to be allocated for each hot-pluggable devices). For e.g.
> consider that while enumeration we find that all the resources are
> already allocated, while there are more devices that need resources. So
> do we simply do not enumerate them? Etc...
> 
> Overall, how does the above look?

I think one wrinkle is that we should try to preserve the resources that the 
firmware has set for devices, at least on x86.  I had also wanted to make use
of multipass for this, but that requires a bit more work to split the PCI
bus attach up into separate steps.

-- 
John Baldwin