Non-sequential AP starts [Was: Re: 8-core server]

[Darren Pilgrim]

Bernd Walter wrote:
> On Fri, Sep 28, 2007 at 07:56:49AM -0700, Darren Pilgrim wrote:
>> Jeremy Chadwick wrote:
>>> I think what you're not noticing is that the cores are being launched in
>>> non-sequential order.  1, 2, 3, 4, 7, 5, 6.  This isn't a problem.
>> Why they wouldn't be launched sequentially?
> 
> First SMP rule:
> Don't expect a specific execution order from things running parallel.
> 
> I don't know the code, but would assume that they are started
> sequentially and each core prints it's own line, so they get
> disordered.

This question bugged me long enough for me to go read some source to 
find the answer, but I ended up with more questions.  I guess that's 
what I get for reading boot code. :)  I know it's basically just a race 
condition, but where does the race actually occur?  In each core's 
execution of startup instructions leading up to the printf() in 
init_secondary() or in the assignment of the value returned by 
PCPU_GET(cpudid)?  Where is the data behind PCPU_GET() anyway?  I 
couldn't find it.  Is AP launch order random or are there CPU 
characteristics that result in reproducible non-sequential ordering?