Non-sequential AP starts [Was: Re: 8-core server]

[John Baldwin]

On Saturday 29 September 2007 12:25:15 am Darren Pilgrim wrote:
> Bernd Walter wrote:
> > On Fri, Sep 28, 2007 at 07:56:49AM -0700, Darren Pilgrim wrote:
> >> Jeremy Chadwick wrote:
> >>> I think what you're not noticing is that the cores are being launched in
> >>> non-sequential order.  1, 2, 3, 4, 7, 5, 6.  This isn't a problem.
> >> Why they wouldn't be launched sequentially?
> > 
> > First SMP rule:
> > Don't expect a specific execution order from things running parallel.
> > 
> > I don't know the code, but would assume that they are started
> > sequentially and each core prints it's own line, so they get
> > disordered.
> 
> This question bugged me long enough for me to go read some source to 
> find the answer, but I ended up with more questions.  I guess that's 
> what I get for reading boot code. :)  I know it's basically just a race 
> condition, but where does the race actually occur?  In each core's 
> execution of startup instructions leading up to the printf() in 
> init_secondary() or in the assignment of the value returned by 
> PCPU_GET(cpudid)?  Where is the data behind PCPU_GET() anyway?  I 
> couldn't find it.  Is AP launch order random or are there CPU 
> characteristics that result in reproducible non-sequential ordering?

The lauch is random.  The boot processor sets a flag to tell the APs (which 
are already running, but spinnin in a loop) "ok, go run".  The APs then 
compete on a spin lock to serialize their final initialization, and 
they "launch" in the order they acquire the spin lock, which is quite 
indeterminate.

-- 
John Baldwin