SMP support for XLR processors.
c.jayachandran at gmail.com
Tue Apr 20 10:49:43 UTC 2010
On Tue, Apr 20, 2010 at 4:03 PM, Rui Paulo <rpaulo at freebsd.org> wrote:
> On 20 Apr 2010, at 11:05, Rui Paulo wrote:
>> On 20 Apr 2010, at 10:52, C. Jayachandran wrote:
>>> On Mon, Apr 19, 2010 at 7:27 PM, C. Jayachandran
>>> <c.jayachandran at gmail.com> wrote:
>>>> I have a possible cause for the panic with invariants - we should not
>>>> schedule the msgring threads unless the smp is completely up. I guess
>>>> we start getting message ring interrupts on before the message ring
>>>> threads can be scheduled. I am trying out some changes for this -
>>>> will send you a patch if this fixes it.
>>> I've attached a patch that should fix the issue. The cause was the way
>>> message ring threads are started on individual cores and the way
>>> interrupts are enabled in the core. I've moved starting message ring
>>> threads on other cpus to be a SYSINIT after SMP is started. I'd
>>> thought originally that it was due to some clash with the changes in
>>> HEAD - but looks like I was completely off-track there.
>>> Please let me know if you don't get multi-user with 32 cpus with this
>>> patch. There is still the original hang in buildworld, but that should
>>> be a bug elsewhere
>>> I have a copy at http://sites.google.com/site/cjayachandran/files too
>> This works perfectly, thanks!
> On further inspection, I noticed that the load avg is now 7.
> last pid: 1613; load averages: 6.99, 6.97, 6.08 up 0+00:30:11 10:32:48
> 108 processes: 40 running, 24 sleeping, 44 waiting
> CPU: 0.0% user, 0.0% nice, 21.9% system, 0.0% interrupt, 78.1% idle
> Mem: 8444K Active, 6028K Inact, 37M Wired, 308K Cache, 6800K Buf, 3190M Free
> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
> 10 root 32 171 ki31 0G 0G CPU0 0 263:26 2500.00% idle
> 17 root 1 -16 - 0K 0G CPU12 2 0:00 100.00% msg_intr12
> 15 root 1 -16 - 0K 0G CPU4 2 0:00 100.00% msg_intr4
> 16 root 1 -16 - 0K 0G CPU8 2 0:00 100.00% msg_intr8
> 20 root 1 -16 - 0K 0G CPU24 1 0:00 100.00% msg_intr24
> 19 root 1 -16 - 0K 0G CPU20 1 0:00 100.00% msg_intr20
> 21 root 1 -16 - 0K 0G CPU28 1 0:00 100.00% msg_intr28
> 18 root 1 -16 - 0K 0G CPU16 1 0:00 100.00% msg_intr16
> What are these msg_intrXX kprocs doing?
They should really be sleeping unless there is a lot of network
traffic :) The msg_intr threads are interrupt handlers which we run
one per core, in the first thread of each core. They were modelled
after interrupt threads (in FreeBSD 6). This should be sleeping until
there is a message ring interrupt (which tells us that an IO has send
data to our core over the message ring).
Thanks for the report - I will look at the sleep logic.
More information about the freebsd-mips