SMP support for XLR processors.

[C. Jayachandran]

I did not have invariants enabled when I moved to HEAD, and I got to
multiuser login prompt. The panic is easily seen with INVARIANTS
enabled with all 32 CPUs. But it does not happen if I cut down the
cpus to 16 [ userapp_mask_cpus 0xffff on boot prompt]. Looks like
there is some race I'm hitting with more CPUs.   Will look at this...

JC.

On Sun, Apr 18, 2010 at 11:07 PM, Randall Stewart <rrs at lakerest.net> wrote:
> OK,
>
> it appears in my early morning fog as I started I missed a patch ;-)
> Anyway... I know better than to work before my first cup of coffee ...
>
> I am rebuilding and will try again with all the patches ;-)
>
> I suspect I will see your panic.. my guess is the lockup JC saw
> was a LOR and he ran with NO WITNESS/INVARIANT in his build.
>
> We should be able to fix this pretty easily ;-)
>
> Give me a bit of time...
>
> R
>
> On Apr 18, 2010, at 10:17 AM, Rui Paulo wrote:
>
>> On 18 Apr 2010, at 13:09, Rui Paulo wrote:
>>
>>> On 17 Apr 2010, at 23:40, C. Jayachandran wrote:
>>>
>>>> I've a set of initial patches to enable SMP for RMI processors. It
>>>> comes up in multi-user with 32 CPUs.  I could do buildworld before I
>>>> updated to HEAD - with head there is a hang during buildworld which
>>>> I'm looking at, but I think the initial work can be checked in.
>>>>
>>>> Neel, can you have a look at the first two patches - one is to enable
>>>> ULE scheduler and the second one is to move platform_init_ap to
>>>> slightly later in the initialization sequence.
>>>>
>>>> The patches are :
>>>> 1. mips-ule-support.patch
>>>> - Enable ULE scheduler for MIPS
>>>>
>>>> 2. mips-smp-move-platform.patch
>>>> - We need a hook to setup message ring and its interrupts, we use
>>>> platform_init_ap now, and move it be called later for XLR
>>>>
>>>> 3. rmi-prid.patch
>>>> Add RMI processor ID prints - right now it prints unknown processor.
>>>>
>>>> 4. rmi-pcib-fix.patch
>>>> XLR pci bridge should be off the IODI bus, currently it is off the
>>>> nexus bus which will cause issue when bus_setup_intr on nexus is
>>>> called
>>>>
>>>> 5. cleanup-reorg-cpuid-rge-kx.patch
>>>> This set has a many changes rolled up:
>>>> - clean up rge driver, remove unused code paths and commented code
>>>> (long way to go still)
>>>> - xlr_cpu_id(), xlr_core_id() and xlr_thr_id() updated and usage updated
>>>> - fix a bug - we cannot use MIPS_PHYS_TO_KSEG0 on physical addresses
>>>> here, I have made changes for using XKPHYS to do the same process.
>>>> - the KX bit is enabled and interrupts are disabled before the
>>>> physical memory is read, to avoid the KX bit setting from affecting
>>>> other code.
>>>> - move message ring code to on_chip.c from xlr_machdep.c, now all
>>>> message ring code is in on_chip.c
>>>>
>>>> 6. rmi-xlr-smp.patch
>>>> SMP Support for XLR
>>>> - mpwait.S added to get the other CPUs out of bootloader code at
>>>> startup.
>>>> - SMP platform related functions.
>>>> - disable shared TLB code for SMP - since threads have different
>>>> mappings.
>>>>
>>>> The patches are also available at
>>>> http://sites.google.com/site/cjayachandran/files
>>>>
>>>> Enabling all 32 threads will need a minor fix in the SMP code, see the
>>>> patch subr_smp-fix.patch at the same place.
>>>
>>> On the netperf XLR, this panics with SCHED_ULE activated:
>>>
>>> Copyright (c) 1992-2010 The FreeBSD Project.
>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>>>        The Regents of the University of California. All rights reserved.
>>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>>> FreeBSD 9.0-CURRENT #2 r206791M: Sun Apr 18 12:02:52 UTC 2010
>>>
>>>  rpaulo at freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/rpaulo/freebsd/head/sys/XLR
>>> mips
>>> real memory  = 3488608256 (3406844K bytes)
>>> avail memory = 3395207168 (3237MB)
>>> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
>>> nexus0: <MIPS32 root nexus>
>>> iodi0 on nexus0
>>> uart0: <16550 or compatible> on iodi0
>>> uart0: [FILTER]
>>> uart0: console (38551,n,8,1)
>>> pcib0: <XLR PCI bus> on iodi0
>>> pci0: <PCI bus> on pcib0
>>> pci0: <mass storage> at device 1.0 (no driver attached)
>>> pci0: <network, ethernet> at device 3.0 (no driver attached)
>>> rge0 on iodi0
>>> rge0: [ITHREAD]
>>> rge0: Ethernet address:
>>> rge0: [10Mbps]
>>> Total Active Core 8
>>> rge1 on iodi0
>>> rge1: [ITHREAD]
>>> rge1: Ethernet address:
>>> rge1: [1000Mbps]
>>> rge2 on iodi0
>>> rge2: [ITHREAD]
>>> rge2: Ethernet address:
>>> rge2: [10Mbps]
>>> rge3 on iodi0
>>> rge3: [ITHREAD]
>>> rge3: Ethernet address:
>>> rge3: [10Mbps]
>>> Enabling MDIO interrupts
>>> Timecounter "MIPS32" frequency 66000000 Hz quality 800
>>> Timecounters tick every 1.000 msec
>>> bootpc_init: wired to interface 'rge1'
>>> Sending DHCP Discover packet from interface rge1
>>> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root
>>> path)
>>> Sending DHCP Request packet from interface rge1
>>> Sending DHCP Discover packet from interface rge1
>>> DHCP/BOOTP timeout for server 255.255.255.255
>>> DHCP/BOOTP timeout for server 255.255.255.255
>>> DHCP/BOOTP timeout for server 255.255.255.255
>>> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root
>>> path)
>>> Sending DHCP Request packet from interface rge1
>>> DHCP/BOOTP timeout for server 255.255.255.255
>>> Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got root
>>> path)
>>> rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org boot
>>> file /zoo/rpaulo/lama1/boot/kernel/kernel
>>> subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs
>>> /zoo/rpaulo/lama1 rootopts nolockd
>>> Adjusted interface rge1
>>> panic: mutex sched lock 4 not owned at
>>> /usr/home/rpaulo/freebsd/head/sys/kern/sched_ule.c:2054
>>> cpuid = 4
>>> KDB: enter: panic
>>
>> If I use SCHED_4BSD this happens: (am I missing something?)
>>
>> Copyright (c) 1992-2010 The FreeBSD Project.
>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>>        The Regents of the University of California. All rights reserved.
>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>> FreeBSD 9.0-CURRENT #4 r206791M: Sun Apr 18 17:10:55 UTC 2010
>>
>> rpaulo at freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/rpaulo/freebsd/head/sys/XLR
>> mips
>> real memory  = 3488608256 (3406844K bytes)
>> avail memory = 3395252224 (3237MB)
>> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
>> nexus0: <MIPS32 root nexus>
>> iodi0 on nexus0
>> uart0: <16550 or compatible> on iodi0
>> uart0: [FILTER]
>> uart0: console (38551,n,8,1)
>> pcib0: <XLR PCI bus> on iodi0
>> pci0: <PCI bus> on pcib0
>> pci0: <mass storage> at device 1.0 (no driver attached)
>> pci0: <network, ethernet> at device 3.0 (no driver attached)
>> rge0 on iodi0
>> rge0: [ITHREAD]
>> rge0: Ethernet address: 00:0f:30:00:20:4e
>> rge0: [10Mbps]
>> Total Active Core 8
>> rge1 on iodi0
>> rge1: [ITHREAD]
>> rge1: Ethernet address: 00:0f:30:00:20:4f
>> rge1: [1000Mbps]
>> rge2 on iodi0
>> rge2: [ITHREAD]
>> rge2: Ethernet address: 00:0f:30:00:20:50
>> rge2: [10Mbps]
>> rge3 on iodi0
>> rge3: [ITHREAD]
>> rge3: Ethernet address: 00:0f:30:00:20:51
>> rge3: [10Mbps]
>> Enabling MDIO interrupts
>> Timecounter "MIPS32" frequency 66000000 Hz quality 800
>> Timecounters tick every 1.000 msec
>> bootpc_init: wired to interface 'rge1'
>> Sending DHCP Discover packet from interface rge1 (00:0f:30:00:20:4f)
>> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root
>> path)
>> Sending DHCP Request packet from interface rge1 (00:0f:30:00:20:4f)
>> Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got root
>> path)
>> rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org boot
>> file /zoo/rpaulo/lama1/boot/kernel/kernel
>> subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/rpaulo/lama1
>> rootopts nolockd
>> Adjusted interface rge1
>> spin lock 0x804b8ea4 (sched lock) held by 0xc98c0260 (tid 100027) too long
>> panic: spin lock held too long
>> cpuid = 12
>> KDB: enter: panic
>>
>>
>> Regards,
>> --
>> Rui Paulo
>>
>>
>
> ------------------------------
> Randall Stewart
> 803-317-4952 (cell)
> 803-345-0391(direct)
>
>



-- 
C. Jayachandran    c.jayachandran at gmail.com