SMP support for XLR processors.

Sun Apr 18 17:17:47 UTC 2010

On 18 Apr 2010, at 13:09, Rui Paulo wrote:

> On 17 Apr 2010, at 23:40, C. Jayachandran wrote:
> 
>> I've a set of initial patches to enable SMP for RMI processors. It
>> comes up in multi-user with 32 CPUs.  I could do buildworld before I
>> updated to HEAD - with head there is a hang during buildworld which
>> I'm looking at, but I think the initial work can be checked in.
>> 
>> Neel, can you have a look at the first two patches - one is to enable
>> ULE scheduler and the second one is to move platform_init_ap to
>> slightly later in the initialization sequence.
>> 
>> The patches are :
>> 1. mips-ule-support.patch
>> - Enable ULE scheduler for MIPS
>> 
>> 2. mips-smp-move-platform.patch
>> - We need a hook to setup message ring and its interrupts, we use
>> platform_init_ap now, and move it be called later for XLR
>> 
>> 3. rmi-prid.patch
>> Add RMI processor ID prints - right now it prints unknown processor.
>> 
>> 4. rmi-pcib-fix.patch
>> XLR pci bridge should be off the IODI bus, currently it is off the
>> nexus bus which will cause issue when bus_setup_intr on nexus is
>> called
>> 
>> 5. cleanup-reorg-cpuid-rge-kx.patch
>> This set has a many changes rolled up:
>> - clean up rge driver, remove unused code paths and commented code
>> (long way to go still)
>> - xlr_cpu_id(), xlr_core_id() and xlr_thr_id() updated and usage updated
>> - fix a bug - we cannot use MIPS_PHYS_TO_KSEG0 on physical addresses
>> here, I have made changes for using XKPHYS to do the same process.
>> - the KX bit is enabled and interrupts are disabled before the
>> physical memory is read, to avoid the KX bit setting from affecting
>> other code.
>> - move message ring code to on_chip.c from xlr_machdep.c, now all
>> message ring code is in on_chip.c
>> 
>> 6. rmi-xlr-smp.patch
>> SMP Support for XLR
>> - mpwait.S added to get the other CPUs out of bootloader code at startup.
>> - SMP platform related functions.
>> - disable shared TLB code for SMP - since threads have different mappings.
>> 
>> The patches are also available at
>> http://sites.google.com/site/cjayachandran/files
>> 
>> Enabling all 32 threads will need a minor fix in the SMP code, see the
>> patch subr_smp-fix.patch at the same place.
> 
> On the netperf XLR, this panics with SCHED_ULE activated:
> 
> Copyright (c) 1992-2010 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> 	The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 9.0-CURRENT #2 r206791M: Sun Apr 18 12:02:52 UTC 2010
>    rpaulo at freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/rpaulo/freebsd/head/sys/XLR mips
> real memory  = 3488608256 (3406844K bytes)
> avail memory = 3395207168 (3237MB)
> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
> nexus0: <MIPS32 root nexus>
> iodi0 on nexus0
> uart0: <16550 or compatible> on iodi0
> uart0: [FILTER]
> uart0: console (38551,n,8,1)
> pcib0: <XLR PCI bus> on iodi0
> pci0: <PCI bus> on pcib0
> pci0: <mass storage> at device 1.0 (no driver attached)
> pci0: <network, ethernet> at device 3.0 (no driver attached)
> rge0 on iodi0
> rge0: [ITHREAD]
> rge0: Ethernet address: 
> rge0: [10Mbps]
> Total Active Core 8
> rge1 on iodi0
> rge1: [ITHREAD]
> rge1: Ethernet address:
> rge1: [1000Mbps]
> rge2 on iodi0
> rge2: [ITHREAD]
> rge2: Ethernet address: 
> rge2: [10Mbps]
> rge3 on iodi0
> rge3: [ITHREAD]
> rge3: Ethernet address: 
> rge3: [10Mbps]
> Enabling MDIO interrupts
> Timecounter "MIPS32" frequency 66000000 Hz quality 800
> Timecounters tick every 1.000 msec
> bootpc_init: wired to interface 'rge1'
> Sending DHCP Discover packet from interface rge1
> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root path)
> Sending DHCP Request packet from interface rge1
> Sending DHCP Discover packet from interface rge1
> DHCP/BOOTP timeout for server 255.255.255.255
> DHCP/BOOTP timeout for server 255.255.255.255
> DHCP/BOOTP timeout for server 255.255.255.255
> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root path)
> Sending DHCP Request packet from interface rge1
> DHCP/BOOTP timeout for server 255.255.255.255
> Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got root path)
> rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org boot file /zoo/rpaulo/lama1/boot/kernel/kernel
> subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/rpaulo/lama1 rootopts nolockd 
> Adjusted interface rge1
> panic: mutex sched lock 4 not owned at /usr/home/rpaulo/freebsd/head/sys/kern/sched_ule.c:2054
> cpuid = 4
> KDB: enter: panic

If I use SCHED_4BSD this happens: (am I missing something?)

Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-CURRENT #4 r206791M: Sun Apr 18 17:10:55 UTC 2010
    rpaulo at freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/rpaulo/freebsd/head/sys/XLR mips
real memory  = 3488608256 (3406844K bytes)
avail memory = 3395252224 (3237MB)
FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
nexus0: <MIPS32 root nexus>
iodi0 on nexus0
uart0: <16550 or compatible> on iodi0
uart0: [FILTER]
uart0: console (38551,n,8,1)
pcib0: <XLR PCI bus> on iodi0
pci0: <PCI bus> on pcib0
pci0: <mass storage> at device 1.0 (no driver attached)
pci0: <network, ethernet> at device 3.0 (no driver attached)
rge0 on iodi0
rge0: [ITHREAD]
rge0: Ethernet address: 00:0f:30:00:20:4e
rge0: [10Mbps]
Total Active Core 8
rge1 on iodi0
rge1: [ITHREAD]
rge1: Ethernet address: 00:0f:30:00:20:4f
rge1: [1000Mbps]
rge2 on iodi0
rge2: [ITHREAD]
rge2: Ethernet address: 00:0f:30:00:20:50
rge2: [10Mbps]
rge3 on iodi0
rge3: [ITHREAD]
rge3: Ethernet address: 00:0f:30:00:20:51
rge3: [10Mbps]
Enabling MDIO interrupts
Timecounter "MIPS32" frequency 66000000 Hz quality 800
Timecounters tick every 1.000 msec
bootpc_init: wired to interface 'rge1'
Sending DHCP Discover packet from interface rge1 (00:0f:30:00:20:4f)
Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root path)
Sending DHCP Request packet from interface rge1 (00:0f:30:00:20:4f)
Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got root path)
rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org boot file /zoo/rpaulo/lama1/boot/kernel/kernel
subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/rpaulo/lama1 rootopts nolockd 
Adjusted interface rge1
spin lock 0x804b8ea4 (sched lock) held by 0xc98c0260 (tid 100027) too long
panic: spin lock held too long
cpuid = 12
KDB: enter: panic


Regards,
--
Rui Paulo