SMP support for XLR processors.

[Rui Paulo]

On 17 Apr 2010, at 23:40, C. Jayachandran wrote:

> I've a set of initial patches to enable SMP for RMI processors. It
> comes up in multi-user with 32 CPUs.  I could do buildworld before I
> updated to HEAD - with head there is a hang during buildworld which
> I'm looking at, but I think the initial work can be checked in.
> 
> Neel, can you have a look at the first two patches - one is to enable
> ULE scheduler and the second one is to move platform_init_ap to
> slightly later in the initialization sequence.
> 
> The patches are :
> 1. mips-ule-support.patch
> - Enable ULE scheduler for MIPS
> 
> 2. mips-smp-move-platform.patch
> - We need a hook to setup message ring and its interrupts, we use
> platform_init_ap now, and move it be called later for XLR
> 
> 3. rmi-prid.patch
> Add RMI processor ID prints - right now it prints unknown processor.
> 
> 4. rmi-pcib-fix.patch
> XLR pci bridge should be off the IODI bus, currently it is off the
> nexus bus which will cause issue when bus_setup_intr on nexus is
> called
> 
> 5. cleanup-reorg-cpuid-rge-kx.patch
> This set has a many changes rolled up:
>  - clean up rge driver, remove unused code paths and commented code
> (long way to go still)
>  - xlr_cpu_id(), xlr_core_id() and xlr_thr_id() updated and usage updated
>  - fix a bug - we cannot use MIPS_PHYS_TO_KSEG0 on physical addresses
> here, I have made changes for using XKPHYS to do the same process.
>  - the KX bit is enabled and interrupts are disabled before the
> physical memory is read, to avoid the KX bit setting from affecting
> other code.
> - move message ring code to on_chip.c from xlr_machdep.c, now all
> message ring code is in on_chip.c
> 
> 6. rmi-xlr-smp.patch
> SMP Support for XLR
> - mpwait.S added to get the other CPUs out of bootloader code at startup.
> - SMP platform related functions.
> - disable shared TLB code for SMP - since threads have different mappings.
> 
> The patches are also available at
> http://sites.google.com/site/cjayachandran/files
> 
> Enabling all 32 threads will need a minor fix in the SMP code, see the
> patch subr_smp-fix.patch at the same place.

On the netperf XLR, this panics with SCHED_ULE activated:

Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-CURRENT #2 r206791M: Sun Apr 18 12:02:52 UTC 2010
    rpaulo at freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/rpaulo/freebsd/head/sys/XLR mips
real memory  = 3488608256 (3406844K bytes)
avail memory = 3395207168 (3237MB)
FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
nexus0: <MIPS32 root nexus>
iodi0 on nexus0
uart0: <16550 or compatible> on iodi0
uart0: [FILTER]
uart0: console (38551,n,8,1)
pcib0: <XLR PCI bus> on iodi0
pci0: <PCI bus> on pcib0
pci0: <mass storage> at device 1.0 (no driver attached)
pci0: <network, ethernet> at device 3.0 (no driver attached)
rge0 on iodi0
rge0: [ITHREAD]
rge0: Ethernet address: 
rge0: [10Mbps]
Total Active Core 8
rge1 on iodi0
rge1: [ITHREAD]
rge1: Ethernet address:
rge1: [1000Mbps]
rge2 on iodi0
rge2: [ITHREAD]
rge2: Ethernet address: 
rge2: [10Mbps]
rge3 on iodi0
rge3: [ITHREAD]
rge3: Ethernet address: 
rge3: [10Mbps]
Enabling MDIO interrupts
Timecounter "MIPS32" frequency 66000000 Hz quality 800
Timecounters tick every 1.000 msec
bootpc_init: wired to interface 'rge1'
Sending DHCP Discover packet from interface rge1
Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root path)
Sending DHCP Request packet from interface rge1
Sending DHCP Discover packet from interface rge1
DHCP/BOOTP timeout for server 255.255.255.255
DHCP/BOOTP timeout for server 255.255.255.255
DHCP/BOOTP timeout for server 255.255.255.255
Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root path)
Sending DHCP Request packet from interface rge1
DHCP/BOOTP timeout for server 255.255.255.255
Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got root path)
rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org boot file /zoo/rpaulo/lama1/boot/kernel/kernel
subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/rpaulo/lama1 rootopts nolockd 
Adjusted interface rge1
panic: mutex sched lock 4 not owned at /usr/home/rpaulo/freebsd/head/sys/kern/sched_ule.c:2054
cpuid = 4
KDB: enter: panic


Regards,
--
Rui Paulo