SMP support for XLR processors.

Randall Stewart rrs at lakerest.net
Sun Apr 18 17:47:46 UTC 2010


Rui:

What type of load are you doing.. NFS or MFS fs?

I am using a root NFS mounted and with all the patchs applied
the machine no longer recognizes the DHCP responses from my daemon.

I can clearly see these in wireshark... I will revert all patches
out and see what happens without the rge-kx cleanup..

R
On Apr 18, 2010, at 10:17 AM, Rui Paulo wrote:

> On 18 Apr 2010, at 13:09, Rui Paulo wrote:
>
>> On 17 Apr 2010, at 23:40, C. Jayachandran wrote:
>>
>>> I've a set of initial patches to enable SMP for RMI processors. It
>>> comes up in multi-user with 32 CPUs.  I could do buildworld before I
>>> updated to HEAD - with head there is a hang during buildworld which
>>> I'm looking at, but I think the initial work can be checked in.
>>>
>>> Neel, can you have a look at the first two patches - one is to  
>>> enable
>>> ULE scheduler and the second one is to move platform_init_ap to
>>> slightly later in the initialization sequence.
>>>
>>> The patches are :
>>> 1. mips-ule-support.patch
>>> - Enable ULE scheduler for MIPS
>>>
>>> 2. mips-smp-move-platform.patch
>>> - We need a hook to setup message ring and its interrupts, we use
>>> platform_init_ap now, and move it be called later for XLR
>>>
>>> 3. rmi-prid.patch
>>> Add RMI processor ID prints - right now it prints unknown processor.
>>>
>>> 4. rmi-pcib-fix.patch
>>> XLR pci bridge should be off the IODI bus, currently it is off the
>>> nexus bus which will cause issue when bus_setup_intr on nexus is
>>> called
>>>
>>> 5. cleanup-reorg-cpuid-rge-kx.patch
>>> This set has a many changes rolled up:
>>> - clean up rge driver, remove unused code paths and commented code
>>> (long way to go still)
>>> - xlr_cpu_id(), xlr_core_id() and xlr_thr_id() updated and usage  
>>> updated
>>> - fix a bug - we cannot use MIPS_PHYS_TO_KSEG0 on physical addresses
>>> here, I have made changes for using XKPHYS to do the same process.
>>> - the KX bit is enabled and interrupts are disabled before the
>>> physical memory is read, to avoid the KX bit setting from affecting
>>> other code.
>>> - move message ring code to on_chip.c from xlr_machdep.c, now all
>>> message ring code is in on_chip.c
>>>
>>> 6. rmi-xlr-smp.patch
>>> SMP Support for XLR
>>> - mpwait.S added to get the other CPUs out of bootloader code at  
>>> startup.
>>> - SMP platform related functions.
>>> - disable shared TLB code for SMP - since threads have different  
>>> mappings.
>>>
>>> The patches are also available at
>>> http://sites.google.com/site/cjayachandran/files
>>>
>>> Enabling all 32 threads will need a minor fix in the SMP code, see  
>>> the
>>> patch subr_smp-fix.patch at the same place.
>>
>> On the netperf XLR, this panics with SCHED_ULE activated:
>>
>> Copyright (c) 1992-2010 The FreeBSD Project.
>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993,  
>> 1994
>> 	The Regents of the University of California. All rights reserved.
>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>> FreeBSD 9.0-CURRENT #2 r206791M: Sun Apr 18 12:02:52 UTC 2010
>>   rpaulo at freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/ 
>> rpaulo/freebsd/head/sys/XLR mips
>> real memory  = 3488608256 (3406844K bytes)
>> avail memory = 3395207168 (3237MB)
>> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
>> nexus0: <MIPS32 root nexus>
>> iodi0 on nexus0
>> uart0: <16550 or compatible> on iodi0
>> uart0: [FILTER]
>> uart0: console (38551,n,8,1)
>> pcib0: <XLR PCI bus> on iodi0
>> pci0: <PCI bus> on pcib0
>> pci0: <mass storage> at device 1.0 (no driver attached)
>> pci0: <network, ethernet> at device 3.0 (no driver attached)
>> rge0 on iodi0
>> rge0: [ITHREAD]
>> rge0: Ethernet address:
>> rge0: [10Mbps]
>> Total Active Core 8
>> rge1 on iodi0
>> rge1: [ITHREAD]
>> rge1: Ethernet address:
>> rge1: [1000Mbps]
>> rge2 on iodi0
>> rge2: [ITHREAD]
>> rge2: Ethernet address:
>> rge2: [10Mbps]
>> rge3 on iodi0
>> rge3: [ITHREAD]
>> rge3: Ethernet address:
>> rge3: [10Mbps]
>> Enabling MDIO interrupts
>> Timecounter "MIPS32" frequency 66000000 Hz quality 800
>> Timecounters tick every 1.000 msec
>> bootpc_init: wired to interface 'rge1'
>> Sending DHCP Discover packet from interface rge1
>> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no  
>> root path)
>> Sending DHCP Request packet from interface rge1
>> Sending DHCP Discover packet from interface rge1
>> DHCP/BOOTP timeout for server 255.255.255.255
>> DHCP/BOOTP timeout for server 255.255.255.255
>> DHCP/BOOTP timeout for server 255.255.255.255
>> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no  
>> root path)
>> Sending DHCP Request packet from interface rge1
>> DHCP/BOOTP timeout for server 255.255.255.255
>> Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got  
>> root path)
>> rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org  
>> boot file /zoo/rpaulo/lama1/boot/kernel/kernel
>> subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/ 
>> rpaulo/lama1 rootopts nolockd
>> Adjusted interface rge1
>> panic: mutex sched lock 4 not owned at /usr/home/rpaulo/freebsd/ 
>> head/sys/kern/sched_ule.c:2054
>> cpuid = 4
>> KDB: enter: panic
>
> If I use SCHED_4BSD this happens: (am I missing something?)
>
> Copyright (c) 1992-2010 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993,  
> 1994
> 	The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 9.0-CURRENT #4 r206791M: Sun Apr 18 17:10:55 UTC 2010
>    rpaulo at freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/ 
> rpaulo/freebsd/head/sys/XLR mips
> real memory  = 3488608256 (3406844K bytes)
> avail memory = 3395252224 (3237MB)
> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
> nexus0: <MIPS32 root nexus>
> iodi0 on nexus0
> uart0: <16550 or compatible> on iodi0
> uart0: [FILTER]
> uart0: console (38551,n,8,1)
> pcib0: <XLR PCI bus> on iodi0
> pci0: <PCI bus> on pcib0
> pci0: <mass storage> at device 1.0 (no driver attached)
> pci0: <network, ethernet> at device 3.0 (no driver attached)
> rge0 on iodi0
> rge0: [ITHREAD]
> rge0: Ethernet address: 00:0f:30:00:20:4e
> rge0: [10Mbps]
> Total Active Core 8
> rge1 on iodi0
> rge1: [ITHREAD]
> rge1: Ethernet address: 00:0f:30:00:20:4f
> rge1: [1000Mbps]
> rge2 on iodi0
> rge2: [ITHREAD]
> rge2: Ethernet address: 00:0f:30:00:20:50
> rge2: [10Mbps]
> rge3 on iodi0
> rge3: [ITHREAD]
> rge3: Ethernet address: 00:0f:30:00:20:51
> rge3: [10Mbps]
> Enabling MDIO interrupts
> Timecounter "MIPS32" frequency 66000000 Hz quality 800
> Timecounters tick every 1.000 msec
> bootpc_init: wired to interface 'rge1'
> Sending DHCP Discover packet from interface rge1 (00:0f:30:00:20:4f)
> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no  
> root path)
> Sending DHCP Request packet from interface rge1 (00:0f:30:00:20:4f)
> Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got  
> root path)
> rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org  
> boot file /zoo/rpaulo/lama1/boot/kernel/kernel
> subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/rpaulo/ 
> lama1 rootopts nolockd
> Adjusted interface rge1
> spin lock 0x804b8ea4 (sched lock) held by 0xc98c0260 (tid 100027)  
> too long
> panic: spin lock held too long
> cpuid = 12
> KDB: enter: panic
>
>
> Regards,
> --
> Rui Paulo
>
>

------------------------------
Randall Stewart
803-317-4952 (cell)
803-345-0391(direct)



More information about the freebsd-mips mailing list