SCHED_ULE + SMP Phenom freeze

[Michal Varga]

Guys, I'd like to point out a still present (rather serious I guess)
problem with SCHED_ULE, possibly triggered when Phenom X3 CPUs are
used.

First time I encountered this problem in early January, it took me
quite a while to narrow the freeze down to ULE with SMP support, well,
after eliminating every other possibility one by one. After that, it
wasn't that hard to find that I'm not the only one seeing this:

http://lists.freebsd.org/pipermail/freebsd-stable/2008-August/044167.html
http://lists.freebsd.org/pipermail/freebsd-stable/2008-October/045714.html
(continued)
http://lists.freebsd.org/pipermail/freebsd-amd64/2008-December/011722.html
(there were few more with the same symptoms back then, but sadly I
can't figure out the right keywords to google them at the moment)

Basically, the system can't get past the boot stage with ULE + SMP
support in kernel and hangs after "SMP: AP CPU #X Launched!" (see the
linked October thread for details).

But it gets better - there are absolutely no problems with either
uniprocessor SCHED_ULE, nor SCHED_4BSD (both UP and SMP). No interrupt
storms, no suspicious behaviour, no crashes, everything acts rock
solid. I've been running this system with SCHED_ULE minus SMP support
for more than a month under heavy load, tested SCHED_4BSD minus SMP
for a brief moment, and then another month with SCHED_4BSD + SMP.
Everything works perfectly and also all three cores behave correctly
with 4BSD + SMP. It only gets messy when ULE + SMP combo is used.

I was also wondering why only Phenom X3 people report this problem,
but this may be just a coincidence, who knows. Still, I'm also one of
them:

FreeBSD 7.1-STABLE #0: Sat Mar  7 21:37:27 CET 2009
    root at xenon:/usr/obj/usr/src/sys/KERNEL

Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Phenom(tm) 8450 Triple-Core Processor (2109.74-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x100f23  Stepping = 3
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x802009<SSE3,MON,CX16,POPCNT>
  AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x7ff<LAHF,CMP,SVM,ExtAPIC,CR8,<b5>,<b6>,<b7>,Prefetch,<b9>,<b10>>
  TSC: P-state invariant
  Cores per package: 3

Same as with other reports, with SCHED_ULE + SMP, as soon as I hit the
"SMP: AP CPU Launched" stage, the system is dead.

m.