Native preemption is the culprit [was Re: today's CURRENT lockups]

[Robert Watson]

On Fri, 9 Jul 2004, Marc G. Fournier wrote:

> On Fri, 9 Jul 2004, Steve Kargl wrote:
> 
> > I've tested it on a UP kernel (HTT enabled, ACPI disabled,
> > APIC enabled, ULE).   Appears to work fine.
> 
> 'k, I just upgraded my machine to latest, running ULE, P4, SMP kernel
> (but doesn't find or start up a second CPU) ... just locked solid here
> too ... unfortunately, was running X at the time, so don't know if it
> locked or panic'd to DDB :( Running non-X right now to see if it happens
> again ... 

I'm now experiencing extremely hard hangs in the following configurations:

  SMP kernel running SCHED_ULE with hyperthreads
  SMP kernel running SCHED_4BSD with hyperthreads

To generate the load, I'm using the "supersmack" benchark with the
select-key.smack query set with 30 client workers and 10,000 transactions. 
I am able to reliable hang the system with one or two runs. 

By disabling the "#define PREEMPTION" entry in param.h with SCHED_4BSD,
I'm able to complete the benchmark several times in a row without apparent
problems. However, I'll leave it running for a few more hours and see if I
didn't just "get lucky".  I'll then try SCHED_ULE w/o PREEMPTION.

By "extremely hard" I mean that I am unable to break into the debugger
using a serial break on the serial console.  I have not yet been able to
run the test on a system with easily accessible NMI but will attempt to do
so in the next few days. 

I'll give UP a spin with various combinations next.

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert at fledge.watson.org      Principal Research Scientist, McAfee Research