CPU time accounting broken on 8-STABLE machine after a few hours of uptime

[Don Lewis]

On 29 Sep, Andriy Gapon wrote:
> on 29/09/2010 11:56 Don Lewis said the following:
>> I'm using the same kernel config as the one on a slower !SMP box which
>> I'm trying to squeeze as much performance out of as possible.  My kernel
>> config file contains these statements:
>> 	nooptions       SMP
>> 	nodevice        apic
>> 
>> Testing with an SMP kernel is on my TODO list.
> 
> SMP or not, it's really weird to see apic disabled nowadays.

I tried enabling apic and got worse results.  I saw ping RTTs as high as
67 seconds.  Here's the timer info with apic enabled:

# sysctl kern.timecounter
kern.timecounter.tick: 1
kern.timecounter.choice: TSC(800) ACPI-fast(1000) i8254(0) dummy(-1000000)
kern.timecounter.hardware: ACPI-fast
kern.timecounter.stepwarnings: 0
kern.timecounter.tc.i8254.mask: 65535
kern.timecounter.tc.i8254.counter: 53633
kern.timecounter.tc.i8254.frequency: 1193182
kern.timecounter.tc.i8254.quality: 0
kern.timecounter.tc.ACPI-fast.mask: 16777215
kern.timecounter.tc.ACPI-fast.counter: 7988816
kern.timecounter.tc.ACPI-fast.frequency: 3579545
kern.timecounter.tc.ACPI-fast.quality: 1000
kern.timecounter.tc.TSC.mask: 4294967295
kern.timecounter.tc.TSC.counter: 1341917999
kern.timecounter.tc.TSC.frequency: 2500014018
kern.timecounter.tc.TSC.quality: 800
kern.timecounter.invariant_tsc: 0

Here's the verbose boot info with apic:
<http://people.freebsd.org/~truckman/AN-M2_HD-8.1-STABLE-apic-verbose.txt>


I've also experimented with SMP as well as SCHED_4BSD (all previous
testing was with !SMP and SCHED_ULE).  I still see occasional problems
with SCHED_4BSD and !SMP, but so far I have not seen any problems with
SCHED_ULE and SMP.

I did manage to catch the problem with lock profiling enabled:
<http://people.freebsd.org/~truckman/AN-M2_HD-8.1-STABLE_lock_profile_freeze.txt>


I'm currently testing SMP some more to verify if it really avoids this
problem.