TSC instead of ACPI: powerd doesn't work anymore (to be expected?)

[Robert Watson]

On Sun, 30 Oct 2005, David O'Brien wrote:

>> ACPI-fast requires an ioport read which takes about 1us (according to 
>> Google).  Do that 1000 times a second and you have each CPU spending 1% 
>> of its time doing nothing but reading the clock.  Yikes.
>
> But we've lived with using the ACPI timercounter (vs. TSC) for quite a 
> while now.  Why all of a sudden are the authors of this thread having an 
> issue with it now.  I know about the recent MySQL thread - but with the 
> TSC being untrustable on MP and power managed systems, why is there such 
> a desire to use the TSC?

Because in the past few weeks, several developers have noticed that if you 
change timercounters from ACPI-fast to TSC, you get significant 
performance boosts on real-world workloads.  Drew reported this a couple 
of weeks ago on net@ after looking at loopback network performance, and I 
suggested he frob the timecounter on his system, which resulted in a 
significant measured performance increase.  Then recently it was reported 
that MySQL also saw a significant boost from faster timekeeping, as it 
performs frequent time measurement.

So we have at least two interesting cases:

(1) We perform time measurement in the kernel context switch code for
     every context switch.  Using cheaper time measurement here can make a
     big different in performance for a broad range of workloads involving
     frequent context switches -- high interrupt rates, loopback or local
     IPC, etc.

(2) Many applications measure the time in order to self-profile, measure
     network timeouts, etc; it's quite possible that self-instrumentation
     with frequent time measurements is increasing due to the relatively
     low cost of time measurement on Linux, which has the effect of
     significantly reducing FreeBSD performance, where time measurement is
     much more expensive.  For example, it was reported that MySQL is
     measuring time with every IPC operation.

It has been suggested that the former can operate quite well with 
significantly reduced quality.  It has alos been suggested that most 
applications could operate fine with somewhat reduced quality, but that 
the API definitions don't say anything about how to specify application 
quality requirements vs performance requirements for time measurement.

Some performance results for loopback TCP on a UP kernel attached below. 
In the default case, a 12.8% performance improvement on loopback TCP is 
observed by switching from ACPI fast to TSC.  With local changes to 
disable some unwise preemptions, I'm able to improve performance by about 
29%, and adding on top of that the TSC change, 35.2%.  In the nopreempt 
case, I've reduced the number of context switches, so the effect of 
changing time counters is reduced, as there are fewer time measurements. 
In this scenario, the performance improvement from nopreempt to 
nopreempt.tsc is 4.7%.  These are non-trivial performance changes.

FYI: measured on SMP, moving to TSC for this benchmark gives a 4.9% 
improvement in the preempt case, and a 2.5% improvement in the nopreempt 
case.  On SMP, the preemption changes I'm working with reduce performance 
rather than improving it, possibly because the effects of preemption are 
still cheaper than the effects of migration.  Regardless, the timecounter 
change is significant, although less so than on UP.

Robert N M Watson

x preempt
+ preempt.tsc
* nopreempt
% nopreempt.tsc
+--------------------------------------------------------------------------+
|     xx                      +                              *          %% |
|     xx                      +                              **         %% |
|     xx                     ++                              **         %% |
|x    xx                    +++                       * **  ****   %%%  %%%|
|   |_A_|                    |A                          |__AM_|     |_AM_||
+--------------------------------------------------------------------------+
     N           Min           Max        Median           Avg        Stddev
x  12        2123.5       2194.31       2186.03     2181.4983     19.144156
+  12       2444.03       2468.44       2463.22     2460.1725     6.5242305
Difference at 95.0% confidence
         278.674 +/- 12.1092
         12.7744% +/- 0.555084%
         (Student's t, pooled s = 14.3015)
*  12       2750.12       2845.31       2829.98     2816.2325     31.188601
Difference at 95.0% confidence
         634.734 +/- 21.9101
         29.0962% +/- 1.00436%
         (Student's t, pooled s = 25.8769)
%  12       2902.27        2979.3       2954.93     2949.0792      25.48312
Difference at 95.0% confidence
         767.581 +/- 19.0828
         35.1859% +/- 0.874755%
         (Student's t, pooled s = 22.5376)