One-shot-oriented event timers management

[Brandon Gooch]

On Tue, Aug 31, 2010 at 3:48 AM, Alexander Motin <mav at freebsd.org> wrote:
> Gary Jennejohn wrote:
>> On Mon, 30 Aug 2010 12:11:48 +0200
>> Gary Jennejohn <gljennjohn at googlemail.com> wrote:
>>
>>> On Mon, 30 Aug 2010 13:07:38 +0300
>>> Alexander Motin <mav at FreeBSD.org> wrote:
>>>
>>>> Gary Jennejohn wrote:
>>>>> Hmm.  I applied your patches and am now running the new kernel.  But I
>>>>> only installed the new kernel and didn't do make buildworld installworld.
>>>>>
>>>>> Mu systat -vm 1 doesn't look anything like yours.  I'm seeing about 2300
>>>>> interrupts per second and most of those are coming from the hpet timers:
>>>>>
>>>>> 1122 hpet0:t0
>>>>> 1124 hpet0:t1
>>>> It means 1000Hz of hardclock (hz) events mixed with 127Hz of statclock
>>>> (stathz) events. HPET timer here works in one-shot mode handling it.
>>>>
>>>>> So, what else did you do to reduce interrupts so much?
>>>>>
>>>>> Ah, I think I see it now.  My desktop has only C1 enabled.  Is that it?
>>>>> Unfortunately, it appears that only C1 is supported :(
>>>> Yes, as I have said, at this moment empty ticks skipped only while CPU
>>>> is in C2/C3 states. In C1 state there is no way to handle lost events on
>>>> wake up. While it may be not very dangerous, it is not very good.
>>>>
>>> Too bad.  I'd say that systems which are limited to C1 don't benefit
>>> much (or not at all) from your changes.
>>>
>>
>> OK, this is purely anecdotal, but I'll report it anyway.
>>
>> I was running pretty much all day with the patched kernel and things
>> seemed to be working quite well.
>>
>> Then, after about 7 hours, everything just stopped.
>>
>> I had gkrellm running and noticed that it updated only when I moved the
>> mouse.
>>
>> This behavior leads me to suspect that the timer interrupts had stopped
>> working and the mouse interrupts were causing processes to get scheduled.
>>
>> Unfortunately, I wasn't able to get a dump and had to hit reset to
>> recover.
>>
>> As I wrote above, this is only anecdotal, but I've never seen anything
>> like this before applying the patches.
>
> One-shot timers have one weak side: if for some reason timer interrupt
> getting lost -- there will be nobody to reload the timer. Such cases
> probably will require special attention. Same funny situation with
> mouse-driven scheduler happens also if LAPIC timer dies when pre-Core-iX
> CPU goes to C3 state.

I too had the "freeze" or "pause" when testing with LAPIC, although
I've been using HPET for a while now, and things seem normal:

# systat -vmstat 1

    3 users    Load  0.28  0.33  0.24                  Aug 31 08:48

Mem:KB    REAL            VIRTUAL                       VN PAGER   SWAP PAGER
        Tot   Share      Tot    Share    Free           in   out     in   out
Act  364524   14244  1600988    17404 1427028  count
All  492712   31000 1075523k    54724          pages
Proc:                                                            Interrupts
  r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Flt        cow     363 total
             74      2153  613 7083  361  144    6      5 zfod        atkbd0 1
                                                          ozfod   115 hpet0:t0 0
 1.6%Sys   0.4%Intr  2.3%User  0.0%Nice 95.7%Idle        %ozfod    56 hpet0:t1 8
|    |    |    |    |    |    |    |    |    |    |       daefr       acpi0 9
=>                                                        prcfr   187 psm0 12
                                        36 dtbuf          totfr       ata0 14
Namei     Name-cache   Dir-cache    111105 desvn          react       uhci2+ 16
   Calls    hits   %    hits   %      1953 numvn          pdwak       ehci1 19
       3       3 100                   327 frevn          pdpgs       uhci0 20
                                                          intrn       ehci0 22
Disks  ada0   cd0 pass0 pass1                      202356 wire        vgapci0
KB/t   0.00  0.00  0.00  0.00                      200264 act         hdac0 258
tps       0     0     0     0                      167392 inact     5 iwn0 259
MB/s   0.00  0.00  0.00  0.00                        4208 cache
%busy     0     0     0     0                     1422820 free

# cat /boot/loader.conf:
...
kern.hz="100"
hint.apic.0.clock="0"
hint.atrtc.0.clock="0"
hint.p4tcc.0.disabled="1"
hint.attimer.0.clock=0
hint.hpet.0.legacy_route=1
hint.acpi_throttle.0.disabled="1"
hw.pci.do_power_nodriver="3"
...

# cat /etc/rc.conf:
...
powerd_enable="YES"
powerd_flags="-a adaptive -b adaptive -n adaptive"
performance_cpu_freq="NONE"     # Online CPU frequency
economy_cpu_freq="NONE"         # Offline CPU frequency
performance_cx_lowest="C3"      # Online CPU idle state
economy_cx_lowest="C3"          # Offline CPU idle state
...

# sysctl -a | grep cx
hw.acpi.cpu.cx_lowest: C3
dev.cpu.0.cx_supported: C1/1 C2/1 C3/57
dev.cpu.0.cx_lowest: C3
dev.cpu.0.cx_usage: 0.00% 0.53% 99.46% last 2791us
dev.cpu.1.cx_supported: C1/1 C2/1 C3/57
dev.cpu.1.cx_lowest: C3
dev.cpu.1.cx_usage: 0.00% 1.13% 98.86% last 1492us

Suspend/resume still works well, for what it's worth :)

-Brandon