powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed]

Sat Apr 6 07:54:15 UTC 2019

On 2019-Apr-5, at 14:31, Mark Millard <marklmi at yahoo.com> wrote:

[I found a document covering PLLs for the 970MP's in the old
PowerMac G5 involved.]

> On 2019-Apr-5, at 04:35, Konstantin Belousov <kostikbel at gmail.com> wrote:
>> 
>> On Fri, Apr 05, 2019 at 09:13:50PM +1100, Bruce Evans wrote:
>>> On Fri, 5 Apr 2019, Mark Millard wrote:
>>> 
>>>> On 2019-Apr-4, at 21:38, Bruce Evans <brde at optusnet.com.au> wrote:
>>>> 
>>>>> On Thu, 4 Apr 2019, Mark Millard wrote:
>>>>>> ...
>>>>>> Unfortunately, all the multi-socket contexts that I sometimes have
>>>>>> access to are old PowerMacs. And, currently, the only such context
>>>>>> is the G5 with 2 sockets, 2 cores per socket (powerpc64). So I've
>>>>>> not been able to set up other types of examples to see if problems
>>>>>> repeat.
>>>>>> 
>>>>>> I do not have access to a single-socket powerpc64 for contrast in
>>>>>> that direction.
>>>>> 
>>>>> Testing 1 socket is time-consuming enough.  Do these old systems
>>>>> use the equivalent of an x86 TSC for the timecounter?  With multiple
>>>>> sockets, it isn't clear how even a hardware timer independent of the
>>>>> CPUs can be distributed so as to appear to be monotonic on all cors.
>>>> 
>>>> "The DEC frequency is based on the same implementation-dependent
>>>> frequency that drives the time base." The frequency may well be
>>>> fixed by the PowerMac G5 model implementation but is not fixed
>>>> by the powerpc64 architecture.
>>>> 
>>>> The Time Base Register (TBR) in a powerpc64 core (cpu in FreeBSD terms)
>>>> increments at 33,333,333 Hz (nominal) and is 64 bits wide. Its value
>>>> can be set (mttb instruction) and the boot sequence in FreeBSD does
>>>> attempt to adjust as the FreeBSD CPU is brought-up/started. mftb is
>>>> used to read the 64-bit value. FreeBSD masks it down to 32-bits to
>>>> contribute to the time-counter.
>>>> 
>>>> (Is that description sufficient for what you were after? I've never
>>>> seen documentation of how the 33,333,333 MHz is produced.)
>>> 
>>> This seems to be equivalent to an x86 TSC.
>> It is equivalent from the interface PoV.  I saw some references that
>> Powers do have per-core PLLs, the best I can find now is
>> https://stackoverflow.com/questions/43844438/are-multiple-clock-domains-common-in-modern-processors
> 
> I finally found a document covering the 970MP in the old
> PowerMac G5 in question:
> 
> http://datasheets.chipdb.org/IBM/PowerPC/970/PowerPC-970MP.pdf
> 
> It has material about the PLL's, pinout, etc. :
> 
> "This section will help in configuring the PLL and determining SYSCLK input
> frequency for PowerPC 970MP systems." Also: 100 MHz to 700 MHz SYSCLK.
> "SYSCLK minimum frequency is for PLL mode. In PLL bypass mode (BYPASS low),
> SYSCLK frequency may be as low as 10MHz." BUS_CFG[0:2], PLL_MULT, and
> PLL_RANGE[1:0] are involved for PLL mode.
> 
> Ratio from BUS_CFG: 3:1, 6:1, 12:1 normally use PLL_MULT=0 for a PLL multiplier of 12.
> Ratio from BUS_CFG: 2:1, 4:1, 8:1 normally use PLL_MULT=1 for a PLL multiplier of 8.
> 
> (It describes the PLL power supply filtering circuit as well.)
> 
> So far I've not seen anything that is directly for the rate that the TBR
> and DEC registers change.
> 
>> For Intel there is no much information, the best guess is that TSC is in
>> uncore and effectively shared by all cores.  See also
>> https://software.intel.com/en-us/articles/best-timing-function-for-measuring-ipp-api-timing/
>> 
>>> 
>>> x86 P-state-invariant TSCs apparently use a 100 MHz clock which corresponds
>>> even more closely to this, except for historical reasons this clock is
>>> scaled and interpolated to a clock resembling the CPU cycle count at a
>>> nominal frequency.
>> For Intel designs, there is indeed a single-source 100MHz signal which
>> is distributed over all consumers using clock fan-out buffers like DB1900Z.
>> 
>>> 
>>>> As FreeBSD supports multi-socket, what are its criteria for a sufficient
>>>> context for it to work with for supporting sbinuptime and the like? Is
>>>> FreeBSD supposed to then make it appear that sbinputime and the like are
>>>> weakly increasing, even as threads migrate between CPUs (cores,
>>>> hw-threads)?
>>> 
>>> CLOCK_MONOTONIC has to be monotonic, but it is only clear what this means
>>> within a single thread: sequential clock_gettime() calls must occur in
>>> program order and the results must be consistent with that order.  Across
>>> threads, I think it should mean that the results must be consistent with
>>> any order established using any supported ordering methods.
>>> 
>>>> ...
>>>>>> One oddity is that the eventtimer's decrementer and timecounter
>>>>>> may change (nearly) together: both change at 33,333,333 Hz, as if
>>>>>> they are tied to the same clock (at least on one socket).
>>>>> 
>>>>> I think this is from a normal hardware implementation.  On all of
>>>>> my x86 systems with a TSC, the TSC frequency is an exact fractional
>>>>> multiple of the i8254, the ACPI timer (if present) and the HPET (if
>>>>> present).  Only the RTC has an independent frequency.  The fraction is
>>>>> changed by changing the nominal TSC frequency in the BIOS, but is not
>>>>> changed by temperature variations.  This must be because most clocks are
>>>>> derived from a common clock using a PLL.  I use this to calibrate all
>>>>> clocks (except the RTC) by calibrating only 1.
>>>> 
>>>> I'm not aware of the OpenFirmware having any control over the
>>>> TBR-change frequency behavior. I've no evidence about any variability
>>>> based on temperature.
>>> 
>>> Temperature changes usually affect the actual frequency but not the
>>> nominal frequency.
>>> 
> 

I found more about the 970MP's TB/DEC rate:

"The TBEN input pin can be used as an enable for the internal
timebase/decrementer or as an external clock input." Details:

HID0 bit 19 = 0: update at 1/16th processor core frequency,
                 but only when TBEN is high.

HID0 bit 19 = 1: update at rising edge of TBEN
                 (must not exceed 1/16th of the core processor's
                  maximum frequency).

So the 33,333,333 Hz TB&DEC update rate vs. 2500 MHz would mean
that rising edges of TBEN are in use (HID0 bit 19 = 1) in the
PowerMac G5 context in question.

I've no information about how closely matched the TBEN signals are
at the 2 sockets, beyond the nominal frequency.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)