powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed]

[Mark Millard]

[I found a document covering PLLs for the 970MP's in the old
PowerMac G5 involved.]

> On 2019-Apr-5, at 04:35, Konstantin Belousov <kostikbel at gmail.com> wrote:
> 
> On Fri, Apr 05, 2019 at 09:13:50PM +1100, Bruce Evans wrote:
>> On Fri, 5 Apr 2019, Mark Millard wrote:
>> 
>>> On 2019-Apr-4, at 21:38, Bruce Evans <brde at optusnet.com.au> wrote:
>>> 
>>>> On Thu, 4 Apr 2019, Mark Millard wrote:
>>>>> ...
>>>>> Unfortunately, all the multi-socket contexts that I sometimes have
>>>>> access to are old PowerMacs. And, currently, the only such context
>>>>> is the G5 with 2 sockets, 2 cores per socket (powerpc64). So I've
>>>>> not been able to set up other types of examples to see if problems
>>>>> repeat.
>>>>> 
>>>>> I do not have access to a single-socket powerpc64 for contrast in
>>>>> that direction.
>>>> 
>>>> Testing 1 socket is time-consuming enough.  Do these old systems
>>>> use the equivalent of an x86 TSC for the timecounter?  With multiple
>>>> sockets, it isn't clear how even a hardware timer independent of the
>>>> CPUs can be distributed so as to appear to be monotonic on all cors.
>>> 
>>> "The DEC frequency is based on the same implementation-dependent
>>> frequency that drives the time base." The frequency may well be
>>> fixed by the PowerMac G5 model implementation but is not fixed
>>> by the powerpc64 architecture.
>>> 
>>> The Time Base Register (TBR) in a powerpc64 core (cpu in FreeBSD terms)
>>> increments at 33,333,333 Hz (nominal) and is 64 bits wide. Its value
>>> can be set (mttb instruction) and the boot sequence in FreeBSD does
>>> attempt to adjust as the FreeBSD CPU is brought-up/started. mftb is
>>> used to read the 64-bit value. FreeBSD masks it down to 32-bits to
>>> contribute to the time-counter.
>>> 
>>> (Is that description sufficient for what you were after? I've never
>>> seen documentation of how the 33,333,333 MHz is produced.)
>> 
>> This seems to be equivalent to an x86 TSC.
> It is equivalent from the interface PoV.  I saw some references that
> Powers do have per-core PLLs, the best I can find now is
> https://stackoverflow.com/questions/43844438/are-multiple-clock-domains-common-in-modern-processors

I finally found a document covering the 970MP in the old
PowerMac G5 in question:

http://datasheets.chipdb.org/IBM/PowerPC/970/PowerPC-970MP.pdf

It has material about the PLL's, pinout, etc. :

"This section will help in configuring the PLL and determining SYSCLK input
frequency for PowerPC 970MP systems." Also: 100 MHz to 700 MHz SYSCLK.
"SYSCLK minimum frequency is for PLL mode. In PLL bypass mode (BYPASS low),
SYSCLK frequency may be as low as 10MHz." BUS_CFG[0:2], PLL_MULT, and
PLL_RANGE[1:0] are involved for PLL mode.

Ratio from BUS_CFG: 3:1, 6:1, 12:1 normally use PLL_MULT=0 for a PLL multiplier of 12.
Ratio from BUS_CFG: 2:1, 4:1, 8:1 normally use PLL_MULT=1 for a PLL multiplier of 8.

(It describes the PLL power supply filtering circuit as well.)

So far I've not seen anything that is directly for the rate that the TBR
and DEC registers change.

> For Intel there is no much information, the best guess is that TSC is in
> uncore and effectively shared by all cores.  See also
> https://software.intel.com/en-us/articles/best-timing-function-for-measuring-ipp-api-timing/
> 
>> 
>> x86 P-state-invariant TSCs apparently use a 100 MHz clock which corresponds
>> even more closely to this, except for historical reasons this clock is
>> scaled and interpolated to a clock resembling the CPU cycle count at a
>> nominal frequency.
> For Intel designs, there is indeed a single-source 100MHz signal which
> is distributed over all consumers using clock fan-out buffers like DB1900Z.
> 
>> 
>>> As FreeBSD supports multi-socket, what are its criteria for a sufficient
>>> context for it to work with for supporting sbinuptime and the like? Is
>>> FreeBSD supposed to then make it appear that sbinputime and the like are
>>> weakly increasing, even as threads migrate between CPUs (cores,
>>> hw-threads)?
>> 
>> CLOCK_MONOTONIC has to be monotonic, but it is only clear what this means
>> within a single thread: sequential clock_gettime() calls must occur in
>> program order and the results must be consistent with that order.  Across
>> threads, I think it should mean that the results must be consistent with
>> any order established using any supported ordering methods.
>> 
>>> ...
>>>>> One oddity is that the eventtimer's decrementer and timecounter
>>>>> may change (nearly) together: both change at 33,333,333 Hz, as if
>>>>> they are tied to the same clock (at least on one socket).
>>>> 
>>>> I think this is from a normal hardware implementation.  On all of
>>>> my x86 systems with a TSC, the TSC frequency is an exact fractional
>>>> multiple of the i8254, the ACPI timer (if present) and the HPET (if
>>>> present).  Only the RTC has an independent frequency.  The fraction is
>>>> changed by changing the nominal TSC frequency in the BIOS, but is not
>>>> changed by temperature variations.  This must be because most clocks are
>>>> derived from a common clock using a PLL.  I use this to calibrate all
>>>> clocks (except the RTC) by calibrating only 1.
>>> 
>>> I'm not aware of the OpenFirmware having any control over the
>>> TBR-change frequency behavior. I've no evidence about any variability
>>> based on temperature.
>> 
>> Temperature changes usually affect the actual frequency but not the
>> nominal frequency.
>> 

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)