powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed]

[Mark Millard]

[Another document was more explicit about 1/16 mode.]

On 2019-Apr-6, at 00:54, Mark Millard <marklmi at yahoo.com> wrote:

> On 2019-Apr-5, at 14:31, Mark Millard <marklmi at yahoo.com> wrote:
> 
> [I found a document covering PLLs for the 970MP's in the old
> PowerMac G5 involved.]
> 
>> On 2019-Apr-5, at 04:35, Konstantin Belousov <kostikbel at gmail.com> wrote:
>>> 
>>> On Fri, Apr 05, 2019 at 09:13:50PM +1100, Bruce Evans wrote:
>>>> On Fri, 5 Apr 2019, Mark Millard wrote:
>>>> 
>>>>> On 2019-Apr-4, at 21:38, Bruce Evans <brde at optusnet.com.au> wrote:
>>>>> 
>>>>>> On Thu, 4 Apr 2019, Mark Millard wrote:
>>>>>>> ...
>>>>>>> Unfortunately, all the multi-socket contexts that I sometimes have
>>>>>>> access to are old PowerMacs. And, currently, the only such context
>>>>>>> is the G5 with 2 sockets, 2 cores per socket (powerpc64). So I've
>>>>>>> not been able to set up other types of examples to see if problems
>>>>>>> repeat.
>>>>>>> 
>>>>>>> I do not have access to a single-socket powerpc64 for contrast in
>>>>>>> that direction.
>>>>>> 
>>>>>> Testing 1 socket is time-consuming enough.  Do these old systems
>>>>>> use the equivalent of an x86 TSC for the timecounter?  With multiple
>>>>>> sockets, it isn't clear how even a hardware timer independent of the
>>>>>> CPUs can be distributed so as to appear to be monotonic on all cors.
>>>>> 
>>>>> "The DEC frequency is based on the same implementation-dependent
>>>>> frequency that drives the time base." The frequency may well be
>>>>> fixed by the PowerMac G5 model implementation but is not fixed
>>>>> by the powerpc64 architecture.
>>>>> 
>>>>> The Time Base Register (TBR) in a powerpc64 core (cpu in FreeBSD terms)
>>>>> increments at 33,333,333 Hz (nominal) and is 64 bits wide. Its value
>>>>> can be set (mttb instruction) and the boot sequence in FreeBSD does
>>>>> attempt to adjust as the FreeBSD CPU is brought-up/started. mftb is
>>>>> used to read the 64-bit value. FreeBSD masks it down to 32-bits to
>>>>> contribute to the time-counter.
>>>>> 
>>>>> (Is that description sufficient for what you were after? I've never
>>>>> seen documentation of how the 33,333,333 MHz is produced.)
>>>> 
>>>> This seems to be equivalent to an x86 TSC.
>>> It is equivalent from the interface PoV.  I saw some references that
>>> Powers do have per-core PLLs, the best I can find now is
>>> https://stackoverflow.com/questions/43844438/are-multiple-clock-domains-common-in-modern-processors
>> 
>> I finally found a document covering the 970MP in the old
>> PowerMac G5 in question:
>> 
>> http://datasheets.chipdb.org/IBM/PowerPC/970/PowerPC-970MP.pdf
>> 
>> It has material about the PLL's, pinout, etc. :
>> 
>> "This section will help in configuring the PLL and determining SYSCLK input
>> frequency for PowerPC 970MP systems." Also: 100 MHz to 700 MHz SYSCLK.
>> "SYSCLK minimum frequency is for PLL mode. In PLL bypass mode (BYPASS low),
>> SYSCLK frequency may be as low as 10MHz." BUS_CFG[0:2], PLL_MULT, and
>> PLL_RANGE[1:0] are involved for PLL mode.
>> 
>> Ratio from BUS_CFG: 3:1, 6:1, 12:1 normally use PLL_MULT=0 for a PLL multiplier of 12.
>> Ratio from BUS_CFG: 2:1, 4:1, 8:1 normally use PLL_MULT=1 for a PLL multiplier of 8.
>> 
>> (It describes the PLL power supply filtering circuit as well.)
>> 
>> So far I've not seen anything that is directly for the rate that the TBR
>> and DEC registers change.
>> 
>>> For Intel there is no much information, the best guess is that TSC is in
>>> uncore and effectively shared by all cores.  See also
>>> https://software.intel.com/en-us/articles/best-timing-function-for-measuring-ipp-api-timing/
>>> 
>>>> 
>>>> x86 P-state-invariant TSCs apparently use a 100 MHz clock which corresponds
>>>> even more closely to this, except for historical reasons this clock is
>>>> scaled and interpolated to a clock resembling the CPU cycle count at a
>>>> nominal frequency.
>>> For Intel designs, there is indeed a single-source 100MHz signal which
>>> is distributed over all consumers using clock fan-out buffers like DB1900Z.
>>> 
>>>> 
>>>>> As FreeBSD supports multi-socket, what are its criteria for a sufficient
>>>>> context for it to work with for supporting sbinuptime and the like? Is
>>>>> FreeBSD supposed to then make it appear that sbinputime and the like are
>>>>> weakly increasing, even as threads migrate between CPUs (cores,
>>>>> hw-threads)?
>>>> 
>>>> CLOCK_MONOTONIC has to be monotonic, but it is only clear what this means
>>>> within a single thread: sequential clock_gettime() calls must occur in
>>>> program order and the results must be consistent with that order.  Across
>>>> threads, I think it should mean that the results must be consistent with
>>>> any order established using any supported ordering methods.
>>>> 
>>>>> ...
>>>>>>> One oddity is that the eventtimer's decrementer and timecounter
>>>>>>> may change (nearly) together: both change at 33,333,333 Hz, as if
>>>>>>> they are tied to the same clock (at least on one socket).
>>>>>> 
>>>>>> I think this is from a normal hardware implementation.  On all of
>>>>>> my x86 systems with a TSC, the TSC frequency is an exact fractional
>>>>>> multiple of the i8254, the ACPI timer (if present) and the HPET (if
>>>>>> present).  Only the RTC has an independent frequency.  The fraction is
>>>>>> changed by changing the nominal TSC frequency in the BIOS, but is not
>>>>>> changed by temperature variations.  This must be because most clocks are
>>>>>> derived from a common clock using a PLL.  I use this to calibrate all
>>>>>> clocks (except the RTC) by calibrating only 1.
>>>>> 
>>>>> I'm not aware of the OpenFirmware having any control over the
>>>>> TBR-change frequency behavior. I've no evidence about any variability
>>>>> based on temperature.
>>>> 
>>>> Temperature changes usually affect the actual frequency but not the
>>>> nominal frequency.
>>>> 
>> 
> 
> I found more about the 970MP's TB/DEC rate:
> 
> "The TBEN input pin can be used as an enable for the internal
> timebase/decrementer or as an external clock input." Details:
> 
> HID0 bit 19 = 0: update at 1/16th processor core frequency,
>                 but only when TBEN is high.

That 1/16th should be of "the full processor frequency, even when the
processor itself is running at a lower frequency".

The 970FX has 1/8th instead of 1/16th.

"Since the mesh clock frequency can be lowered to 1/64th of the
full-speed, the time base/decrementer may be increased/decreased
by more than one at a time." The decrementer tests for wrap, not
for reaching zero.


> HID0 bit 19 = 1: update at rising edge of TBEN
>                 (must not exceed 1/16th of the core processor's
>                  maximum frequency).
> 
> So the 33,333,333 Hz TB&DEC update rate vs. 2500 MHz would mean
> that rising edges of TBEN are in use (HID0 bit 19 = 1) in the
> PowerMac G5 context in question.
> 
> I've no information about how closely matched the TBEN signals are
> at the 2 sockets, beyond the nominal frequency.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)