spinlock_enter, head -r346144 (and before) and use of nop_prio_mhigh vs. PowerISA document suggestions for lock code

[Mark Millard]

[Thanks for the notes.]

On 2019-Apr-22, at 08:48, Justin Hibbits <chmeeedalf at gmail.com> wrote:

> On Fri, 19 Apr 2019 17:22:24 -0700
> Mark Millard <marklmi at yahoo.com> wrote:
> 
>> [Looks like nop_prio_mhigh has some vintage-specific
>> behavior, based on if there is a PSPB (Problem State
>> Priority Boost Register) and how it is configured.]
> 
> As mentioned in the references, these nops are hints only.  They're
> completely ignored on older hardware, so the existence of a PSPBR is
> inconsequential.

I was more thinking that if one potentially wanted PSPB use
with medium high, then trying to mix in medium high use that
was not after the PSPB related behavior might not go well.
So reserving medium high to potential PSPB uses might then
be appropriate if such were expected.

That might be enough to cause their wording that only suggested
"low" on the waiting side.

>> 
>> On 2019-Apr-19, at 15:56, Mark Millard <marklmi at yahoo.com> wrote:
>> 
>>> I found the following text in each of the 2.03, 2.04, 2.05,
>>> 2.06B V2, 2.07, and 3.0B PowerISA documents, in a "Programming
>>> Note" in the "Program Priority Registers" section:
>>> 
>>> ". . . if a program is waiting on a lock (...), it could set low
>>> priority with the result that more processor resources would be
>>> diverted to the program that holds the lock. This diversion of
>>> resources may enable the lock-holding program to complete the
>>> operation under the lock more quickly, and then relinquish the lock
>>> to the waiting program."
>>> 
>>> The wording suggests working via normal/medium and lower
>>> priorities instead of via normal/medium and higher priorities.
>>> (May be more than just the relative priority matters in the
>>> behavior changes that result each way? Unfortunately the
>>> wording is not very explicit.)
> 
> But it's not waiting on a lock here, it's holding the lock.  The idea
> is to hold the lock at high priority so it gets more cycles to complete
> its work, then drops back to normal priority as soon as possible.
> There is work remaining to drop to very low priority during the
> spinwait phase of mutexes, which will be done soon.

Ahh, eventually going both higher and lower than normal/medium. Good
to know.

>>> 
>>> All of the documents list "or rx,rx,rx" for:
>>> Rx being 31 or 1 or 6 or 2 or 5 or 3 or 7
>>> (listed lowest to highest relative priority),
>>> 2 being normal/medium. Some table(s) might not
>>> list 3 or 7 in a document but at least one does
>>> in each.  
>> 
>> Actually, going back to 2.03, for example, only lists 31
>> in one place as well. Only 1, 6, and 2 are in all
>> such tables.
>> 
>>> In the following powerpc64 and 32-bit powerpc
>>> FreeBSD seem to be going in the opposite direction
>>> relative to normal/medium vs. the suggestion
>>> of "low priority":
> 
> These nops are only relevant on hardware that only properly supports
> 64-bit kernels.

Yep: 32-bit fits with the "ignored on older hardware" that
you mentioned earlier.

>>> 
>>> void
>>> spinlock_enter(void)
>>> {
>>> 	struct thread *td;
>>> 	register_t msr;
>>> 
>>> 	td = curthread;
>>> 	if (td->td_md.md_spinlock_count == 0) {
>>> 		nop_prio_mhigh();
>>> 		msr = intr_disable();
>>> 		td->td_md.md_spinlock_count = 1;
>>> 		td->td_md.md_saved_msr = msr;
>>> 	} else
>>> 		td->td_md.md_spinlock_count++;
>>> 	critical_enter();
>>> }
>>> 
>>> void
>>> spinlock_exit(void)
>>> {
>>> 	struct thread *td;
>>> 	register_t msr;
>>> 
>>> 	td = curthread;
>>> 	critical_exit();
>>> 	msr = td->td_md.md_saved_msr;
>>> 	td->td_md.md_spinlock_count--;
>>> 	if (td->td_md.md_spinlock_count == 0) {
>>> 		intr_restore(msr);
>>> 		nop_prio_medium();
>>> 	}
>>> }
>>> 
>>> and previously:
>>> 
>>> void
>>> spinlock_enter(void)
>>> {
>>>       struct thread *td;
>>>       register_t msr;
>>> 
>>>       td = curthread;
>>>       if (td->td_md.md_spinlock_count == 0) {
>>>               __asm __volatile("or 2,2,2"); /* Set high thread
>>> priority */ msr = intr_disable();
>>>               td->td_md.md_spinlock_count = 1;
>>>               td->td_md.md_saved_msr = msr;
>>>       } else
>>>               td->td_md.md_spinlock_count++;
>>>       critical_enter();
>>> }
>>> 
>>> void
>>> spinlock_exit(void)
>>> {
>>>       struct thread *td;
>>>       register_t msr;
>>> 
>>>       td = curthread;
>>>       critical_exit();
>>>       msr = td->td_md.md_saved_msr;
>>>       td->td_md.md_spinlock_count--;
>>>       if (td->td_md.md_spinlock_count == 0) {
>>>               intr_restore(msr);
>>>               __asm __volatile("or 6,6,6"); /* Set normal thread
>>> priority */ }
>>> }
>>> 
>>> (2,2,2 was higher then 6,6,6 but 2,2,2 is
>>> normal/medium and 6,6,6 is "medium low" the
>>> way the PowerISA documentation reads.)
>>> 
>>> 2.06B V2 and 2.07 also list special meanings for:
>>> 27 and 29 and 30. (cpu_spinwait in FreeBSD uses 27.)
>>> But 3.0B does not list them any more.
>>> 
>>> 2.07 and 3.0B lists a special meaning for: 26.
>>> No prior version that I looked at does.
>>> 
>> 
>> PSPB (Problem State Priority Boost Register)is not 
>> mentioned until 2.07 of the PowerISA (for those I've
>> been looking at).
>> 
>> PSPB can count down and force Medium High back
>> to Medium, for  example. A hardware form of
>> timed-temporary priority boost when used that way.
>> 
>> It appears that medium and lower priorities do
>> not have such a means of control. Nor do
>> higher priorities, just Medium High.
> 
> Medium-high is the highest non-privileged can go, no higher.  You can
> see that we make no reference to the priority boost register because it
> doesn't matter.  If it exists, userspace might get a slight boost for a
> few milliseconds, but if it doesn't, or is configured to not permit it,
> then the mhigh nop is a complete nop.  The nop_prio_*() inlines are
> intended for kernel only, but aren't held to it because it's not
> strictly necessary.
> 

Thanks again for the explanations.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)