970MP PowerMac G5s: What printf's show about cpu_mp_unleash hangups on the test variant of head -r347003 (found cpu_mp_unleash counterexample)

[Mark Millard]

[Exeriment with instead disabling the code in each of
the nop_prio_ routines instead of commenting out specific
calls. But mostly: more test runs. It does not support
what I thought yesterday's cpu_mp_unlead suggested.]

On 2019-May-2, at 22:23, Mark Millard <marklmi at yahoo.com> wrote:

> [Note: I still have your requested loop change, my
> isync additions, and my libc string compare code
> change in what I'm working with for head -r347003 .]
> 
> I started using printf to help identify more about what
> code managed to execute vs what code did not for
> hang-ups.
> 
> This note is just about cpu_mp_unleash observations and
> experiments related to what printf's showed.
> 
> I did:
> 
> static void
> cpu_mp_unleash(void *dummy)
> {
> . . . (omitted as all earlier printf's printed) . . .
> printf("cpu_mp_unleash: before DELAY\n");
>        /* Let the APs get into the scheduler */
>        DELAY(10000);
> printf("cpu_mp_unleash: after DELAY\n");
> 
> }
> 
> What I saw was only the first of the twoDEALY printf's
> shown above was printing when cpu_mp_unleash hung up,
> such a hangup being the normal case when vt_upgrade
> did not hang-up first.
> 
> So I looked at /mnt/usr/src/sys/powerpc/powerpc/clock.c
> and its DELAY routine and came up with only one thing
> that looked like a useful experiment. Note what I
> then commented out:
> 
> # svnlite diff /mnt/usr/src/sys/powerpc/powerpc/clock.c
> Index: /mnt/usr/src/sys/powerpc/powerpc/clock.c
> ===================================================================
> --- /mnt/usr/src/sys/powerpc/powerpc/clock.c	(revision 347003)
> +++ /mnt/usr/src/sys/powerpc/powerpc/clock.c	(working copy)
> @@ -309,10 +309,10 @@
> 	TSENTER();
> 	tb = mftb();
> 	ttb = tb + howmany((uint64_t)n * 1000000, ps_per_tick);
> -	nop_prio_vlow();
> +	//nop_prio_vlow();
> 	while (tb < ttb)
> 		tb = mftb();
> -	nop_prio_medium();
> +	//nop_prio_medium();
> 	TSEXIT();
> }
> 
> After this change I've not (yet?) seen another cpu_mp_unleash
> hangup in my test context.
> 
> Even if not documented to do so, it appears to me that
> ori Rx,Rx,Rx code that is behind the nop_prio_vlow() does
> something specific on the 970MP's in the 2-socket/2-core-each
> G5 PowerMac11,2's --and what it does interferes with making
> progress in DELAY, in at least that specific use of it and/or
> any others on the ap's during cpu_mp_unleash.
> 
> Of course, this testing process is of a probabilistic context
> and I do not have hundreds or more of examples of any specific
> condition at this point. But, so far, the change in behavior
> seems clear: I went from always-hanging-up-so-far to
> always-booting-so-far (when vt_upgrade did not prevent the
> test in each context).

So I uncommented the 2 calls commented out the contents of
the nop_prio_ routines, summarized here via:


# egrep -r 'or.*(31,31,31|1,1,1|6,6,6|2,2,2|5,5,5|3,3,3)' /mnt/usr/src/sys/powerpc/ | more                                                                                                       /mnt/usr/src/sys/powerpc/include/cpufunc.h:     //__asm __volatile("or 31,31,31");
/mnt/usr/src/sys/powerpc/include/cpufunc.h:     //__asm __volatile("or 1,1,1");
/mnt/usr/src/sys/powerpc/include/cpufunc.h:     //__asm __volatile("or 6,6,6");
/mnt/usr/src/sys/powerpc/include/cpufunc.h:     //__asm __volatile("or 2,2,2");
/mnt/usr/src/sys/powerpc/include/cpufunc.h:     //__asm __volatile("or 5,5,5");
/mnt/usr/src/sys/powerpc/include/cpufunc.h:     //__asm __volatile("or 3,3,3");

Then I've continued running tests based on the rebuild. The
results so far are . . .

vt_upgrade behavior is unchanged, still frequently hanging-up
in the same place.

fhanew_init behavior is unchanged, still frequently hanging-up
in the same place (when vt_upgrade does not hang-up).

It will take some time to get a number of examples where the
prior 2 hang-ups do not occur, but for cpu_mp_unlead I have
seen a "3 CPUs woken" notice followed by only the
"before DELAY" message, no "after DELAY" one. I confirmed
with objdump that DELAY did not have the ori Rx,Rx,Rx
instructions.

So, despite how things looked yesterday with no cpu_mp_unlead
hang-ups after using the code with the 2 calls commented out,
ori 31,31,31 use is not involved in the latest example, matching
your expectations.

(And this illustrates why I try to accumulate a fair number of
examples over time: the hang-ups are probabilistic and small
example counts can be misleading.)



===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)