Looks like ap_letgo use needs platform specific code to allow avoiding the "sleep-gets-stuck" problem on PowerMac11,2's . . .

[Justin Hibbits]

On Fri, Apr 19, 2019 at 10:36 PM Mark Millard <marklmi at yahoo.com> wrote:
>
> [Note: My context is tied to getting usefdt mode operable on
> old PowerMacs. The below is only tested for usefdt mode
> so far.]
>
> The following investigatory patch has so-far stopped my having
> sleep-gets-stuck problems (only seen on 2-socket/1-core-each
> 970 MP G5 Powermac11,2's as far as I know):
>
> # svnlite diff  /usr/src/sys/powerpc/powerpc/mp_machdep.c | more                                                                                                                                 Index: /usr/src/sys/powerpc/powerpc/mp_machdep.c
> ===================================================================
> --- /usr/src/sys/powerpc/powerpc/mp_machdep.c   (revision 345758)
> +++ /usr/src/sys/powerpc/powerpc/mp_machdep.c   (working copy)
> @@ -77,9 +77,10 @@
>         PCPU_SET(awake, 1);
>         __asm __volatile("msync; isync");
>
> +       powerpc_sync();
>         while (ap_letgo == 0)
> -               __asm __volatile("or 31,31,31");
> -       __asm __volatile("or 6,6,6");
> +               powerpc_sync();
> +       isync();
>
>         /*
>          * Set timebase as soon as possible to meet an implicit rendezvous
> @@ -262,8 +263,11 @@
>         __asm __volatile("msync; isync");
>
>         /* Let APs continue */
> -       atomic_store_rel_int(&ap_letgo, 1);
> +       ap_letgo= 1;    // depend on prior sync, no need to lwsync first
>
> +       powerpc_sync(); // analogous to what the ap's do (more similar time frame?)
> +       if (ap_letgo) isync();
> +
>         platform_smp_timebase_sync(ap_timebase, 0);
>
>         while (ap_awake < smp_cpus)
>
> Apparently, the use of "or 31,31,31" causes sizable
> variations in the time frame when the platform_smp_timebase_sync
> happens on the various cores across the two 970MPs.
>
> It looks something like a platform_ap_letgo_wait is appropriate,
> with a powermac_ap_letgo_wait specific one, say. (Or, possibly,
> AIM specific but spanning powermac?)
>
> The above patch has booted and operated the 2-socket PowerMac7,2
> context fine as well so far. I'll check the G5's and a G4
> dual-socket with a 32-bit powerpc build.
>
> I've no clue if there are any time-mismatch issues across
> sockets/cores/hw-threads for the "8-way SMT" contexts with
> "dozens to hundreds of CPUs".
>
> I've only been testing for part of today and I do not have
> access to any non-PowerMac PowerPC contexts. So this is
> preliminary but I do not expect "or 31,31,31" is going to
> be appropriate to the PowerMac11,2 contexts that caused
> my investigation of the issue.

Those nops are just that on the G5: nops.  They do absolutely nothing
on any processor that's not multithreaded.  On multithreaded CPUs they
are priority hints.

What most likely 'fixed' the problem for you was the addition of the
synchronization primitives in the code path.  You can add them without
sacrificing the nops.  Does this patch fix the boot issue on the G5
quad without the usefdt=1 setting, and without reverting the KVA
change?

- Justin