Looks like ap_letgo use needs platform specific code to allow avoiding the "sleep-gets-stuck" problem on PowerMac11,2's . . .

[Mark Millard]

[Note: My context is tied to getting usefdt mode operable on
old PowerMacs. The below is only tested for usefdt mode
so far.]

The following investigatory patch has so-far stopped my having
sleep-gets-stuck problems (only seen on 2-socket/1-core-each
970 MP G5 Powermac11,2's as far as I know):

# svnlite diff  /usr/src/sys/powerpc/powerpc/mp_machdep.c | more                                                                                                                                 Index: /usr/src/sys/powerpc/powerpc/mp_machdep.c
===================================================================
--- /usr/src/sys/powerpc/powerpc/mp_machdep.c   (revision 345758)
+++ /usr/src/sys/powerpc/powerpc/mp_machdep.c   (working copy)
@@ -77,9 +77,10 @@
        PCPU_SET(awake, 1);
        __asm __volatile("msync; isync");
 
+       powerpc_sync();
        while (ap_letgo == 0)
-               __asm __volatile("or 31,31,31");
-       __asm __volatile("or 6,6,6");
+               powerpc_sync();
+       isync();
 
        /*
         * Set timebase as soon as possible to meet an implicit rendezvous
@@ -262,8 +263,11 @@
        __asm __volatile("msync; isync");
        
        /* Let APs continue */
-       atomic_store_rel_int(&ap_letgo, 1);
+       ap_letgo= 1;    // depend on prior sync, no need to lwsync first
 
+       powerpc_sync(); // analogous to what the ap's do (more similar time frame?)
+       if (ap_letgo) isync();
+
        platform_smp_timebase_sync(ap_timebase, 0);
 
        while (ap_awake < smp_cpus)

Apparently, the use of "or 31,31,31" causes sizable
variations in the time frame when the platform_smp_timebase_sync
happens on the various cores across the two 970MPs.

It looks something like a platform_ap_letgo_wait is appropriate,
with a powermac_ap_letgo_wait specific one, say. (Or, possibly,
AIM specific but spanning powermac?)

The above patch has booted and operated the 2-socket PowerMac7,2
context fine as well so far. I'll check the G5's and a G4
dual-socket with a 32-bit powerpc build.

I've no clue if there are any time-mismatch issues across
sockets/cores/hw-threads for the "8-way SMT" contexts with
"dozens to hundreds of CPUs".

I've only been testing for part of today and I do not have
access to any non-PowerMac PowerPC contexts. So this is
preliminary but I do not expect "or 31,31,31" is going to
be appropriate to the PowerMac11,2 contexts that caused
my investigation of the issue.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)