How many segments does it take to span from VM_MIN_KERNEL_ADDRESS through VM_MAX_SAFE_KERNEL_ADDRESS? 128 in moea64_late_bootstrap

[Mark Millard]

On 2019-May-1, at 14:54, Justin Hibbits <chmeeedalf at gmail.com> wrote:

> On Wed, 1 May 2019 14:35:56 -0700
> Mark Millard <marklmi at yahoo.com> wrote:
> 
>>>> What happens if you revert all your patches,  
>>> 
>>> Most of the patches in Bugzilla 233863 are not for this
>>> issue at all and are not tied to starting the non-bsp
>>> cpus. (The one for improving how close the Time Base
>>> registers are is tied to starting these cpus.) Only the
>>> aim/mp_cpudep.c and aim/slb.c changes seem relevant.
>>> 
>>> Are you worried about some form of interaction that means
>>> I need to avoid patches for other issues?
>>> 
>>> Note: for now I'm staying at using head -r345758 as the
>>> basis for my experiments.
>>> 
>>>> and change this loop to
>>>> stop at n_slb?  So something more akin to:
>>>> 
>>>> 	int i = 0;
>>>> 
>>>> 	for (va = virtual_avail; va < virtual_end && i < n_slb -
>>>> 1; va += SEGMENT_LENGTH, i++);
>>>> 		...
>>>> 
>>>> If it reliably boots with that, then that's fine.  We can prefault
>>>> as much as we can and leave the rest for on-demand.  
>>> 
>>> I'm happy to experiment with this loop without my hack
>>> for forcing the slb entry to exist in cpudep_ap_bootstrap.
>>> 
>>> But, it seems to presume that the pc_curpcb's will
>>> all always point into the lower address range spanned
>>> when cpudep_ap_bootstrap is executing on the cpu.
>>> Does some known property limit the pc_curpcb->
>>> references to such? Only that would be sure to
>>> avoid an slb-miss at that stage. Or is this just an
>>> alternate hack or a means of getting evidence, not a
>>> proposed solution?
>>> 
>>> (Again, I'm happy to disable my hack that forces the
>>> slb entry and to try the loop suggested.)  
> ...
>> And the patch for the loop looks like:
>> 
>> 	virtual_end = VM_MAX_SAFE_KERNEL_ADDRESS; 
>> 
>> 	/*
>> -	 * Map the entire KVA range into the SLB. We must not fault
>> there.
>> +	 * Map the lower-address part of the KVA range into the SLB.
>> We must not fault there. */
>> 	#ifdef __powerpc64__
>> -	for (va = virtual_avail; va < virtual_end; va +=
>> SEGMENT_LENGTH)
>> +	i = 0;
>> +	for (va = virtual_avail; va < virtual_end && i<n_slbs-1; va
>> += SEGMENT_LENGTH, i++) moea64_bootstrap_slb_prefault(va, 0);
>> 	#endif
>> 
> 
> Yep, that's the patch I was going for.
> 
>> 
>> So I've built, installed, and have tested some: it did not go well
>> overall.
>> 
>> Using:
>> 
>> OK set debug.verbose_sysinit=1
>> 
>> to show better context about where the hangs occur, shows:
>> (Typed from a screen picture.)
>> 
>> subsystem a800000
>>  boot_run_interrupt_driven_config_hooks(0)...
>> . . . (omitted) . . .
>> done.
>>  vt_upgrade(&vt_consdev). . .
>> 
>> The "vt_upgrade(&vt_consdev). . ." never says done when booting
>> hangs with the above changes.
>> 
>> Trying to boot a bunch of times did produce one
>> completed boot, all 4 cpus working. Otherwise I'm
>> using kernel.old to manage to complete a boot.
>> 
>> I'll note that "vt_upgrade(&vt_consdev). . ." is where
>> Dennis Clarke reported for the hangups that he was
>> seeing, without any of my patches being available back
>> then: 2019-Feb-14.
> 
> Maybe try the commit that caused the problem back in July?  r334498.
> 

I'd already started down the path of getting materials from:

https://artifact.ci.freebsd.org/snapshot/head/r347003/powerpc/powerpc64/

and putting them on a separate SSD that I sometimes use for artifact.ci
or snapshot experiments. Also: checking out matching svn sources for
-r347003 and then doing a buildworld buildkernel with a bootstrap gcc
4.2.1 compiler used. I'm verifying that I can build it before making
the source changes for the kernel. The build is of a debug kernel
(GENERIC64).

The test buildworld is still in process.

Let me know if this is insufficient for your purposes. I could revert
to:

https://artifact.ci.freebsd.org/snapshot/head/r334594/powerpc/powerpc64/

(There is no head/r334498/ and the first after that with a
powerpc64/ is head/r334594/ .)

For either head/r347003/ or head/r334594/ :

Use of artifact materials allows using officially built files for
every file but some specific file(s) that I replace. It also allows
comparison/contrast of the behavior of the official files vs. when
adjusted ones are substituted.

Use of artifact-version materials also means that I know I'm using
a vintage that actually built --and so I hope to avoid other problems
getting in the way.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)