PINE64+ Release APs, APs not started - a solution...

[Henri Hennebert]

On 07/25/2017 19:22, Henri Hennebert wrote:
> Hello,
> 
> With FreeBSD 12.0-CURRENT #0 r320599 I can boot normally:
> 
> ...
> Release APs
> CPU  0: ARM Cortex-A53 r0p4 affinity:  0
>   Instruction Set Attributes 0 = <AES+PMULL,SHA1,SHA2,CRC32>
>   Instruction Set Attributes 1 = <0>
>           Processor Features 0 = <AdvSIMD,Float,EL3 32,EL2 32,EL1 32,EL0 
> 32>
>           Processor Features 1 = <0>
>        Memory Model Features 0 = <4k Granule,64k 
> Granule,MixedEndian,S/NS Mem,16bit ASID,1TB PA>
>        Memory Model Features 1 = <>
>               Debug Features 0 = <2 CTX Breakpoints,4 Watchpoints,6 
> Breakpoints,PMUv3,Debug v8>
>               Debug Features 1 = <0>
>           Auxiliary Features 0 = <0>
>           Auxiliary Features 1 = <0>
> CPU  1: ARM Cortex-A53 r0p4 affinity:  1
> CPU  2: ARM Cortex-A53 r0p4 affinity:  2
> CPU  3: ARM Cortex-A53 r0p4 affinity:  3
> ...
> 
> With r320869, I get
> 
> Release APs
> APs not started
>    x0: ffff000000977d80
>    x1: fffffd000299ba80
>    x2:                3
> 
> I succeed one time to boot with r321003 but after that I try multiple 
> times and get the same error: APs not started
> 
> I try r321371 to no avail.
> 
> Any idea ?
> 
> Henri

If I boot with kernel r320599 (unload; load /boot/kernel.r320599/kernel;
set module_path=/boot/kernel.r320599; load zfs; load opensolaris; boot),
the boot complete without problem.
Then I do a shutdown -r now to reboot with my default kernel
(/boot/kernel) the boot complete without problem:

[root at norquay ~]# uname -a
FreeBSD norquay.restart.bel 12.0-CURRENT FreeBSD 12.0-CURRENT #0 
r322109M: Wed Aug  9 17:16:41 CEST 2017 
root at norquay.restart.bel:/usr/obj/usr/src/sys/NORQUAY  arm64

with my rpool on 2 USB disks connected to a self powered hub:

root at norquay ~]# zpool status
   pool: rpool
  state: ONLINE
   scan: scrub repaired 0 in 3h3m with 0 errors on Sat Jul 29 16:33:00 2017
config:

	NAME               STATE     READ WRITE CKSUM
	rpool              ONLINE       0     0     0
	  mirror-0         ONLINE       0     0     0
	    gpt/disk0-zfs  ONLINE       0     0     0
	    gpt/disk1-zfs  ONLINE       0     0     0

errors: No known data errors


BUT after a power off power on, this same kernel encounter the
  APs not started error.

If I boot with /boot/kernel.r320599 and then do a shutdown -r now
the r322109 boot without problem.

Henri