A investigative hack that makes (for example) head -r356529 boot and operate normally an RPi4B (finally!): protect all armstub8-gic.bin's loaded content from replacement by the kernel
Mark Millard
marklmi at yahoo.com
Thu Feb 13 18:27:57 UTC 2020
On 2020-Feb-13, at 09:55, Mark Millard <marklmi at yahoo.com> wrote:
On 2020-Feb-13, at 09:36, Mark Millard <marklmi at yahoo.com> wrote:
>
>> On 2020-Feb-13, at 08:50, Mark Millard <marklmi at yahoo.com> wrote:
>>
>>> On 2020-Feb-13, at 07:22, Kyle Evans <kevans at freebsd.org> wrote:
>>>
>>>> On Thu, Feb 13, 2020 at 9:05 AM Ralf Wenk <iz-rpi03 at hs-karlsruhe.de> wrote:
>>>>>
>>>>> On 2020-02-13 at 15:26 +0100 Ralf Wenk wrote:
>>>>>> On 2020-02-13 at 7:49 -0600 Kyle Evans wrote:
>>>>>>> On Thu, Feb 13, 2020 at 7:43 AM Ralf Wenk <iz-rpi03 at hs-karlsruhe.de> wrote:
>>>>>>>>
>>>>>>>> On 2020-02-12 at 18:00 -0800 Mark Millard wrote via freebsd-arm:
>>>>>>>>> [...]
>>>>>>>>>
>>>>>>>>> # svnlite diff /usr/src/sys/dev/fdt/fdt_common.c
>>>>>>>>> Index: /usr/src/sys/dev/fdt/fdt_common.c
>>>>>>>>> ===================================================================
>>>>>>>>> --- /usr/src/sys/dev/fdt/fdt_common.c (revision 357529)
>>>>>>>>> +++ /usr/src/sys/dev/fdt/fdt_common.c (working copy)
>>>>>>>>> @@ -485,7 +485,18 @@
>>>>>>>>>
>>>>>>>>> tuples = res_len / tuple_size;
>>>>>>>>> reservep = (pcell_t *)&reserve;
>>>>>>>>> +#ifdef __aarch64__
>>>>>>>>> + //HACK!!!
>>>>>>>>> + // Reserve the first few pages, for example to
>>>>>>>>> + // preserve armstub8-gic.bin or armstub.bin
>>>>>>>>> + // content.
>>>>>>>>> + mr[0].mr_start= 0;
>>>>>>>>> + mr[0].mr_size= 2*4096;
>>>>>>>>> + tuples++;
>>>>>>>>> + for (i = 1; i < tuples; i++) {
>>>>>>>>> +#else
>>>>>>>>> for (i = 0; i < tuples; i++) {
>>>>>>>>> +#endif
>>>>>>>>>
>>>>>>>>> rv = fdt_data_to_res(reservep, addr_cells, size_cells,
>>>>>>>>> (u_long *)&mr[i].mr_start, (u_long *)&mr[i].mr_size);
>>>>>>>>> @@ -512,6 +523,11 @@
>>>>>>>>>
>>>>>>>>> root = OF_finddevice("/reserved-memory");
>>>>>>>>> if (root == -1) {
>>>>>>>>> + // Fail over to checking for and handling memreserve,
>>>>>>>>> + // such as for a RPi4B.
>>>>>>>>> + if (0 == fdt_get_reserved_regions(reserved,mreserved))
>>>>>>>>> + return (0);
>>>>>>>>> +
>>>>>>>>> return (ENXIO);
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>
>>>>>>>> I can confirm that with your patch(es) my RPi3 does not freeze any more
>>>>>>>> when loading mac_ntpd.ko. The patches are applied against r357853M.
>>>>>
>>>>> An reboot is working again too.
>>>>>
>>>>>>> Have you tested the RPi3 with just this second hunk of patch to
>>>>>>> fallover to memreserve, or is the first hunk definitely required as
>>>>>>> well?
>>>>>>
>>>>>> Good question. I tested both hunks together.
>>>>>> Will try what happens when just applying the second and report back.
>>>>>
>>>>> Here it is:
>>>>> Without the first hunk the system freezes again when loading mac_ntpd.ko.
>>>>> Also the CPU information during boot for CPUs 1 to 3 looks strange again.
>>>>>
>>>>
>>>> Yeah- I see it now; both armstubs are about 5k. I've raised an
>>>> issue[0] with upstream for armstub/rpi bits to work out the proper
>>>> solution, because I don't necessarily want to commit the workaround.
>>>> I'll throw up the second hunk on phabricator for review by #arm/#arm64
>>>> folks, because that seems to me the proper fallback.
>>>>
>>>> I also discovered some issues when trying to read /memreserve/ with
>>>> our dtc and filed a PR[1] to fix those.
>>>>
>>>> Thanks,
>>>>
>>>> Kyle Evans
>>>>
>>>> [0] https://github.com/raspberrypi/tools/issues/107
>>>> [1] https://github.com/davidchisnall/dtc/pull/59
>>>
>>> The DTB information below is from:
>>>
>>> U-Boot> fdt addr 0x7ef2000
>>> U-Boot> fdt print /
>>>
>>> on a RPi4B 4 GiByte.
>>>
>>> On at least the RPi4B memreserve is not what causes
>>> the first page to be excluded:
>>>
>>> memreserve = <0x3b400000 0x04c00000>;
>>>
>>> Nor is memory at 0 the cause:
>>>
>>> memory at 0 {
>>> device_type = "memory";
>>> reg = <0x00000000 0x00000000 0x3b400000 0x00000000 0x40000000 0xbc000000>;
>>> };
>>>
>>> (That also skips the memreserve area.)
>>>
>>> I do not find anything in the DTB that indicates
>>> to exclude the first page.
>>>
>>> My hypothesis is that the FreeBSD code excludes
>>> the page based on some less obvious relationship
>>> that I'm not identifying.
>>>
>>> There is the cpu-rlease-addr information that seems
>>> to refer to some 1st memory page content:
>>>
>>> cpus {
>>> #address-cells = <0x00000001>;
>>> #size-cells = <0x00000000>;
>>> enable-method = "brcm,bcm2836-smp";
>>> phandle = <0x000000be>;
>>> cpu at 0 {
>>> device_type = "cpu";
>>> compatible = "arm,cortex-a72";
>>> reg = <0x00000000>;
>>> enable-method = "spin-table";
>>> cpu-release-addr = <0x00000000 0x000000d8>;
>>> phandle = <0x0000001d>;
>>> };
>>> cpu at 1 {
>>> device_type = "cpu";
>>> compatible = "arm,cortex-a72";
>>> reg = <0x00000001>;
>>> enable-method = "spin-table";
>>> cpu-release-addr = <0x00000000 0x000000e0>;
>>> phandle = <0x0000001e>;
>>> };
>>> cpu at 2 {
>>> device_type = "cpu";
>>> compatible = "arm,cortex-a72";
>>> reg = <0x00000002>;
>>> enable-method = "spin-table";
>>> cpu-release-addr = <0x00000000 0x000000e8>;
>>> phandle = <0x0000001f>;
>>> };
>>> cpu at 3 {
>>> device_type = "cpu";
>>> compatible = "arm,cortex-a72";
>>> reg = <0x00000003>;
>>> enable-method = "spin-table";
>>> cpu-release-addr = <0x00000000 0x000000f0>;
>>> phandle = <0x00000020>;
>>> };
>>> };
>>
>>
>>
>>
>> Looking at the code there is:
>>
>> /* Load the physical memory ranges */
>> efihdr = (struct efi_map_header *)preload_search_info(kmdp,
>> MODINFO_METADATA | MODINFOMD_EFI_MAP);
>> if (efihdr != NULL)
>> add_efi_map_entries(efihdr);
>> #ifdef FDT
>> else {
>> /* Grab physical memory regions information from device tree. */
>> if (fdt_get_mem_regions(mem_regions, &mem_regions_sz,
>> NULL) != 0)
>> panic("Cannot get physical memory regions");
>> arm_physmem_hardware_regions(mem_regions, mem_regions_sz);
>> }
>> if (fdt_get_reserved_mem(mem_regions, &mem_regions_sz) == 0)
>> arm_physmem_exclude_regions(mem_regions, mem_regions_sz,
>> EXFLAG_NODUMP | EXFLAG_NOALLOC);
>> #endif
>>
>> /* Exclude the EFI framebuffer from our view of physical memory. */
>> efifb = (struct efi_fb *)preload_search_info(kmdp,
>> MODINFO_METADATA | MODINFOMD_EFI_FB);
>> if (efifb != NULL)
>> arm_physmem_exclude_region(efifb->fb_addr, efifb->fb_size,
>> EXFLAG_NOALLOC);
>> . . .
>> if (boothowto & RB_VERBOSE) {
>> print_efi_map_entries(efihdr);
>> arm_physmem_print_tables();
>> }
>>
>>
>> It looks to me like the boot -v text:
>>
>> Type Physical Virtual #Pages Attr
>> Reserved 000000000000 0 00000001 WB
>> ConventionalMemory 000000001000 1000 00007ef1 WB
>> BootServicesData 000007ef2000 7ef2000 0000001c WB
>> ConventionalMemory 000007f0e000 7f0e000 00029f93 WB
>> BootServicesData 000031ea1000 31ea1000 00000001 WB
>> LoaderData 000031ea2000 31ea2000 00008001 WB
>> LoaderCode 000039ea3000 39ea3000 000000a6 WB
>> Reserved 000039f49000 39f49000 00000007 WB
>> BootServicesData 000039f50000 39f50000 00000001 WB
>> Reserved 000039f51000 39f51000 00000002 WB
>> RuntimeServicesData 000039f53000 39f53000 00000001 WB RUNTIME
>> Reserved 000039f54000 39f54000 00000001 WB
>> BootServicesData 000039f55000 39f55000 00000002 WB
>> RuntimeServicesData 000039f57000 39f57000 00000001 WB RUNTIME
>> LoaderData 000039f58000 39f58000 00001408 WB
>> RuntimeServicesCode 00003b360000 3b360000 00000010 WB RUNTIME
>> LoaderData 00003b370000 3b370000 00000090 WB
>> BootServicesData 000040000000 40000000 000bc000 WB
>> MemoryMappedIO 0000fe100000 fe100000 00000001 RUNTIME
>>
>> is from print_efi_map_entries via the efihdr instead
>> of from the FreeBSD FDT code and the DTB.
>>
>> So is it u-boot that provides the efihdr for which
>> add_efi_map_entries generated those regions?
>>
>> That might explain why I do not find matching DTB
>> material for all of it.
>
>
> Looks like the efi memory map traces back to the loader
> and its use of GetMemoryMap (as far as FreeBSD goes):
>
> # grep -r "GetMemoryMap" /usr/src/sys/ | more
> /usr/src/sys/amd64/amd64/machdep.c: * Memory map data provided by UEFI via the GetMemoryMap
> /usr/src/sys/arm64/arm64/machdep.c: * Memory map data provided by UEFI via the GetMemoryMap
> /usr/src/sys/arm/arm/machdep_boot.c: * Memory map data provided by UEFI via the GetMemoryMap
> /usr/src/sys/contrib/edk2/Include/Uefi/UefiSpec.h: EFI_GET_MEMORY_MAP GetMemoryMap;
>
> # grep -r "GetMemoryMap" /usr/src/stand/ | more
> /usr/src/stand/efi/loader/copy.c: status = BS->GetMemoryMap(&sz, map, &key, &dsz, &dver);
> /usr/src/stand/efi/loader/main.c: status = BS->GetMemoryMap(&sz, 0, &key, &dsz, &dver);
> /usr/src/stand/efi/loader/main.c: status = BS->GetMemoryMap(&sz, map, &key, &dsz, &dver);
> /usr/src/stand/efi/loader/bootinfo.c: status = BS->GetMemoryMap(&sz, mm, &efi_mapkey, &dsz, &mmver);
> /usr/src/stand/efi/loader/bootinfo.c: printf("%s: GetMemoryMap error %lu\n", __func__,
> /usr/src/stand/efi/include/efiapi.h: EFI_GET_MEMORY_MAP GetMemoryMap;
Here is the memmap from the loader that is just based on EFI's
GetMemoryMap from what I can tell from a wuick look:
Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel] in 8 seconds...
Type '?' for a list of commands, 'help' for more detailed help.
OK memmap
Type Physical Virtual #Pages Attr
Reserved 000000000000 000000000000 00000001 WB
ConventionalMemory 000000001000 000000001000 00007ef1 WB
BootServicesData 000007ef2000 000007ef2000 0000001c WB
ConventionalMemory 000007f0e000 000007f0e000 00029f93 WB
BootServicesData 000031ea1000 000031ea1000 00000001 WB
LoaderData 000031ea2000 000031ea2000 00004000 WB
BootServicesData 000035ea2000 000035ea2000 00000001 WB
LoaderData 000035ea3000 000035ea3000 00004000 WB
LoaderCode 000039ea3000 000039ea3000 000000a6 WB
Reserved 000039f49000 000039f49000 00000007 WB
BootServicesData 000039f50000 000039f50000 00000001 WB
Reserved 000039f51000 000039f51000 00000002 WB
RuntimeServicesData 000039f53000 000039f53000 00000001 WB
Reserved 000039f54000 000039f54000 00000001 WB
BootServicesData 000039f55000 000039f55000 00000002 WB
RuntimeServicesData 000039f57000 000039f57000 00000001 WB
LoaderData 000039f58000 000039f58000 00001408 WB
RuntimeServicesCode 00003b360000 00003b360000 00000010 WB
LoaderData 00003b370000 00003b370000 00000090 WB
BootServicesData 000040000000 000040000000 000bc000 WB
MemoryMappedIO 0000fe100000 0000fe100000 00000001
OK
So at the loader time frame, the 2nd page of memory is not
Reserved (in EFI terms).
It would appear to me that avoiding messing up the
armstub8*.bin content should apply this early as well.
So it may be sysutils/u-boot-rpi{3,4} that needs to
arrange sufficient room to prevent messing up such.
(Unless armstub8*.bin can adjust something that
u-boot's EFI interface is based on for that initial
"Reserved" area?)
===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
More information about the freebsd-arm
mailing list