Raspberry Pi 4 boot hangs in sched_idletd.

[Robert Crowston]

I need a bit of a hand with this. I've been working on getting FreeBSD 13.0-Current up and running on my pi4. I'm using the GENERIC configuration for now. I have a JTAG hardware debugger so I've got a pretty good introspection on what's going on at the detail level, but I'm missing some of the bigger picture.

The first problem is, this board has two interrupt controllers on the FDT; the bcm2836-l1-intc local interrupt controller (local_intc) that was present on the RPi3, and also a new gic400. Both the drivers call intr_pic_claim_root(), which causes a panic. If I remove the gic400 from the FDT, very little of the hardware enumeration succeeds and the kernel panics because there is no event timer. If I remove the local_intc, a few devices, including the BCM2835 DMA controller, the SD card controllers, and the GPIO drivers fail to start, but the rest of the hardware enumeration finishes. I don't know enough about the hardware topology to figure out which one is the real root and fix the drivers accordingly.

So, without local_intc, we get another problem. Late in the boot sequence, the idle thread gets swapped in, and there are still threads on the sleep queue, including thread0, but no other threads get put on the run queue, and the Pi loops through sched_idletd thread forever instead of finishing the boot.

By poking things in the debugger I can progress as far as vfs_mountroot_wait(), and I think there is a race condition because the behaviour is different if I step through manually without changing anything, but whatever happens, eventually I end up stuck in the sched_idletd loop.

I think perhaps event timer interrupts are not being delivered so the sleep queue never gets moved to the run queue?, but I'm guessing here.

More detail:

So far I'm in mi_startup(), and about ~1150 out of ~1200 SYSINIT tasks are done. Eventually we get to initialize the usbus. One of its worker threads calls _sleep():

Breakpoint 6, _sleep (ident=0xffff000000c5eb28 <pause_wchan>, lock=0x0, priority=0,
    wmesg=0xffff000000775d55 "USBWAIT", sbt=47244637, pr=0, flags=256)
    at /skeleton/root/sandbox/src/sys/kern/kern_synch.c:139
139             td = curthread;
(gdb) bt
#0  _sleep (ident=0xffff000000c5eb28 <pause_wchan>, lock=0x0, priority=0,
    wmesg=0xffff000000775d55 "USBWAIT", sbt=47244637, pr=0, flags=256)
    at /skeleton/root/sandbox/src/sys/kern/kern_synch.c:139
#1  0xffff00000029db08 in usbd_req_set_address (udev=0xfffffd0000d23000, mtx=0x0,
    addr=<optimized out>) at /skeleton/root/sandbox/src/sys/dev/usb/usb_request.c:1580
#2  0xffff00000028d05c in usb_alloc_device (parent_dev=<optimized out>, bus=0xffff0000405fe000,
    parent_hub=0x0, depth=<optimized out>, port_index=<optimized out>, port_no=<optimized out>,
    speed=USB_SPEED_HIGH, mode=<optimized out>)
    at /skeleton/root/sandbox/src/sys/dev/usb/usb_device.c:1824
#3  0xffff000000281524 in usb_bus_attach (pm=<optimized out>)
    at /skeleton/root/sandbox/src/sys/dev/usb/controller/usb_controller.c:767
#4  0xffff00000029b750 in usb_process (arg=0xffff0000405fe130)
    at /skeleton/root/sandbox/src/sys/dev/usb/usb_process.c:178
#5  0xffff0000003b78c0 in fork_exit (callout=0xffff00000029b65c <usb_process>,
    arg=0xffff0000405fe130, frame=0xffff00004024cba0)
    at /skeleton/root/sandbox/src/sys/kern/kern_fork.c:1056
#6  0xffff00000071fe88 in fork_trampoline ()
    at /skeleton/root/sandbox/src/sys/arm64/arm64/swtch.S:214

_sleep() swaps the usb_process thread off the CPU. If I step through, another few threads are put on the CPU, including taskqueue_thread_loop, soaio_kproc_loop, random_kthread. Sometimes I also see the interrupt event thread, but not always. Sometimes if I step, I'll find thread0 does get back on the CPU, in which case we can progress as far as release_aps(), but then the other CPUs get stuck in sched_idletd and eventually CPU0 will hang in smp_rendezvous() waiting for the other CPUs.

Does anyone have any ideas what I should investigate next?

Boot log attached.


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: rpi4 boot log GENERIC 2019-07-21T1440.txt
URL: <http://lists.freebsd.org/pipermail/freebsd-arm/attachments/20190721/be5fc14d/attachment.txt>