Using qemu to pre-populate crossbuilds

[Ian Lepore]

On Sun, 2019-03-03 at 16:25 -0600, Karl Denninger wrote:
> I have qemu-user-static loaded, which I use to build poudriere
> packages
> for the Pi2 (since that platform has no pre-builts available for 12-
> Stable.)
> 
> This brings up an interesting question, since qemu can obviously run
> (albeit slowly) native Arm code -- can I use it as a means to, for
> example, pre-run "pkg" -- or kldxref -- on a built image?
> 
> Devmatch, for example, wants a linker hints file, but there's no good
> way to build one, since the AMD64 system will refuse to do so as the
> kernel modules are of the wrong type.  If the new system is one that
> runs read-only on root then once running it can't build one either
> without manual intervention.
> 
> The obvious ("qemu-arm-static /mnt/usr/sbin/kldxref
> /mnt/boot/kernel/"),
> where "/mnt" is where I have the completed image's root directory
> mounted, fails with a complaint:
> 
> Unable to load interpreter
> 
> I'm probably missing something stupid, since obviously it can execute
> arm binaries or the poudriere jail wouldn't work, and it does.
> 
> binmiscctl appears to show the correct magic numbers are set to be
> able
> to run these; I assume the problem is with the shared libraries
> required
> since kldxref is dynamically-linked.
> 
> Any assistance is appreciated.
> 

You need to have a chroot or jail, and you need to do a bit of setup
for it. Let's say you've done an installworld and installkernel of your
crossbuilt rpi stuff to /rpi on the amd64 machine.  Do...

 sudo mount -t devfs devfs /rpi/dev
 sudo cp /usr/local/bin/qemu-arm-static /rpi/usr/local/bin
 sudo chroot /rpi service ldconfig onestart
 sudo chroot /rpi service kldxref onestart
 sudo umount /rpi/dev

Also, once you've done the ldconfig (which only needs to be done once
after a fresh build) you can just "sudo chroot /rpi" and you have an
interactive shell where you can do anything (that doesn't require rpi
hardware like gpios) as if you were on the rpi itself.  So you can pkg
install or whatever.

-- Ian