Unable to start 11.0-RELEASE aarch64 jail created by poudriere

Naram Qashat cyberbotx at cyberbotx.com
Fri Sep 1 21:58:19 UTC 2017


I have been attempting to make an 11.0-RELEASE jail for aarch64 to use 
with poudriere, with my world source being existing sources in /usr/src, 
but I am unable to start the resulting jail. I am currently using 
poudriere-devel 3.1.99.20170803 and use UFS for the file system (I do 
not use ZFS at all). I have qemu-user-static installed and can verify 
that there is an aarch64 entry in 'binmiscctl list'. The following are 
the steps I have taken:

1) Made sure that there is no existing jail named local_aarch64 (the 
name I was using for the jail) in poudriere.

2) Made sure to delete my /usr/obj/arm64.aarch64 directory.

3) Built the aarch64 world from /usr/src using 'make -j4 buildworld 
TARGET_ARCH=aarch64'.

4) Created the poudriere jail using 'poudriere jail -c -J 4 -j 
local_aarch64 -a arm64.aarch64 -f none -m src=/usr/src -x'. (I was told 
on IRC by RootWyrm that poudriere-devel and the -x flag when creating 
the jail as required for aarch64.)

5) Attempting to chroot into the resulting jail using 'chroot 
/usr/local/poudriere/jails/local_aarch64 /bin/sh' results in this:
chroot: /bin/sh: No such file or directory

6) Attempting to start the jail with poudriere using 'poudriere jail -s 
-j local_aarch64' results in this:
[00:00:00] Cross-building ports for arm64.aarch64 on amd64 requires QEMU
[00:00:00] Creating the reference jail... done
[00:00:36] Mounting system devices for local_aarch64-default
[00:00:36] Mounting ports/packages/distfiles
[00:00:36] Mounting packages from:
[00:00:36] Setting up native-xtools environment in jail... done
[00:00:36] Raising MAX_EXECUTION_TIME and NOHANG_TIME for QEMU
[00:00:36] Copying latest version of the emulator from: 
/usr/local/bin/qemu-aarch64-static
[00:00:36] Copying aarch64-binutils ld from 
/usr/local/aarch64-freebsd/bin/ld
[00:00:36] Appending to make.conf: 
/usr/local/etc/poudriere.d/local_aarch64-make.conf
/etc/resolv.conf -> 
/usr/local/poudriere/data/.m/local_aarch64-default/ref/etc/resolv.conf
[00:00:36] Starting jail local_aarch64-default
/etc/rc.subr: /bin/ps: not found
/etc/rc.subr: /bin/ps: not found
eval: /sbin/ldconfig: not found
eval: sysctl: not found
eval: sysctl: not found
eval: sysctl: not found
/usr/ports/Mk/Scripts/ports_env.sh: grep: not found
sh: /usr/bin/uname: not found
make: "/usr/ports/Mk/bsd.port.mk" line 1128: warning: "/usr/bin/uname 
-s" returned non-zero status
sh: /usr/bin/uname: not found
make: "/usr/ports/Mk/bsd.port.mk" line 1133: warning: "/usr/bin/uname 
-r" returned non-zero status
make: "/usr/ports/Mk/bsd.port.mk" line 1177: UNAME_r () and OSVERSION 
(1100122) do not agree on major version number.
jexec: execvp: ldconfig: No such file or directory

7) Attempting to run any executables created for the jail:
# /usr/local/poudriere/jails/local_aarch64/usr/bin/ldd
/usr/local/poudriere/jails/local_aarch64/usr/bin/ldd: Command not found.
# qemu-aarch64-static 
/usr/local/poudriere/jails/local_aarch64/usr/bin/ldd
Unable to load interpreter
# /usr/local/poudriere/jails/local_aarch64/rescue/ls /var/empty
/usr/local/poudriere/jails/local_aarch64/rescue/ls: Command not found.
# qemu-aarch64-static /usr/local/poudriere/jails/local_aarch64/rescue/ls 
/var/empty
#
(In other words, only running a statically linked executable with 
qemu64-aarch64-static works, all other ways do not work. All the files, 
when checked with the 'file' command, do say they are built for ARM 
aarch64 as opposed to the host's x86-64.)

As verification, this is within the output of 'binmiscctl list':
[...]
name: aarch64
interpreter: /usr/local/bin/qemu-aarch64-static
flags: ENABLED USE_MASK
magic size: 20
magic offset: 0
magic: 0x7f 0x45 0x4c 0x46  0x02 0x01 0x01 0x00  0x00 0x00 0x00 0x00
        0x00 0x00 0x00 0x00  0x02 0x00 0xb7 0x00
mask:  0xff 0xff 0xff 0xff  0xff 0xff 0xff 0x00  0xff 0xff 0xff 0xff
        0xff 0xff 0xff 0xff  0xfe 0xff 0xff 0xff
[...]

There were no errors in the buildworld nor in the jail creation via 
poudriere. Prior to RootWyrm's suggestion of using poudriere-devel and 
the -x flag for jail creation, I had been using poudriere 3.1.19 and 
creating the jail without the -x flag. In that scenario, I was able to 
successfully chroot into the jail's file system (step 5 above) and I was 
able to attempt bulk port building (technically step 6 above although I 
had not done that with the 3.1.19 scenario because it actually was 
building ports for a while), but building ports failed after a while 
with execvp errors (one of which was 'jexec: execvp: /usr/bin/limits: No 
such file or directory') and then finally resulted in all 4 builders 
getting stuck in the clean_pool status before I stopped the jail.

Thanks,
Naram Qashat


More information about the freebsd-ports mailing list