How to get a crosscompile toolchain for aarch64 for use in poudriere[?]

Fri Feb 10 15:03:06 UTC 2017

Hello,

I am gaining experience in building packages for aarch64 on an amd64 host using poudriere and it works (most of the time) quite well but it is really slow.

In order to accelerate the build I wanted to crosscompile within poudriere using the instructions provided in
http://phaq.phunsites.net/2015/10/11/freebsd-on-armv6-cross-compile-performance-optimization-for-poudriere/comment-page-1/
and getting it to work with aarch64.

The first problems arose when trying to build the xdev target:

> make XDEV=arm64 XDEV_ARCH=aarch64 xdev

The build fails and after doing some not very educated hacks to workaround the problems I get to the point where I do not have a clue on how to fix this. The error message looks like:

===> lib/libc++ (obj,all,install)
c++ -isystem //usr/aarch64-freebsd/usr/include -L//usr/aarch64-freebsd/usr/lib  --sysroot=//usr/aarch64-freebsd/ -B//usr/aarch64-freebsd/usr/libexec  -B//usr/aarch64-freebsd/usr/bin -B//usr/aarch64-freebsd/usr/lib  -O2 -pipe -isystem /usr/FBSD_src/src_head/contrib/libc++/include -isystem /usr/FBSD_src/src_head/contrib/libcxxrt -nostdinc++ -nostdlib -DLIBCXXRT -MD -MF.depend.algorithm.o -MTalgorithm.o -fstack-protector-strong -Qunused-arguments  -std=c++11 -Wno-c++11-extensions  -c /usr/FBSD_src/src_head/contrib/libc++/src/algorithm.cpp -o algorithm.o
In file included from /usr/FBSD_src/src_head/contrib/libc++/src/algorithm.cpp:11:
In file included from /usr/FBSD_src/src_head/contrib/libc++/include/random:1638:
/usr/FBSD_src/src_head/contrib/libc++/include/cmath:313:12: error: no member named 'signbit' in
      namespace 'std'
using std::signbit;
      ~~~~~^
... + some more errors of the same kind.

Hence I tried another way to get a crosscompile toolchain for use in poudriere. When crosscompiling on the host we use at the moment the stuff generated by the kernel-toolchain target of the src tree (+ the sysroot and + the aarch64-binutils from the ports as the kernel-toolchain does not build a crosscompile lld for aarch64). So I looked whether I would be able to import this setup in a poudriere jail. Therefore I made the kernel-toolchain build with the MAKEOBJDIRPREFIX set to the /usr/obj dir within the jail as shown in the above mentioned link. The contents of the $MAKEOBJDIRPREFIX/usr/src/tmp/ directory were copied to the /usr/aarch64-freebsd dir in the jail to have a seperated toolchain directory.

In the first try I did not add a cross-linker hoping the native lld in the jail would be able to do the work. Well it won't. After learning how to enter the poudriere jail and looking at the verbose output from the compilation I concluded that the compiler seemed to work but the linker failed. Next step was installing the aarch64-binutils from ports in the jail (in my /usr/aarch-freebsd cross-toolchain directory). Still no success as the linker complained about some

ELF interpreter /libexec/ld-elf.so.1 not found

error. Which I ascribed to the fact that the executables from the aarch64-binutls port are dynamically linked. Adding a

LDFLAGS+= -all-static

line to the binutils ports Makefile I was able to create a statically linked linker and installed it in my poudriere jail. Still no success though, as again similar errors occured like for the usage of the native linker (complaining about missing crt1.o files and missing lgcc_somethings).

To make the long story short, after comparing the verbose output of my compiling in the jails with that of a direct crosscompile I found out that it was able to correctly link some object files when providing absolute paths to the crt*.o files and correct library search path for the libgcc_something libraries. Which of course doesn't help when the linker is implicitly called by the compiler. Further research showed that I could crosscompile/link when providing the --sysroot=/ flag to overwrite the sysroot configuration from the build of the crosscompiler. Amending the lines

CFLAGS+=--sysroot=/
CPPFLAGS+=--sysroot=/  (Don't know if needed)
CXXFLAGS+=--sysroot=/
LDFLAGS+=--sysroot=/

to my poudriere make.conf file I finally was able to produce a working cross-compiled package for aarch64 in poudriere. And the gains are quite significant (around 21 minutes for building pkg package with qemu-user emulation vs 5 minutes using cross-compilation, native amd64 build in poudriere finishes in less than a minute).

Still the creation of the cross-compiling toolchain seems to be quite cumbersome. Hence advice on getting such a toolchain in a cleaner way (ideally probably a working build for the xdev target) are greatly appreciated.

________________________________

Follow HOB:

- HOB: http://www.hob.de/redirect/hob.html
- Xing: http://www.hob.de/redirect/xing.html
- LinkedIn: http://www.hob.de/redirect/linkedin.html
- HOBLink Mobile: http://www.hob.de/redirect/hoblinkmobile.html
- Facebook: http://www.hob.de/redirect/facebook.html
- Twitter: http://www.hob.de/redirect/twitter.html
- YouTube: http://www.hob.de/redirect/youtube.html
- E-Mail: http://www.hob.de/redirect/mail.html

HOB GmbH & Co. KG
Schwadermuehlstr. 3
D-90556 Cadolzburg

Geschaeftsfuehrung: Klaus Brandstaetter, Zoran Adamovic

AG Fuerth, HRA 5180
Steuer-Nr. 218/163/00107
USt-ID-Nr. DE 132747002

Komplementaerin HOB electronic Beteiligungs GmbH
AG Fuerth, HRB 3416