Total confusion over toolchain/xdev behavior

Mon Jul 21 20:06:07 UTC 2014

On 18 Jul 2014, at 19:55, Sean Bruno <sbruno at ignoranthack.me> wrote:
> On Sun, 2014-07-06 at 16:07 -0700, Sean Bruno wrote:
>> Objective:  install an xcompile toolchain into a jail for use by
>> poudriere during arm/mips/sparc/power ports pkgs builds.  The build
>> should be possible from a non-root user.
>> 
>> As far as I can tell, the xdev target is completely busted for
>> non-clang
>> arch's right now as it tries to build clang no matter what I do.  Its
>> missing some pretty key documentation to making it work correctly, so
>> a
>> lot of my attempts have been "guess and check" with verbose make.
>> 
>> 
> 
> Quite a bit of success with one blocking failure.  Thanks to Warner,
> Baptiste and Dimitry for plugging along through some of my ranting (on
> bcc here).
> 
> The xdev target can be used to produce a compiler toolchain that can be
> used to build ports.  However, final linking seems to fail and makes it
> impossible to use for building many, many ports.
> 
> Regardless of the cross compile bits, simply using the xdev target for
> building ports natively on amd64 for amd64 manifests this issue, this
> leads me to believe that xdev is *not* building the tool chain
> correctly.  There is no chroot/jail/emualtion involved in this test
> case.
> 
> My test case:
> 
> 1.  build amd64 xdev tool chain:
> make -s -j4 xdev XDEV=amd64 XDEV_ARCH=amd64

For me, this failed due to a problem with xdev-install trying to install
include files into /usr/amd64-freebsd/usr/include/atf-c, which did not
exist.  I had to add the following part to Makefile.inc1 to make it work:

Index: Makefile.inc1
===================================================================

--- Makefile.inc1	(revision 268920)
+++ Makefile.inc1	(working copy)
@@ -1942,6 +1942,10 @@
 	    -p ${XDDESTDIR}/usr >/dev/null
 	mtree -deU -f ${.CURDIR}/etc/mtree/BSD.include.dist \
 	    -p ${XDDESTDIR}/usr/include >/dev/null
+.if ${MK_TESTS} != "no"
+	mtree -deU -f ${.CURDIR}/etc/mtree/BSD.tests.dist \
+	    -p ${XDDESTDIR}/usr >/dev/null
+.endif

 .ORDER: xdev-build _xi-mtree _xi-cross-tools _xi-includes _xi-libraries
 xdev-install: xdev-build _xi-mtree _xi-cross-tools _xi-includes _xi-libraries


> 2.  modify make.conf to use this toolchain:
> CC=/usr/amd64-freebsd/usr/bin/cc
> CPP=/usr/amd64-freebsd/usr/bin/cpp
> CXX=/usr/amd64-freebsd/usr/bin/cc++
> AS=/usr/amd64-freebsd/usr/bin/as
> NM=/usr/amd64-freebsd/usr/bin/nm
> RANLIB=/usr/amd64-freebsd/usr/bin/ranlib
> LD=/usr/amd64-freebsd/usr/bin/ld
> OBJCOPY=/usr/amd64-freebsd/usr/bin/objcopy
> SIZE=/usr/amd64-freebsd/usr/bin/llvm-size
> STRIPBIN=/usr/amd64-freebsd/usr/bin/strip
> 
> 3.  attempt to build ports-mgmt/pkg

In my case, this would fail much earlier, because it would try to use
the native nm and ar, not the ones from /usr/amd64-freebsd/usr/bin.

I had to specify X_BUILD_FOR=amd64-freebsd in make.conf, and modify
bsd.port.mk as follows:

Index: Mk/bsd.port.mk
===================================================================
--- Mk/bsd.port.mk      (revision 361353)
+++ Mk/bsd.port.mk      (working copy)
@@ -1157,13 +1157,16 @@
 .else
 X_SYSROOT=     /usr/${X_BUILD_FOR}
 .endif
-CC=            ${X_SYSROOT}/usr/bin/cc
-CXX=   ${X_SYSROOT}/usr/bin/c++
-PKG_ENV+=      ABI_FILE=${X_SYSROOT}/usr/lib/crt1.o
-NM=            ${X_BUILD_FOR}-nm
-STRIP_CMD=     ${X_BUILD_FOR}-strip
-MAKE_ENV+=     NM=${NM} STRIPBIN=${X_BUILD_FOR}-strip PKG_CONFIG_SYSROOT_DIR="${X_SYSROOT}"
-CONFIGURE_ENV+=        PKG_CONFIG_SYSROOT_DIR="${X_SYSROOT}"
+.for t in ADDR2LINE AR AS CC CPP CPPFILT GPROF LD NM OBJCOPY OBJDUMP RANLIB READELF SIZE STRINGS
+${t}=                  ${X_SYSROOT}/usr/bin/${t:C/PP/++/:tl}
+CONFIGURE_ENV+=        ${t}="${${t}}"
+MAKE_ENV+=             ${t}="${${t}}"
+.endfor
+CXX=                   ${X_SYSROOT}/usr/bin/c++
+STRIP_CMD=             ${X_SYSROOT}/usr/bin/strip
+CONFIGURE_ENV+=        CXX=${CXX} STRIPBIN=${STRIP_CMD} PKG_CONFIG_SYSROOT_DIR="${X_SYSROOT}"
+MAKE_ENV+=             CXX=${CXX} STRIPBIN=${STRIP_CMD} PKG_CONFIG_SYSROOT_DIR="${X_SYSROOT}"
+PKG_ENV+=              ABI_FILE=${X_SYSROOT}/usr/lib/crt1.o
 # only bmake support the below
 STRIPBIN=      ${STRIP_CMD}
 .export.env STRIPBIN

This to ensure that all cross-tools are passed as environment variables
to configure and make.


> *NOTE* if you add -lbsdxml to CFLAGS this will not happen.  Other
> packages manfiest similar issues, with different libs.

After quite a bit of debugging and instrumenting ld, it turns out this
is because the "native" ld has access to a populated hints file
(/var/run/ld-elf.so.hints), while the "cross" ld in /usr/amd64-freebsd
does not.

This can be seen as a bug, or at least inconsistent behavior on the part
of ld.  The code path followed in each case is subtly different.  I'm
still investigating what upstream did with it.

For now, the way of least resistance would seem to be building ldconfig
as part of xdev, and running it post-install.  However, it turns out the
hints file path is rather hardcoded in ldconfig, so to have that work,
it should be made configurable at build time first.

That said, the behavior of the "cross" ld seems *more* correct though,
in my opinion: our choice (or at least the intention of it) was to stop
pulling in DT_NEEDED libraries automatically, for compatibility with
newer upstream binutils.

E.g. adding just -larchive to a program that uses it should fail with
several missing symbols, and you should manually specify the full
dependency list:

-larchive -lz -bz2 -llzma -lbsdxml -lcrypto

This is also what the latest ld 2.24 requires.

-Dimitry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 203 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-arch/attachments/20140721/85ec0c39/attachment.sig>
