Starting to cross-compile non-base software for MIPS - what it's like

[Warner Losh]

> On Jun 14, 2015, at 1:13 PM, Adrian Chadd <adrian at freebsd.org> wrote:
> 
> Hi!
> 
> I've been hacking at cross compiling things on FreeBSD for a while,
> and I have made some progress. This email is less of a "how you can do
> it" and more of a "these are the roadblocks that prevent it from
> working out of the box."
> 
> I'm using gcc-4.9.2 from ports - there's a cross compiler for
> mips/mips64, and there's mips-binutils/mips64-binutils. I'm using
> those to compile world and kernel. The kernel works; world requires
> some patching to compile. I'll write a post when I have a complete
> system booting/running from gcc-4.9.2.
> 
> My test application is dropbear. Ie, I'm cross-compiling dropbear for
> mips from freebsd/i386.
> 
> * There are some issues in libgcc - I have a patch that I've emailed
> out for review. I'll do some more testing and commit it next week when
> I know it works.
> * There are some environment variables that need to be set before I
> compile dropbear. Here's what it looks like:
> 
> CPU_ARGS="-march=mips32 -msoft-float -Wa,-msoft-float"
> INCS="-I/home/adrian/work/freebsd/head-embedded-2/root/mips/usr/include"
> 
> export CROSS_COMPILE=mips-portbld-freebsd11.0
> export CC=${CROSS_COMPILE}-gcc
> export CXX=${CROSS_COMPILE}-g++
> #export LD=${CROSS_COMPILE}-ld
> #export AR=${CROSS_COMPILE}-ar
> #export RANLIB=${CROSS_COMPILE}-ranlib
> #export STRIP=${CROSS_COMPILE}-strip
> export LD=mips-freebsd-l
> export AR=mips-freebsd-ar
> export RANLIB=mips-freebsd-ranlib
> export STRIP=mips-freebsd-strip
> export SYSROOT=/home/adrian/work/freebsd/head-embedded-2/root/mips/
> export CFLAGS="--sysroot=$SYSROOT ${CPU_ARGS} ${INCS} -O"
> export CXXFLAGS="--sysroot=$SYSROOT ${CPU_ARGS} ${INCS} -O"
> export CPPFLAGS="--sysroot=$SYSROOT ${CPU_ARGS} ${INCS} -O"
> export LDFLAGS=--sysroot=$SYSROOT
> 
> # now run:
> # /configure --host=mips-portbld-freebsd11.0
> # gmake
> 
> ... then I hit a problem with how we populate links in /usr/lib and
> such. TL;DR - the symlinks we use are not relative paths, they're
> absolute paths - and this makes cross building using sysroot /
> explicit library paths fail to work right. Sigh. In particular, I
> needed to manually undo the libgcc symlink for it to compile.
> compilation complained about other libraries, but the resulting
> binaries did run.

That’s one of about 6 reasons that what we produce isn’t a
sysroot. We need to create a sysroot target that’s very close
to the xdev targets with all the compiler bits stripped out. We’re
close though. I have about 1/2 of the patch needed to remove
xdev entirely and replace it with sysroot.

You shouldn’t need the INCS args if you have a proper sysroot.
And the CPU_ARGS is an indication we’re building our port
incorrectly. This is the same feedback I gave on the kernel
make changes you suggested.

> * yes, I tested the dropbear/dbclient binaries in a mips qemu VM (not
> qemu-mips-user, but a full mips MALTA VM.) Both worked fine.
> 
> * the sysroot setup didn't correctly set the include path stuff right
> - I need to chase that down. Ie, it was still searching in
> /usr/include/ rather than ${SYSROOT}/usr/include/ . The library code
> obeyed the library searching, except for that hardlink.

That sounds like a bug in the default path of our gcc port.

> So, the very basics of "how can we cross compile things to run inside
> a freebsd-mips machine" work fine. It's amusing and annoying that the
> sysroot thing popped up. I do wonder if this class of problem keeps
> popping up when people do builds from one freebsd version to another;
> but I currently have no clear idea how to automate detecting this.

For one version to another we have a very controlled and constrained
environment. Inside of that, there are enough people building in purposely
broken environments to detect regressions. Once you involve configure
and foreign software things become rather unconstrained...

> On the plus side, it did work.

I’ve had hacks like this for “simple” arts for years. It’s why I wrote
xdev in the first place. I had it working in the FreeBSD 6 time frame
for cross building some ports. But once you get past a certain
class of very well behaved ports, it goes to hell in a hurry.

> So:
> 
> * I'll try to wrangle juli/bsdimp to review the libgcc change I have
> so userland compiles using gcc-4.9 with warnings disabled.

I’ll be happy to take a look.

> * I'll try to wrangle juli/bsdimp to review the kernel changes needed
> to compile up the kernel with gcc-4.9.

I’ve already reviewed that one. gcc isn’t being compiled correctly
and that need to be fixed. This is the forth time I’ve given this
feedback.

> * I'd like some help / comments on what we can do about the absolute
> symlinks used for various things inside an installed root. Ideally
> we'd have an option that would let us populate only relative links -
> that'd instantly make this problem go away.

All installworld-ish targets should use relative path symbolic links.
The fact they are using absolute paths is a bug. In the FreeBSD 6
time frame, I fought this battle at work and it wasn’t hard to fix. I
no longer have access to those sources, and I honestly can’t
recall how I fixed it (it may have been in a post-processing
step).

> * I'll go hit up bapt and some netbsd friends for some help with
> expected sysroot behaviour on gcc/binutils. It's possible there are
> still bugs with the dynamic sysroot (ie, when you don't compile your
> toolchain with an explicit sysroot to something other than '/') that
> need addressing.

bapt and I have been talking about this for a while. We’ve both been
busy doing other things so haven’t focused on it other than in off
moments.

> Then I'll worry about starting to convert a few ports to be native
> cross building.

You should talk to bapt about the rather extensive work he’s done
in this area, which is why I’ve looked at transitioning the xdev
support to just sysroot support.

Warner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-embedded/attachments/20150615/afa55ba5/attachment.sig>