Meta mode toolchain bootstrapping [was Re: FreeBSD targets/ out-of-date]

[Bryan Drewery]

On 9/3/2015 11:46 AM, Simon J. Gerraty wrote:
> Anyway, bootstrap-toolchain leverages src/Makefile.inc1 to build an
> initial set of tools.  It then attempts to use that to build toolchain
> for MACHINE=host which is currently failing because in-tree libelf and
> libdwarf are needed and libelf needs sys/sys/elf_common.h but but that
> doesn't currently get staged for host (we try to minimize what we put in
> host stage tree).

Using the special hack you came up with for lib/libelf, and a lot of
other fixes for replacing binutils with elftoolchain, I've managed to
get the targets/pseduo/bootstrap-tools build working again. I will
commit this likely today or tomorrow.

However...

> 
> It may be better to simply skip targets/pseudo/toolchain.host
> that will work so long as we set TOOLSDIR to point to the stuff built by
> Makefile.inc1 
> 
> Depending on where we are with external toolchian support that would
> make sense - might be able to skip bootstrap-toolchain completely
> which would be nice.

I think this is more right because there's many more people thinking
about, testing daily, and maintaining the bootstrap logic in
Makefile.inc1. The way targets/pseudo/bootstrap-tools works currently
lead to bitrot and breakage with the elftoolchain replacement. It will
very easily happen again.

Right now the targets/pseduo/bootstrap-tools target builds
legacy,build-tools,cross-tools from Makefile.inc1 for everything but the
compiler and toolchain, and then building the toolchain itself. There's
a subtle problem with this in that we end up building clang in
targets/pseudo/toolchain with -nostdinc (from local.init.mk) which leads
to a missing header (which actually is staged, just lacks a -I or
--sysroot to be captured). But really there's no reason to do this work
since Makefile.inc1 does it fine and is maintained well to handle it.

There's another subtle thing here, because we set CC=HOST_CC=/usr/bin/cc
when calling cross-tools from the pseudo/targets/bootstrap-tools build,
it invokes the external compiler support in Makefile.inc1 which avoids
building the bootstrap clang and some of the toolchain, leading us to
have to do it from pseudo/targets/toolchain. I actually extended this
logic to pass HOST_AS to avoid *more* redundant toolchain building in
Makefile.inc1 from our own targets/pseduo/toolchain, before realizing this.

What I plan to do is make pseudo/targets/bootstrap-tools always build
all of cross-tools,etc, from Makefile.inc1 respecting its own internal
logic for when to bootstrap the compiler (without us telling it to skip
the bootstrap compiler) and then add pseudo/targets/bootstrap-tools as a
global DIRDEP. Given the use of cookies, this will only be done once per
meta build, but it at least won't be a manual step anymore. For the
record, I do plan to extend .MAKE.MODE=meta to buildworld and its
bootstrapping as well, which will give a lot of incremental build
improvements to it.

This means that the meta build will then default to bootstrapping the
compiler, which we really must do to build the source tree reliably.
There's really no reason the meta build should default to not
bootstrapping the compiler. Setting WITHOUT_CROSS_COMPILER or
WITHOUT_CLANG_BOOTSTRAP will avoid this and use just the host compiler
as the meta build does now. So there's no real problem for those people
who want to skip the clang bootstrap.

Then, I will work on the project of "building clang less" which will
avoid building the bootstrap compiler for both the meta mode build and
the buildworld build (and universe eventually) if /usr/bin/cc satisfies
the needs (can cross compile the TARGET and is "new enough" compared to
the version we want). This is not trivial but I think it is possible and
it will make everyone happy!

Related tidbit, using WITHOUT_CROSS_COMPILER (or WITHOUT_*_BOOTSTRAP)
with buildworld leads to a broken build currently since it is the user
basically asking to use their default external toolchain of /usr/bin/*,
but the logic does not kick in to use --sysroot against WORLDTMP. I plan
to fix this soon as well.

-- 
Regards,
Bryan Drewery

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-toolchain/attachments/20151112/6c29c8a5/attachment.bin>