Building a Linuxulator userland from source

Hi all,

for the last two weeks, I've been working on a spike in ports which now
reached a state where I want to show it to and discuss it with fellow
ports hackers.

First, a link to my feature branch (warning, will be rebased every now
and then):
<https://github.com/Zirias/zfbsd-ports/commits/linux>

The goal is to create a replacement for the now antiquated linux-c7
userland. While the classic approach would be to find another Linux
distribution that's not too much of a moving target and start
"repackaging" that, I want to try something different: Build the
required packages from source.

** Why

It will be quite some work to do this, I'm not really sure about it yet
(and how it would compare to the repackaging approach), so feasibility
is yet to be decided. But I hope to get at least these two advantages:

- Provide the newest GNU libs (glibc, libstdc++, ...) built against
  exactly the Linux version emulated by the FreeBSD version this will
  run on. This should make it possible to run a lot more Linux binaries
  without relying on e.g. Linux jails.
- When binaries don't work for missing Linux libraries, make it somewhat
  easy to add them, maybe based on already existing FreeBSD ports.

** State

I just reached a state where I can build a working Linux-native GNU
toolchain (binutils, glibc, gcc) for C and C++ on aarch64, amd64 and
i386. From here on, it should be simpler, there are already two ports in
my branch (archivers/linux-bzip2 and archivers/linux-xz) using that
native toolchain for building.

** How

The native toolchain is built by a cross toolchain, the packages for
this cross-toolchain are prefixed "lxcross-". For building this cross
toolchain, bootstrapping versions of binutils and gcc are needed to
build the initial glibc, these versions are suffixed "-bootstrap".

lxcross ports set PREFIX to ${LXCROSSBASE}, which defaults to
${LOCALBASE}/linux-cross. lxcross-*-bootstrap ports set PREFIX to
${LXBOOTSTRAP}, this one defaults to ${LXCROSSBASE}/bootstrap.

** Open issues

This is an unordered list off my head, so most likely incomplete.

- Some trickery with PREFIX is currently needed. The ports framework
  expects PREFIX to be used as is by the upstream build system. This
  won't hold for building Linux packages, PREFIX must be /compat/linux
  for that, but passed to the upstream build system in DESTDIR.
- LIB_DEPENDS don't work, which could probably be solved in the
  framework. Right now, I'm using a hacky workaround to define
  LINLIB_DEPENDS and add it to both RUN_ and BUILD_DEPENDS.
- A lot of smaller things that *should* be provided by the framework,
  some of them probably by USES=linux, are currently copy&pasted to
  every port needing them. I wanted to keep it simple while first trying
  to get it to work, so the framework isn't touched yet at all.
- Some stage-qa checks get confused, some (e.g. checking that everything
  is stripped) don't work.
- In my tests, "poudriere testport" failed at least on i386, because it
  mounts linprocfs on /compat/linux/proc and then tries to remove
  /compat/linux (remove pre-existing PREFIX). To test the ports, I had
  to slightly modify the testport script for now.
- For the Linux headers, there should be a metaport picking the Linux
  version based on ${OSVERSION}. This doesn't exist yet, Linux 4.4.x is
  always used.
- Building the final linux-gcc ports, I get weird error messages
  directly to poudriere's terminal (they do NOT appear in the build
  log!) like this:
    ELF interpreter /usr/lib/ld-linux.so.2 not found, error 2
  I have no idea where this comes from, so far I couldn't identify any
  negative effect though.

Acknowledgement: I found quite some useful info for doing this in the
"Linux from Scratch" book. Of course you can't just follow the book
(very different scenario, it assumes building on Linux and not doing any
staging/packaging), but it *does* have some helpful hints.

Cheers, Felix

-- 
 Felix Palmen <zirias@FreeBSD.org>     {private}   felix@palmen-it.de
 -- ports committer --                     {web}  http://palmen-it.de
 {pgp public key}  http://palmen-it.de/pub.txt
 {pgp fingerprint} 6936 13D5 5BBF 4837 B212  3ACC 54AD E006 9879 F231