Help cross-building kernel from macos

Reply: Kyle Evans : "Re: Help cross-building kernel from macos"
Reply: Andrew Turner : "Re: Help cross-building kernel from macos"
Go to: [ bottom of page ] [ top of archives ] [ this month ]
From: Colin S. Gordon <csgordon_at_fastmail.com>
Date: Wed, 08 Nov 2023 19:54:33 UTC
This is a longshot but: I'm trying to get cross-building working from my mac so I don't have to lug 2 laptops around every day to make any progress on some driver work, but so far haven't had too much luck.  This is (to my pleasant surprise!) supported (https://docs.freebsd.org/en/books/handbook/cutting-edge/#building-on-non-freebsd-hosts) and as Ed pointed out on Mastodon, even runs successfully in CI (https://github.com/freebsd/freebsd-src/blob/main/.github/workflows/cross-bootstrap-tools.yml). But so far I'm unable get this working on my own machine.  I was initially trying to get this running with my slightly-stale branch where I'm doing some RISC-V stuff, but hit the same problem building a recent commit from HEAD that does cross-build from macos in CI, as evidenced here: https://github.com/freebsd/freebsd-src/actions/runs/6801518714/job/18492566834

I've installed the same dependencies (same version of LLVM) specified in the github action file, and am building with the same commands (with different local directories, of course).  I appear to be one macos release ahead of the version used in CI, though it doesn't seem like that should affect the failure I'm encountering.  The first command,

    MAKEOBJDIRPREFIX=~/freebsd-obj/ tools/build/make.py --debug --cross-bindir=/usr/local/opt/llvm@13/bin TARGET=riscv TARGET_ARCH=riscv64 -n

seems to work fine. But the second, a pre-requisite to building the kernel, fails:

MAKEOBJDIRPREFIX=~/freebsd-obj/ tools/build/make.py --debug --cross-bindir=/usr/local/opt/llvm@13/bin TARGET=riscv TARGET_ARCH=riscv64 kernel-toolchain -s -DWITH_DISK_IMAGE_TOOLS_BOOTSTRAP
Inferred CC as /usr/bin/cc
Running ['/usr/bin/cc', '--version']
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: x86_64-apple-darwin22.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
Inferred CXX as /usr/bin/c++
Running ['/usr/bin/c++', '--version']
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: x86_64-apple-darwin22.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
Inferred CPP as /usr/bin/cpp
Running ['/usr/bin/cpp', '--version']
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: x86_64-apple-darwin22.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
Inferred XCC as /usr/local/opt/llvm@13/bin/clang
Running ['/usr/local/opt/llvm@13/bin/clang', '--version']
Homebrew clang version 13.0.1
Target: x86_64-apple-darwin22.6.0
Thread model: posix
InstalledDir: /usr/local/opt/llvm@13/bin
Inferred XCXX as /usr/local/opt/llvm@13/bin/clang++
Running ['/usr/local/opt/llvm@13/bin/clang++', '--version']
Homebrew clang version 13.0.1
Target: x86_64-apple-darwin22.6.0
Thread model: posix
InstalledDir: /usr/local/opt/llvm@13/bin
Inferred XCPP as /usr/local/opt/llvm@13/bin/clang-cpp
Running ['/usr/local/opt/llvm@13/bin/clang-cpp', '--version']
Homebrew clang version 13.0.1
Target: x86_64-apple-darwin22.6.0
Thread model: posix
InstalledDir: /usr/local/opt/llvm@13/bin
Inferred XLD as /usr/local/opt/llvm@13/bin/ld.lld
Running ['/usr/local/opt/llvm@13/bin/ld.lld', '--version']
Homebrew LLD 13.0.1 (compatible with GNU linkers)
Source bmake version: 20230909
Installed bmake version: 20230909
Configure args: --with-default-sys-path=.../share/mk:/Users/.../freebsd-obj/bmake-install/share/mk --with-machine=amd64 --without-filemon --prefix=/Users/.../freebsd-obj/bmake-install
Last configure args: --with-default-sys-path=.../share/mk:/Users/.../freebsd-obj/bmake-install/share/mk --with-machine=amd64 --without-filemon --prefix=/Users/.../freebsd-obj/bmake-install
Adding -DWITH_AUTO_OBJ
Running `env CC=/usr/bin/cc CXX=/usr/bin/c++ CPP=/usr/bin/cpp XCC=/usr/local/opt/llvm@13/bin/clang XCXX=/usr/local/opt/llvm@13/bin/clang++ XCPP=/usr/local/opt/llvm@13/bin/clang-cpp XLD=/usr/local/opt/llvm@13/bin/ld.lld /Users/.../freebsd-obj/bmake-install/bin/bmake TARGET=riscv TARGET_ARCH=riscv64 kernel-toolchain -s -DWITH_DISK_IMAGE_TOOLS_BOOTSTRAP -DWITH_AUTO_OBJ -DWITHOUT_CLEAN`
>>> Deleting stale files in build tree...
        6.29 real         4.95 user         1.61 sys

--------------------------------------------------------------
>>> Rebuilding the temporary build tree
--------------------------------------------------------------
Linking host tools into /Users/.../freebsd-obj/Users/.../freebsd-src/riscv.riscv64/tmp/legacy/bin

--------------------------------------------------------------
>>> stage 1.1: legacy release compatibility shims
--------------------------------------------------------------
===> tools/build (obj,includes,all,install)

real 0m1.938s
user 0m0.624s
sys 0m0.697s

--------------------------------------------------------------
>>> stage 1.2: bootstrap tools
--------------------------------------------------------------
>>> Deleting stale dependencies...

real 0m0.385s
user 0m0.132s
sys 0m0.211s
===> lib/clang/libllvmminimal (obj,all,install)
===> usr.bin/clang/llvm-tblgen (obj,all,install)
bmake[3]: /Users/.../freebsd-obj/Users/.../freebsd-src/riscv.riscv64/tmp/obj-tools/usr.bin/clang/llvm-tblgen/.depend, 1: ignoring stale .depend for /Users/.../freebsd-obj//Users/.../freebsd-src/riscv.riscv64/tmp/legacy/usr/lib/libz.a
bmake[3]: /Users/.../freebsd-obj/Users/.../freebsd-src/riscv.riscv64/tmp/obj-tools/usr.bin/clang/llvm-tblgen/.depend, 1: ignoring stale .depend for /usr/lib/libprivatezstd.a
ld: warning: directory not found for option '-L/Users/.../freebsd-obj/Users/.../freebsd-src/riscv.riscv64/tmp/obj-tools/lib/libthr'
ld: warning: directory not found for option '-L/Users/.../freebsd-obj/Users/.../freebsd-src/riscv.riscv64/tmp/obj-tools/lib/libz'
ld: warning: directory not found for option '-L/Users/.../freebsd-obj/Users/.../freebsd-src/riscv.riscv64/tmp/obj-tools/lib/libthr'
ld: library not found for -lprivatezstd
clang: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1

Stop.
bmake[3]: stopped in /Users/.../freebsd-src/usr.bin/clang/llvm-tblgen
*** Error code 1

Stop.
bmake[2]: stopped in /Users/.../freebsd-src
*** Error code 1

Stop.
bmake[1]: stopped in /Users/.../freebsd-src
*** Error code 1

Stop.
bmake: stopped in /Users/.../freebsd-src


I get the same failure if I just try to buildworld instead of build kernel-toolchain. Above I'm trying to cross-build riscv64, but the same things happens when I try other platforms as well (e.g., aarch64, which specifically is built in CI).

I figure relatively few people have actually debugged a cross-build from macos, but has anyone seen similar build errors about privatezstd before, or other llvm-tblgen build failures?

Thanks for any pointers,
Colin