Re: Cross compiling user applications for armv7

On Sep 13, 2025, at 09:45, Michał Kruszewski <mkru@protonmail.com> wrote:

> Ok, let's start from the beginning because I feel a little bit lost.
> The buildenv concept is ok when you have no dependencies.
> However, I have.
> 
> I see 3 potential options.
>   1. Use arm64 as an intermediate platform.
>      This is unacceptable approach for me.
>   2. Setup poudriere for armv7 manually.
>      I tried for 2 days and simply failed.
>   3. Setup qemu, and install dependencies and build custom code inside VM.
> 
> Solutions 2 and 3 are acceptable for me.
> I guess 2 is better than 3, as it should be more performant.
> However, I have no idea how to implement idea 2 or 3 in practice.
> 
> I also don't understand why setting up poudriere for armv7 is so hard.
> What is missing by default that I can't do it with a single command?
> Why the things that are missing are so hard to recreate manually?
> 
> Regards,
> Michał Kruszewski
> 
> 
> Sent with Proton Mail secure email. 
> 
> On Saturday, September 13th, 2025 at 6:09 PM, Warner Losh <imp@bsdimp.com> wrote:
>> 
>> 
>> On Sat, Sep 13, 2025 at 9:50 AM Michał Kruszewski <mkru@protonmail.com> wrote:
>> It is more about how hard it is to setup an environment for cross compiling your own kernel modules or applications for armv7.
>> I don't really care whether I use Buildroot way of doing things or FreeBSD way.
. . .

Ahh. Okay. I read too much into some of your
wording.

I do not know what all you need to build. That
may prove critical for you.

My boot context for amd64 is main 16 at this point,
so the below is based on that. I'm using pkgbase
via poudriere here, pkgbase only exists for FreeBSD
14+ .

Here is what I just tried for poudriere on amd64. But,
in the end, it got hung up at the end of my test bulk
run with a bunch of:

/usr/local/bin/qemu-arm-static /.p/pkg-static repo -o /tmp/packages /packages{qemu-arm-static}

all stuck in STATE uwait, with the system 99.9% to
100% idle. The last status message was:

Packing files for repository:   0%

It stayed that way for over 40 minutes before
I ^C'd the run.

You might have better results with a 14.3-RELEASE or
some such. I do not have an appropriate context
set up for that experiment. main 16 as a context
does not seem appropriate.


What I recoded as I tried, showing steps used . . .

There is:

emulators/qemu-user-static-devel
and:
emulators/qemu-user-static

But:

https://portsfallout.com/fallout?port=qemu-user-static&maintainer=&env=&category=&flavor=

shows emulators/qemu-user-static-devel fails to build on
main (16 and older 15). (That matched my build attempt,
as my boot context is main based. My detailed failure
got: "ld: error: undefined symbol: __realpathat". )

I was going to try it because, for emulators/qemu-user-static ,

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221185

reported at 2017-08-02 19:14 UTC is still is active and has a
2024-12-09 16:01:45 UTC report confirming the still-then-true
status of the inability to build lang/rust . (Comments
6, 13, 23 mention armv7 status. But aarch64 is at issue
as well.)

So I've built: emulators/qemu-user-static

You may be able to use official port-package
installation. Such does not exist for amd64
main 16 yet, so I'm doing my own builds.

So my steps below are based on use of that
emulators/qemu-user-static :

Get to the point that you can do the likes of:

# pkg install qemu-user-static
pkg: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
Updating FreeBSD-base repository catalogue...
Fetching data.pkg: 100%   53 KiB  54.1kB/s    00:01    
Processing entries: 100%
FreeBSD-base repository update completed. 470 packages processed.
Updating main-amd64-default repository catalogue...
Fetching meta.conf: 100%    179 B   0.2kB/s    00:01    
Fetching data.pkg: 100%  189 KiB 193.3kB/s    00:01    
Processing entries: 100%
main-amd64-default repository update completed. 561 packages processed.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	qemu-user-static: 3.1.0_15 [main-amd64-default]

Number of packages to be installed: 1

The process will require 74 MiB more space.

Proceed with this action? [y/N]: y
[1/1] Installing qemu-user-static-3.1.0_15...
[1/1] Extracting qemu-user-static-3.1.0_15: 100%

# binmiscctl list
. . .
name: armv7
interpreter: /usr/local/bin/qemu-arm-static
flags: ENABLED USE_MASK 
magic size: 20
magic offset: 0
magic: 0x7f 0x45 0x4c 0x46  0x01 0x01 0x01 0x00  0x00 0x00 0x00 0x00 
       0x00 0x00 0x00 0x00  0x02 0x00 0x28 0x00 
mask:  0xff 0xff 0xff 0xff  0xff 0xff 0xff 0x00  0xff 0xff 0xff 0xff 
       0xff 0xff 0xff 0xff  0xfe 0xff 0xff 0xff 
. . .

Note: Some details of the below command are specific to targeting
      main 16 for armv7.

# poudriere jail -c -jmain-armv7-qemu -aarmv7 -U https://pkg.freebsd.org -mpkgbase=base_latest -v 16
[00:00:00] Cross-building ports for armv7 on amd64 requires QEMU
[00:00:00] Creating main-armv7-qemu fs at /usr/local/poudriere/jails/main-armv7-qemu... done
. . .
[00:01:34] Recording filesystem state for clean... done
[00:01:34] Jail main-armv7-qemu 16.0-CURRENT armv7 is ready to be used

main 16 does not yet have a populated FreeBSD-ports* so I've been
editing the likes of (here):

/usr/local/poudriere/jails/main-armv7-qemu/etc/pkg/FreeBSD.conf 

to indicate "enabled: no" for FreeBSD-ports and FreeBSD-ports*-kmods.

Note: I picked to build poudriere-devel below only because it
      is simple/small.

# poudriere bulk -jmain-armv7-qemu ports-mgmt/poudriere-devel
[00:00:00] Cross-building ports for armv7 on amd64 requires QEMU
[00:00:00] Creating the reference jail... done
[00:00:01] Mounting system devices for main-armv7-qemu-default
[00:00:01] Converting package repository to new format
[00:00:01] Stashing existing package repository
[00:00:01] Mounting ports from: /usr/ports
[00:00:01] Mounting packages from: /usr/local/poudriere/data/packages/main-armv7-qemu-default
[00:00:01] Mounting distfiles from: /usr/ports/distfiles
[00:00:01] Copying /var/db/ports from: /usr/local/etc/poudriere.d/options
[00:00:01] Raising MAX_EXECUTION_TIME and NOHANG_TIME for QEMU from QEMU_ values
[00:00:01] Copying latest version of the emulator from: /usr/local/bin/qemu-arm-static
[00:00:01] Appending to make.conf: /usr/local/etc/poudriere.d/default-make.conf
[00:00:01] Warning: DEVELOPER=yes ignored from make.conf. Use 'bulk -t' or 'testport' for testing instead.
/etc/resolv.conf -> /usr/local/poudriere/data/.m/main-armv7-qemu-default/ref/etc/resolv.conf
[00:00:01] Starting jail main-armv7-qemu-default
Updating /var/run/os-release done.
[00:00:02] Will build as root:wheel (0:0)
[00:00:19] Ports supports: FLAVORS SUBPACKAGES SELECTED_OPTIONS
[00:00:19] Inspecting /usr/local/poudriere/data/.m/main-armv7-qemu-default/ref//usr/ports for modifications to git checkout... yes
[00:00:19] Ports top-level git hash: 7e86a0d71167 (dirty)
[00:00:19] Acquiring build logs lock for main-armv7-qemu-default... done
[00:00:19] Logs: /usr/local/poudriere/data/logs/bulk/main-armv7-qemu-default/2025-09-13_12h09m03s
[00:00:19] Loading MOVED for /usr/local/poudriere/data/.m/main-armv7-qemu-default/ref/usr/ports
[00:00:20] Gathering ports metadata
[00:00:20] Calculating ports order and dependencies
[00:00:20] Sanity checking the repository
[00:00:20] Trimming IGNORED and blacklisted ports
[00:00:20] pkg bootstrap missing: unable to inspect existing packages, cleaning all packages... done
[00:00:20] Deleting stale symlinks... done
[00:00:20] Deleting empty directories... done
[00:00:20] Unqueueing existing packages
[00:00:20] Unqueueing orphaned build dependencies
[00:00:20] Sanity checking build queue
[00:00:20] [main-armv7-qemu-default] [2025-09-13_12h09m03s] [pkgqueue_sanity_check] Time: 00:00:01
           Queued: 5 Inspected: 0 Ignored: 0 Built: 0 Failed: 0 Skipped: 0 Fetched: 0 Remaining: 5
[00:00:20] Recording filesystem state for prepkg... done
[00:00:20] Processing PRIORITY_BOOST
[00:00:20] Building 5 packages using up to 5 builders
[00:00:20] Hit CTRL+t at any time to see build progress and stats
[00:00:20] [01] [00:00:00] Builder starting
[00:00:20] [01] [00:00:00] Builder started
[00:00:20] [01] [00:00:00] Building   ports-mgmt/pkg | pkg-2.3.1
[00:10:07] [01] [00:09:47] Finished   ports-mgmt/pkg | pkg-2.3.1: Success
. . .
[00:12:08] Stopping 5 builders
[00:12:08] Creating pkg repository
pkg-static: Both ABI_FILE and OSVERSION are set, ABI_FILE overrides OSVERSION
pkg-static: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
Creating repository in /tmp/packages: 100%
Packing files for repository:   0%


At this point it hung up as described earlier: STATE uwait
for a bunch of:

/usr/local/bin/qemu-arm-static /.p/pkg-static repo -o /tmp/packages /packages{qemu-arm-static}


NOTE: main 16 can not yet "pkg bootstrap -f" as there
      is no upstream FreeBSD-ports distributed yet.


Note: My history with use of amd64 for cross builds targeting armv7
      and aarch64 and powerpc* families cross builds via
      poudriere-devel is many, many years old now, starting back in
      2017 and going through mid-May of 2020, at which point my
      interactions with Kyle Evans on the subject stopped as I was
      no longer contributing. The above sort of result does not
      seem surprising based on that old experience for armv7 and
      aarch64. My views are most definitely "based on [my]
      experience", but other than the above attempt today are
      based on very old information, for sure.


===
Mark Millard
marklmi at yahoo.com