Re: Cross compiling user applications for armv7

[Adding a note and fixing a bad edit.]

On Sep 13, 2025, at 13:31, Mark Millard <marklmi@yahoo.com> wrote:

> On Sep 13, 2025, at 09:45, Michał Kruszewski <mkru@protonmail.com> wrote:
> 
>> Ok, let's start from the beginning because I feel a little bit lost.
>> The buildenv concept is ok when you have no dependencies.
>> However, I have.
>> 
>> I see 3 potential options.
>>  1. Use arm64 as an intermediate platform.
>>     This is unacceptable approach for me.
>>  2. Setup poudriere for armv7 manually.
>>     I tried for 2 days and simply failed.
>>  3. Setup qemu, and install dependencies and build custom code inside VM.
>> 
>> Solutions 2 and 3 are acceptable for me.
>> I guess 2 is better than 3, as it should be more performant.
>> However, I have no idea how to implement idea 2 or 3 in practice.
>> 
>> I also don't understand why setting up poudriere for armv7 is so hard.
>> What is missing by default that I can't do it with a single command?
>> Why the things that are missing are so hard to recreate manually?
>> 
>> Regards,
>> Michał Kruszewski
>> 
>> 
>> Sent with Proton Mail secure email. 
>> 
>> On Saturday, September 13th, 2025 at 6:09 PM, Warner Losh <imp@bsdimp.com> wrote:
>>> 
>>> 
>>> On Sat, Sep 13, 2025 at 9:50 AM Michał Kruszewski <mkru@protonmail.com> wrote:
>>> It is more about how hard it is to setup an environment for cross compiling your own kernel modules or applications for armv7.
>>> I don't really care whether I use Buildroot way of doing things or FreeBSD way.
> . . .
> 
> Ahh. Okay. I read too much into some of your
> wording.
> 
> I do not know what all you need to build. That
> may prove critical for you.
> 
> My boot context for amd64 is main 16 at this point,
> so the below is based on that. I'm using pkgbase
> via poudriere here, pkgbase only exists for FreeBSD
> 14+ .
> 
> Here is what I just tried for poudriere on amd64. But,
> in the end, it got hung up at the end of my test bulk
> run with a bunch of:
> 
> /usr/local/bin/qemu-arm-static /.p/pkg-static repo -o /tmp/packages /packages{qemu-arm-static}
> 
> all stuck in STATE uwait, with the system 99.9% to
> 100% idle. The last status message was:
> 
> Packing files for repository:   0%
> 
> It stayed that way for over 40 minutes before
> I ^C'd the run.
> 
> You might have better results with a 14.3-RELEASE or
> some such. I do not have an appropriate context
> set up for that experiment. main 16 as a context
> does not seem appropriate.
> 
> 
> What I recoded as I tried, showing steps used . . .
> 
> There is:
> 
> emulators/qemu-user-static-devel
> and:
> emulators/qemu-user-static
> 
> But:
> 
> https://portsfallout.com/fallout?port=qemu-user-static&maintainer=&env=&category=&flavor=
> 
> shows emulators/qemu-user-static-devel fails to build on
> main (16 and older 15). (That matched my build attempt,
> as my boot context is main based. My detailed failure
> got: "ld: error: undefined symbol: __realpathat". )
> 
> I was going to try it because, for emulators/qemu-user-static ,
> 
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221185
> 
> reported at 2017-08-02 19:14 UTC is still is active and has a
> 2024-12-09 16:01:45 UTC report confirming the still-then-true
> status of the inability to build lang/rust . (Comments
> 6, 13, 23 mention armv7 status. But aarch64 is at issue
> as well.)
> 
> So I've built: emulators/qemu-user-static
> 
> You may be able to use official port-package
> installation. Such does not exist for amd64
> main 16 yet, so I'm doing my own builds.
> 
> So my steps below are based on use of that
> emulators/qemu-user-static :
> 
> Get to the point that you can do the likes of:
> 
> # pkg install qemu-user-static
> pkg: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
> Updating FreeBSD-base repository catalogue...
> Fetching data.pkg: 100%   53 KiB  54.1kB/s    00:01    
> Processing entries: 100%
> FreeBSD-base repository update completed. 470 packages processed.
> Updating main-amd64-default repository catalogue...
> Fetching meta.conf: 100%    179 B   0.2kB/s    00:01    
> Fetching data.pkg: 100%  189 KiB 193.3kB/s    00:01    
> Processing entries: 100%
> main-amd64-default repository update completed. 561 packages processed.
> All repositories are up to date.
> Checking integrity... done (0 conflicting)
> The following 1 package(s) will be affected (of 0 checked):
> 
> New packages to be INSTALLED:
> qemu-user-static: 3.1.0_15 [main-amd64-default]
> 
> Number of packages to be installed: 1
> 
> The process will require 74 MiB more space.
> 
> Proceed with this action? [y/N]: y
> [1/1] Installing qemu-user-static-3.1.0_15...
> [1/1] Extracting qemu-user-static-3.1.0_15: 100%

Note: As I remember, running binmiscctlwas required
as it sets things up the first time. Thus the below
was more than informational, despite my not
reporting that at the time.

> # binmiscctl list
> . . .
> name: armv7
> interpreter: /usr/local/bin/qemu-arm-static
> flags: ENABLED USE_MASK 
> magic size: 20
> magic offset: 0
> magic: 0x7f 0x45 0x4c 0x46  0x01 0x01 0x01 0x00  0x00 0x00 0x00 0x00 
>       0x00 0x00 0x00 0x00  0x02 0x00 0x28 0x00 
> mask:  0xff 0xff 0xff 0xff  0xff 0xff 0xff 0x00  0xff 0xff 0xff 0xff 
>       0xff 0xff 0xff 0xff  0xfe 0xff 0xff 0xff 
> . . .
> 
> Note: Some details of the below command are specific to targeting
>      main 16 for armv7.
> 
> # poudriere jail -c -jmain-armv7-qemu -aarmv7 -U https://pkg.freebsd.org -mpkgbase=base_latest -v 16
> [00:00:00] Cross-building ports for armv7 on amd64 requires QEMU
> [00:00:00] Creating main-armv7-qemu fs at /usr/local/poudriere/jails/main-armv7-qemu... done
> . . .
> [00:01:34] Recording filesystem state for clean... done
> [00:01:34] Jail main-armv7-qemu 16.0-CURRENT armv7 is ready to be used
> 
> main 16 does not yet have a populated FreeBSD-ports* so I've been
> editing the likes of (here):
> 
> /usr/local/poudriere/jails/main-armv7-qemu/etc/pkg/FreeBSD.conf 
> 
> to indicate "enabled: no" for FreeBSD-ports and FreeBSD-ports*-kmods.
> 
> Note: I picked to build poudriere-devel below only because it
>      is simple/small.
> 
> # poudriere bulk -jmain-armv7-qemu ports-mgmt/poudriere-devel
> [00:00:00] Cross-building ports for armv7 on amd64 requires QEMU
> [00:00:00] Creating the reference jail... done
> [00:00:01] Mounting system devices for main-armv7-qemu-default
> [00:00:01] Converting package repository to new format
> [00:00:01] Stashing existing package repository
> [00:00:01] Mounting ports from: /usr/ports
> [00:00:01] Mounting packages from: /usr/local/poudriere/data/packages/main-armv7-qemu-default
> [00:00:01] Mounting distfiles from: /usr/ports/distfiles
> [00:00:01] Copying /var/db/ports from: /usr/local/etc/poudriere.d/options
> [00:00:01] Raising MAX_EXECUTION_TIME and NOHANG_TIME for QEMU from QEMU_ values
> [00:00:01] Copying latest version of the emulator from: /usr/local/bin/qemu-arm-static
> [00:00:01] Appending to make.conf: /usr/local/etc/poudriere.d/default-make.conf
> [00:00:01] Warning: DEVELOPER=yes ignored from make.conf. Use 'bulk -t' or 'testport' for testing instead.
> /etc/resolv.conf -> /usr/local/poudriere/data/.m/main-armv7-qemu-default/ref/etc/resolv.conf
> [00:00:01] Starting jail main-armv7-qemu-default
> Updating /var/run/os-release done.
> [00:00:02] Will build as root:wheel (0:0)
> [00:00:19] Ports supports: FLAVORS SUBPACKAGES SELECTED_OPTIONS
> [00:00:19] Inspecting /usr/local/poudriere/data/.m/main-armv7-qemu-default/ref//usr/ports for modifications to git checkout... yes
> [00:00:19] Ports top-level git hash: 7e86a0d71167 (dirty)
> [00:00:19] Acquiring build logs lock for main-armv7-qemu-default... done
> [00:00:19] Logs: /usr/local/poudriere/data/logs/bulk/main-armv7-qemu-default/2025-09-13_12h09m03s
> [00:00:19] Loading MOVED for /usr/local/poudriere/data/.m/main-armv7-qemu-default/ref/usr/ports
> [00:00:20] Gathering ports metadata
> [00:00:20] Calculating ports order and dependencies
> [00:00:20] Sanity checking the repository
> [00:00:20] Trimming IGNORED and blacklisted ports
> [00:00:20] pkg bootstrap missing: unable to inspect existing packages, cleaning all packages... done
> [00:00:20] Deleting stale symlinks... done
> [00:00:20] Deleting empty directories... done
> [00:00:20] Unqueueing existing packages
> [00:00:20] Unqueueing orphaned build dependencies
> [00:00:20] Sanity checking build queue
> [00:00:20] [main-armv7-qemu-default] [2025-09-13_12h09m03s] [pkgqueue_sanity_check] Time: 00:00:01
>           Queued: 5 Inspected: 0 Ignored: 0 Built: 0 Failed: 0 Skipped: 0 Fetched: 0 Remaining: 5
> [00:00:20] Recording filesystem state for prepkg... done
> [00:00:20] Processing PRIORITY_BOOST
> [00:00:20] Building 5 packages using up to 5 builders
> [00:00:20] Hit CTRL+t at any time to see build progress and stats
> [00:00:20] [01] [00:00:00] Builder starting
> [00:00:20] [01] [00:00:00] Builder started
> [00:00:20] [01] [00:00:00] Building   ports-mgmt/pkg | pkg-2.3.1
> [00:10:07] [01] [00:09:47] Finished   ports-mgmt/pkg | pkg-2.3.1: Success
> . . .
> [00:12:08] Stopping 5 builders
> [00:12:08] Creating pkg repository
> pkg-static: Both ABI_FILE and OSVERSION are set, ABI_FILE overrides OSVERSION
> pkg-static: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
> Creating repository in /tmp/packages: 100%
> Packing files for repository:   0%
> 
> 
> At this point it hung up as described earlier: STATE uwait
> for a bunch of:
> 
> /usr/local/bin/qemu-arm-static /.p/pkg-static repo -o /tmp/packages /packages{qemu-arm-static}
> 
> 
> NOTE: main 16 can not yet "pkg bootstrap -f" as there
>      is no upstream FreeBSD-ports distributed yet.
> 
> 
> Note: My history with use of amd64 for cross builds targeting armv7

The above "cross builds" should not be there:

Note: My history with use of amd64 for targeting armv7

>      and aarch64 and powerpc* families cross builds via
>      poudriere-devel is many, many years old now, starting back in
>      2017 and going through mid-May of 2020, at which point my
>      interactions with Kyle Evans on the subject stopped as I was
>      no longer contributing. The above sort of result does not
>      seem surprising based on that old experience for armv7 and
>      aarch64. My views are most definitely "based on [my]
>      experience", but other than the above attempt today are
>      based on very old information, for sure.
> 


===
Mark Millard
marklmi at yahoo.com