Re: Cross compiling user applications for armv7

On Sep 21, 2025, at 03:08, Michał Kruszewski <mkru@protonmail.com> wrote:

> I have finally managed to cross compile packages.
> It looks like the only working configuration is as follows:
>  1. The system must be installed as ROOT.
>  2. BUILD_AS_NON_ROOT in poudriere config must be set to NO.
> No other configuration works, and I tested a couple of them.

Hopefully Bryan D. and/or Baptiste D. will reply to my earlier
question about use of pkg's INSTALL_AS_USER with poudriere(-devel).

> There is still one problem.
> The poudriere process freezes on:
>  Packing files for repository:   0%

So your context replicated the qemu hangup that I reported to
Warner from my attempt at doing a tiny build [poudriere(-devel)
itself, with pkg also building]. What version of FreeBSD was
involved in the replication. (Mine was a main 16 boot
kernel and world and poudriere jail world.)

QUOTE from the exchange with Warner:
>> I'll note that my historical (2017..2020-May) problem with
>> poudriere-devel use involving qemu-user-static with
>> poudriere buidlers has been incorrect qemu-user-static
>> operation, especially processes that hang up (e.g, uwait)
>> in the poudriere builders --that work fine on real hardare.
>> It was not getting the combination set up in the first
>> place that was the issue. In mid 2020-May I gave up trying
>> to involve qemu-user-static, which also stopped my
>> interactions with Kyle E. in the subject area.
>> 
>> On Sep 13, 2025, at 17:07, Warner Losh <imp@bsdimp.com> wrote:
>>> Kyle and I have fixed a lot of these... Rust has been hit or miss... The rest seem to be OK...
> 
> I tried to build poudriere-devel (and implicitly pkg) as armv7
> on amd64 today. It hungup after the builders completed:
> 
> . . .
> [00:12:08] Stopping 5 builders
> [00:12:08] Creating pkg repository
> pkg-static: Both ABI_FILE and OSVERSION are set, ABI_FILE overrides OSVERSION
> pkg-static: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
> Creating repository in /tmp/packages: 100%
> Packing files for repository:   0%
> 
> 
> At this point it hung up: STATE uwait
> for a bunch of:
> 
> /usr/local/bin/qemu-arm-static /.p/pkg-static repo -o /tmp/packages /packages{qemu-arm-static}
> 
> 
> NOTE: main 16 can not yet "pkg bootstrap -f" as there
>      is no upstream FreeBSD-ports distributed yet.
END QUOTE

Such historical qemu problems majorly contributed
to why I use hardware that is armv7 capable for
user-mode code for builds targeting armv7.

> I do all of this to evaluate FreeBSD as a potential replacement for Linux for embedded projects.
> I already gathered some experience and have some questions that maybe some of you will be able to answer.
> 
> Is it possible to cross compile FreeBSD system for embedded target without root privileges?
> If you work on your personal PC having a root privileges is not a problem.
> However, I would like to enable multiple users to cross compile on a server machine.
> I don't want to grant root privileges to all of them.
> I know that I can configure doas or sudo.
> However, this is a lot of manual tweaking.
> Buildroot is capable of generating embedded Linux system without root privileges.

I do not know if there is any plan for 
poudriere(-devel) to support use of pkg's
INSTALL_AS_USER vs. not. (I did not find
evidence that it now does.)

NOTE:
Warner found in his experiments that the
INSTALL_AS_USER assignment has to be via
the pkg command having a -o INSTALL_AS_USER=
on the pkg command line.

> How to cross compile packages for a newer system version than the host machine?
> In embedded projects it is very common that the target platform utilizes newer version of the kernel/system than the host machine.

Additional question: over what timescale might the need
for armv7 support to last?

While lib32 support for armv7 may last indefinitely,
including whatever in the kernel is required for that,
I'm unclear on the long term status of either of:

) chroot into an armv7 world
) jails based on an armv7 world

after freebsd 16.* (FreeBSD 17+).

poudiere(-devel) based port package builds depends on:
jails.

If jails were not to be supported indefinitely, could you
get by with targeting FreeBSD 16.*-RELEASE's indefinitely
but not FreeBSD 17 or later? Something to think about.

> Cross compiling newer FreeBSD kernel and world is easy.

While FreeBSD 16.* will have armv7 kernels, after that
has no such guarantee --and seems unlikely for the kernel
to support booting atively, given that even for FreeBSD
16.*-RELEASE armv7 is the last and only 32-bit boot
context supported by a FreeBSD 16.* kernel.

If the support time frame is spanned by FreeBSD 16.* then
this might not be a problem. But it you need the support
after that, ???

> However, how to compile packages?
> You can't create a jail utilizing a newer system version.
> I tried building packages for 16-CURRENT and the process simply failed.

If  the jail is executing a newer world, then the
jail is executing code that may attempt to use new
kernel services or depend on new kernel behavior.
Such is not supported.

> Poudriere even warned me that it is going to fail.
> 
> Cross-compiling packages using poudriere for armv7 is slow, it is extremely slow.

Such historical performance contributed some to
why I use hardware that is armv7 capable for
user-mode code --and possibly would have even
without the historical hangup problems mentioned
earlier.

> I utilize 11 cores for building.
> Cross compiling Python, I don't count kernel and world compilation, takes approximately 1.5 times more than compiling the whole Linux based system plus Python using Buildroot.
> Is there anything I can do about that?

If python bootstraps partially via the system C/C++
toolchain, that part would likely not be a major
contributor. But if much of its activity is based
on executing some subset of phython itself, it is
going to be qemu emulation executing the python
interpreter: 2 layers of emulation/interpretation.

If you wanted a comparison/contrast to armv7
native-capable aarch64 timing, I've access to
the following that still seem to be available
for purchase (as new, via official channels):

RPi3B
RPi4B's
RPi5B's (using an EDK2 draft for booting)
SolidRun Honeycomb (16 cortex-A72 cores)

I could build a set of port packages from
scratch, one one of those gathering and
reporting time.

But, based on your prior notes, I'm guessing
you have no interest in this route or type
of information.

===
Mark Millard
marklmi at yahoo.com