Re: Cross compiling user applications for armv7

On Sat, Sep 13, 2025 at 11:00 PM Warner Losh <imp@bsdimp.com> wrote:

>
>
> On Sat, Sep 13, 2025 at 9:46 PM Mark Millard <marklmi@yahoo.com> wrote:
>
>>
>>
>> On Sep 13, 2025, at 17:07, Warner Losh <imp@bsdimp.com> wrote:
>>
>> > On Sat, Sep 13, 2025 at 10:45 AM Michał Kruszewski <mkru@protonmail.com>
>> wrote:
>> >> Ok, let's start from the beginning because I feel a little bit lost.
>> >> The buildenv concept is ok when you have no dependencies.
>> >> However, I have.
>> >>
>> >> I see 3 potential options.
>> >>   1. Use arm64 as an intermediate platform.
>> >>      This is unacceptable approach for me.
>> >>   2. Setup poudriere for armv7 manually.
>> >>      I tried for 2 days and simply failed.
>> >>   3. Setup qemu, and install dependencies and build custom code inside
>> VM.
>> >>
>> >> Solutions 2 and 3 are acceptable for me.
>> >> I guess 2 is better than 3, as it should be more performant.
>> >> However, I have no idea how to implement idea 2 or 3 in practice.
>> >>
>> >> I also don't understand why setting up poudriere for armv7 is so hard.
>> >> What is missing by default that I can't do it with a single command?
>> >>
>> > You can do it with a single command, just not from binaries because our
>> armv7 packaging was always just for embedded, so we didn't do the right
>> thing for normal installs.
>>
>> If one is okay with use of 14.3 pkgbase materials as the source for
>> a armv7 poudriere-devel jail on amd64,
>>
>> # poudriere jail -c -jarmv7 -aarmv7 -U https://pkg.freebsd.org
>> -mpkgbase=base_release_3 -v 14
>>
>
This worked for me. pkgbase has come a long way...

I think we need cross-build(7) to document these steps.


> should produce a poudriere-devel jail targeting armv7 that is made from
>> pre-built
>> binaries (pkgbase packages to be installed).
>>
>> The above does presume that emulators/qemu-user-static (or
>> emulators/qemu-user-static-devel ?) has been installed
>> already and that binmiscctl has been run so that armv7 has
>> been set up.
>>
>> > However,
>> https://forums.freebsd.org/threads/poudriere-image-creation-for-arm-architecture.91870/
>> has a good recipe for the source build. Have you tried it.
>> >
>> > poudriere jail -c -x -j armv7 -m git+https -a arm.armv7 -v releng/14.3
>> -K GENERIC
>> > poudriere ports -c -m git+https -B main -p HEAD
>>
>> "man 8 poudriere-jail" no longer lists -x . Instead it lists:
>>
>>      -X               Do not build and setup native-xtools cross compile
>> tools
>>                       in jail when building for a different TARGET ARCH
>> than
>>                       the host.  Only applies if TARGET_ARCH and
>> HOST_ARCH are
>>                       different.
>>
>
> Hmmm. I don't recall what it.
>

sudo poudriere -e /vidpool/qemu/poudriere jail -c -j armv7-current -a armv7
-D -m git+ssh

is what I use (I'd omit the -e XXXX), not -x either.

>
>
>> > are the commands to setup the root and to get ports so you can do a
>> poudriere bulk to build the ports you are interested in.
>> >
>> >> Why the things that are missing are so hard to recreate manually?
>> >
>> >
>> > So to do #2 manually... The following blog, something I wrote long ago,
>> but that should still work
>> >
>> https://wiki.freebsd.org/WarnerLosh/A_Brief_Guide_To_Cross_Compiling_FreeBSD
>> > has the steps (though you'll need to s/armeb/armv7/ in it). There's
>> three commands to creating an image. But the results are almost the same as
>> the above the first poudriere command while being less flexible or
>> performant.
>> >
>> > As for #3, it's a bit less performant, but you can run FreeBSD inside
>> of qemu using the virt* devices easily enough (though I think there's a bug
>> with 14.3's kernel so you have to use em NIC not virtnet):
>> >
>> > https://bsdimp.blogspot.com/2023/12/freebsdarmv7-in-qemu.html
>> >
>> > has a writeup I did a while ago. It's still accurate and suggests the
>> virtnet bug may be fixed in all 14.x releases. I run this setup from time
>> to time still to test things in a armv7 system. It's OK if what you are
>> building is small, but can get tedious if you have lots of complex
>> dependencies.
>> >
>> > Hope this helps.
>>
>> I'll note that my historical (2017..2020-May) problem with
>> poudriere-devel use involving qemu-user-static with
>> poudriere buidlers has been incorrect qemu-user-static
>> operation, especially processes that hang up (e.g, uwait)
>> in the poudriere builders --that work fine on real hardare.
>> It was not getting the combination set up in the first
>> place that was the issue. In mid 2020-May I gave up trying
>> to involve qemu-user-static, which also stopped my
>> interactions with Kyle E. in the subject area.
>>
>
> Kyle and I have fixed a lot of these... Rust has been hit or miss... The
> rest seem to be OK...
>
> Warner
>
>
>> > Warner
>> >
>> >>
>> >> Regards,
>> >> Michał Kruszewski
>> >>
>> >>
>> >> Sent with Proton Mail secure email.
>> >>
>> >> On Saturday, September 13th, 2025 at 6:09 PM, Warner Losh <
>> imp@bsdimp.com> wrote:
>> >>>
>> >>>
>> >>> On Sat, Sep 13, 2025 at 9:50 AM Michał Kruszewski <
>> mkru@protonmail.com> wrote:
>> >>>> It is more about how hard it is to setup an environment for cross
>> compiling your own kernel modules or applications for armv7.
>> >>>> I don't really care whether I use Buildroot way of doing things or
>> FreeBSD way.
>> >>>> The difference is that with Buildroot it takes 10 minutes to setup
>> such environment.
>> >>>> With FreeBSD I am struggling for two days.
>> >>>> And the only advice I get is to use arm64 machine.
>> >>>> Well, I don't have any.
>> >>>> And even if I had, I wouldn't use it, because I find this concept
>> deeply "misengineered".
>> >>>>
>> >> And it is bad engineering. Mark sometimes gives bad advice that's
>> based on his experience where it was "easier" to do it this way. I just
>> wish I'd replied more quickly.
>> >>> I know that you want to help, and I appreciate it.
>> >>> I just feel frustrated with the complexity of this conceptually
>> trivial task.
>> >>
>> >> No. I get it. I'm frustrated too since I've been working for years to
>> make it better, and there's still these rough edges that turn people off.
>> >>
>> >> Warner
>> >>> Regards,
>> >>> Michał Kruszewski
>> >>>
>> >>>
>> >>> Sent with Proton Mail secure email.
>> >>>
>> >>> On Saturday, September 13th, 2025 at 4:29 PM, Mark Millard <
>> marklmi@yahoo.com> wrote:
>> >>>
>> >>> >
>> >>> > On Sep 13, 2025, at 00:20, Michał Kruszewski mkru@protonmail.com
>> wrote:
>> >>> >
>> >>> > > Basically you are saying that I should have a separate physical
>> arm64 machine for doing embedded armv7 development on FreeBSD.
>> >>> > > If I understand this correctly, I just can't understand how can
>> people say that FreeBSD is embedded friendly.
>> >>> > > This is so much complex and convoluted compared to just using,
>> for example, Buildroot on Linux.
>> >>> > > This also doesn't feel like a *nix way of solving things.
>> >>> > > I thought that after cross compiling world and kernel, there is
>> some shell script that I can simply source to start cross-development for
>> target platform.
>> >>> >
>> >>> >
>> >>> > I found this AI based note about Buildroot's way of
>> >>> > putting cross-build toolchains to use:
>> >>> >
>> >>> > QUOTE
>> >>> > Buildroot pre-configures and compiles all necessarily
>> >>> > packages during the build process to create a custom
>> >>> > root filesystem.
>> >>> > . . .
>> >>> > Buildroot focuses on static builds, meaing all selected
>> >>> > packages are compiled and integrated into the final root
>> >>> > filesystem image during the build process. There is no
>> >>> > concept of installing or updating packages on the target
>> >>> > after the initial build.
>> >>> > END QUOTE
>> >>> >
>> >>> > This is certainly not what the FreeBSD ports tree is
>> >>> > set up for. Buildroot or analogous does not exist for
>> >>> > FreeBSD --or most *BSD or Unix-- to my knowledge.
>> >>> >
>> >>> > May be NetBSD's pkgsrc cross build support:
>> >>> >
>> >>> >
>> https://cdn.netbsd.org/pub/pkgsrc/current/pkgsrc/doc/HOWTO-use-crosscompile
>> >>> >
>> >>> > would be closer to how you like to work? (I've never
>> >>> > used such.)
>> >>> >
>> >>> > > Regards,
>> >>> > > Michał Kruszewski
>> >>> > >
>> >>> > > Sent with Proton Mail secure email.
>> >>> > >
>> >>> > > On Saturday, September 13th, 2025 at 12:23 AM, Mark Millard
>> marklmi@yahoo.com wrote:
>> >>> > >
>> >>> > > > On Sep 12, 2025, at 14:10, Michał Kruszewski
>> mkru@protonmail.com wrote:
>> >>> > > >
>> >>> > > > > A simple way is to set up an armv7 chroot / jail on an arm64
>> host such as a Raspberry Pi (Apple devices don't work!).
>> >>> > > > > Then it's just like a native environment, but usually much
>> faster.
>> >>> > > >
>> >>> > > > > I don't have physical arm64 host.
>> >>> > > > > I also don't want to create a VM.
>> >>> > > > > The whole idea seems weird and unnatural.
>> >>> > > >
>> >>> > > > Ignoring the RPi4B detail:
>> >>> > > >
>> >>> > > > FreeBSD official armv7 port-packages are built this way
>> >>> > > > on arm64 hardware that natively supports user-space
>> >>> > > > armv7 code. (ampere* systems are used.)
>> >>> > > >
>> >>> > > > FreeBSD official i386 port-packages are built this way
>> >>> > > > on amd64 hardware that natively supports user-space
>> >>> > > > i386 code.
>> >>> > > >
>> >>> > > > No use of qemu variants of any kind: no attempt at
>> >>> > > > non-native-capable host environments.
>> >>> > > >
>> >>> > > > (Back when amd64 and qemu was used for the likes
>> >>> > > > of armv7, lots of stuff failed to build mcuh of
>> >>> > > > the time over the years --stuff that builds
>> >>> > > > just fine now. This was abandoned after native
>> >>> > > > became available. armv6 was always qemu based
>> >>> > > > on amd64 and could not build much as of the last
>> >>> > > > time a build was run.)
>> >>> > > >
>> >>> > > > > It sounds more like an exotic workaround.
>> >>> > > >
>> >>> > > > It is the official technique used for what FreeBSD
>> >>> > > > distributes for armv7 port-packages.
>> >>> > > >
>> >>> > > > > Regards,
>> >>> > > > > Michał Kruszewski
>> >>> > > > >
>> >>> > > > > Sent with Proton Mail secure email.
>> >>> > > > >
>> >>> > > > > On Friday, September 12th, 2025 at 10:46 PM, Robert
>> Clausecker fuz@fuz.su wrote:
>> >>> > > > >
>> >>> > > > > > Hi Michał,
>> >>> > > > > >
>> >>> > > > > > A simple way is to set up an armv7 chroot / jail on an
>> arm64 host such as
>> >>> > > > > > a Raspberry Pi (Apple devices don't work!). Then it's just
>> like a native
>> >>> > > > > > environment, but usually much faster.
>> >>> > > > > >
>> >>> > > > > > Yours,
>> >>> > > > > > Robert Clausecker
>> >>> > > > > >
>> >>> > > > > > Am Fri, Sep 12, 2025 at 04:19:42PM +0000 schrieb Michał
>> Kruszewski:
>> >>> > > > > >
>> >>> > > > > > > Is there any tutorial on how to cross compile custom user
>> application for armv7?
>> >>> > > > > > > I struggle t find any.
>> >>> > > > > > > Cross compiling the system is pretty easy.
>> >>> > > > > > > However, how can one cross compile custom user
>> application or kernel drivers.
>> >>> > > > > > >
>> >>> > > > > > > Regards,
>> >>> > > > > > > Michał Kruszewski
>> >>> > > > > > >
>> >>> > > > > > > Sent with Proton Mail secure email.
>> >>> > > > > >
>> >>> > > > > > --
>> >>> > > > > > () ascii ribbon campaign - for an encoding-agnostic world
>> >>> > > > > > /\ - against html email - against proprietary attachments
>> >>> > > >
>> >>> > > > ===
>> >>> > > > Mark Millard
>> >>> > > > marklmi at yahoo.com
>> >>> >
>> >>> >
>> >>> > ===
>> >>> > Mark Millard
>> >>> > marklmi at yahoo.com
>> >>>
>> >>
>> >
>> ===
>> Mark Millard
>> marklmi at yahoo.com
>>
>>