Re: Cross compiling user applications for armv7
Date: Sun, 14 Sep 2025 05:00:13 UTC
On Sat, Sep 13, 2025 at 9:46 PM Mark Millard <marklmi@yahoo.com> wrote: > > > On Sep 13, 2025, at 17:07, Warner Losh <imp@bsdimp.com> wrote: > > > On Sat, Sep 13, 2025 at 10:45 AM Michał Kruszewski <mkru@protonmail.com> > wrote: > >> Ok, let's start from the beginning because I feel a little bit lost. > >> The buildenv concept is ok when you have no dependencies. > >> However, I have. > >> > >> I see 3 potential options. > >> 1. Use arm64 as an intermediate platform. > >> This is unacceptable approach for me. > >> 2. Setup poudriere for armv7 manually. > >> I tried for 2 days and simply failed. > >> 3. Setup qemu, and install dependencies and build custom code inside > VM. > >> > >> Solutions 2 and 3 are acceptable for me. > >> I guess 2 is better than 3, as it should be more performant. > >> However, I have no idea how to implement idea 2 or 3 in practice. > >> > >> I also don't understand why setting up poudriere for armv7 is so hard. > >> What is missing by default that I can't do it with a single command? > >> > > You can do it with a single command, just not from binaries because our > armv7 packaging was always just for embedded, so we didn't do the right > thing for normal installs. > > If one is okay with use of 14.3 pkgbase materials as the source for > a armv7 poudriere-devel jail on amd64, > > # poudriere jail -c -jarmv7 -aarmv7 -U https://pkg.freebsd.org > -mpkgbase=base_release_3 -v 14 > > should produce a poudriere-devel jail targeting armv7 that is made from > pre-built > binaries (pkgbase packages to be installed). > > The above does presume that emulators/qemu-user-static (or > emulators/qemu-user-static-devel ?) has been installed > already and that binmiscctl has been run so that armv7 has > been set up. > > > However, > https://forums.freebsd.org/threads/poudriere-image-creation-for-arm-architecture.91870/ > has a good recipe for the source build. Have you tried it. > > > > poudriere jail -c -x -j armv7 -m git+https -a arm.armv7 -v releng/14.3 > -K GENERIC > > poudriere ports -c -m git+https -B main -p HEAD > > "man 8 poudriere-jail" no longer lists -x . Instead it lists: > > -X Do not build and setup native-xtools cross compile > tools > in jail when building for a different TARGET ARCH > than > the host. Only applies if TARGET_ARCH and HOST_ARCH > are > different. > Hmmm. I don't recall what it. > > are the commands to setup the root and to get ports so you can do a > poudriere bulk to build the ports you are interested in. > > > >> Why the things that are missing are so hard to recreate manually? > > > > > > So to do #2 manually... The following blog, something I wrote long ago, > but that should still work > > > https://wiki.freebsd.org/WarnerLosh/A_Brief_Guide_To_Cross_Compiling_FreeBSD > > has the steps (though you'll need to s/armeb/armv7/ in it). There's > three commands to creating an image. But the results are almost the same as > the above the first poudriere command while being less flexible or > performant. > > > > As for #3, it's a bit less performant, but you can run FreeBSD inside of > qemu using the virt* devices easily enough (though I think there's a bug > with 14.3's kernel so you have to use em NIC not virtnet): > > > > https://bsdimp.blogspot.com/2023/12/freebsdarmv7-in-qemu.html > > > > has a writeup I did a while ago. It's still accurate and suggests the > virtnet bug may be fixed in all 14.x releases. I run this setup from time > to time still to test things in a armv7 system. It's OK if what you are > building is small, but can get tedious if you have lots of complex > dependencies. > > > > Hope this helps. > > I'll note that my historical (2017..2020-May) problem with > poudriere-devel use involving qemu-user-static with > poudriere buidlers has been incorrect qemu-user-static > operation, especially processes that hang up (e.g, uwait) > in the poudriere builders --that work fine on real hardare. > It was not getting the combination set up in the first > place that was the issue. In mid 2020-May I gave up trying > to involve qemu-user-static, which also stopped my > interactions with Kyle E. in the subject area. > Kyle and I have fixed a lot of these... Rust has been hit or miss... The rest seem to be OK... Warner > > Warner > > > >> > >> Regards, > >> Michał Kruszewski > >> > >> > >> Sent with Proton Mail secure email. > >> > >> On Saturday, September 13th, 2025 at 6:09 PM, Warner Losh < > imp@bsdimp.com> wrote: > >>> > >>> > >>> On Sat, Sep 13, 2025 at 9:50 AM Michał Kruszewski <mkru@protonmail.com> > wrote: > >>>> It is more about how hard it is to setup an environment for cross > compiling your own kernel modules or applications for armv7. > >>>> I don't really care whether I use Buildroot way of doing things or > FreeBSD way. > >>>> The difference is that with Buildroot it takes 10 minutes to setup > such environment. > >>>> With FreeBSD I am struggling for two days. > >>>> And the only advice I get is to use arm64 machine. > >>>> Well, I don't have any. > >>>> And even if I had, I wouldn't use it, because I find this concept > deeply "misengineered". > >>>> > >> And it is bad engineering. Mark sometimes gives bad advice that's based > on his experience where it was "easier" to do it this way. I just wish I'd > replied more quickly. > >>> I know that you want to help, and I appreciate it. > >>> I just feel frustrated with the complexity of this conceptually > trivial task. > >> > >> No. I get it. I'm frustrated too since I've been working for years to > make it better, and there's still these rough edges that turn people off. > >> > >> Warner > >>> Regards, > >>> Michał Kruszewski > >>> > >>> > >>> Sent with Proton Mail secure email. > >>> > >>> On Saturday, September 13th, 2025 at 4:29 PM, Mark Millard < > marklmi@yahoo.com> wrote: > >>> > >>> > > >>> > On Sep 13, 2025, at 00:20, Michał Kruszewski mkru@protonmail.com > wrote: > >>> > > >>> > > Basically you are saying that I should have a separate physical > arm64 machine for doing embedded armv7 development on FreeBSD. > >>> > > If I understand this correctly, I just can't understand how can > people say that FreeBSD is embedded friendly. > >>> > > This is so much complex and convoluted compared to just using, for > example, Buildroot on Linux. > >>> > > This also doesn't feel like a *nix way of solving things. > >>> > > I thought that after cross compiling world and kernel, there is > some shell script that I can simply source to start cross-development for > target platform. > >>> > > >>> > > >>> > I found this AI based note about Buildroot's way of > >>> > putting cross-build toolchains to use: > >>> > > >>> > QUOTE > >>> > Buildroot pre-configures and compiles all necessarily > >>> > packages during the build process to create a custom > >>> > root filesystem. > >>> > . . . > >>> > Buildroot focuses on static builds, meaing all selected > >>> > packages are compiled and integrated into the final root > >>> > filesystem image during the build process. There is no > >>> > concept of installing or updating packages on the target > >>> > after the initial build. > >>> > END QUOTE > >>> > > >>> > This is certainly not what the FreeBSD ports tree is > >>> > set up for. Buildroot or analogous does not exist for > >>> > FreeBSD --or most *BSD or Unix-- to my knowledge. > >>> > > >>> > May be NetBSD's pkgsrc cross build support: > >>> > > >>> > > https://cdn.netbsd.org/pub/pkgsrc/current/pkgsrc/doc/HOWTO-use-crosscompile > >>> > > >>> > would be closer to how you like to work? (I've never > >>> > used such.) > >>> > > >>> > > Regards, > >>> > > Michał Kruszewski > >>> > > > >>> > > Sent with Proton Mail secure email. > >>> > > > >>> > > On Saturday, September 13th, 2025 at 12:23 AM, Mark Millard > marklmi@yahoo.com wrote: > >>> > > > >>> > > > On Sep 12, 2025, at 14:10, Michał Kruszewski mkru@protonmail.com > wrote: > >>> > > > > >>> > > > > A simple way is to set up an armv7 chroot / jail on an arm64 > host such as a Raspberry Pi (Apple devices don't work!). > >>> > > > > Then it's just like a native environment, but usually much > faster. > >>> > > > > >>> > > > > I don't have physical arm64 host. > >>> > > > > I also don't want to create a VM. > >>> > > > > The whole idea seems weird and unnatural. > >>> > > > > >>> > > > Ignoring the RPi4B detail: > >>> > > > > >>> > > > FreeBSD official armv7 port-packages are built this way > >>> > > > on arm64 hardware that natively supports user-space > >>> > > > armv7 code. (ampere* systems are used.) > >>> > > > > >>> > > > FreeBSD official i386 port-packages are built this way > >>> > > > on amd64 hardware that natively supports user-space > >>> > > > i386 code. > >>> > > > > >>> > > > No use of qemu variants of any kind: no attempt at > >>> > > > non-native-capable host environments. > >>> > > > > >>> > > > (Back when amd64 and qemu was used for the likes > >>> > > > of armv7, lots of stuff failed to build mcuh of > >>> > > > the time over the years --stuff that builds > >>> > > > just fine now. This was abandoned after native > >>> > > > became available. armv6 was always qemu based > >>> > > > on amd64 and could not build much as of the last > >>> > > > time a build was run.) > >>> > > > > >>> > > > > It sounds more like an exotic workaround. > >>> > > > > >>> > > > It is the official technique used for what FreeBSD > >>> > > > distributes for armv7 port-packages. > >>> > > > > >>> > > > > Regards, > >>> > > > > Michał Kruszewski > >>> > > > > > >>> > > > > Sent with Proton Mail secure email. > >>> > > > > > >>> > > > > On Friday, September 12th, 2025 at 10:46 PM, Robert Clausecker > fuz@fuz.su wrote: > >>> > > > > > >>> > > > > > Hi Michał, > >>> > > > > > > >>> > > > > > A simple way is to set up an armv7 chroot / jail on an arm64 > host such as > >>> > > > > > a Raspberry Pi (Apple devices don't work!). Then it's just > like a native > >>> > > > > > environment, but usually much faster. > >>> > > > > > > >>> > > > > > Yours, > >>> > > > > > Robert Clausecker > >>> > > > > > > >>> > > > > > Am Fri, Sep 12, 2025 at 04:19:42PM +0000 schrieb Michał > Kruszewski: > >>> > > > > > > >>> > > > > > > Is there any tutorial on how to cross compile custom user > application for armv7? > >>> > > > > > > I struggle t find any. > >>> > > > > > > Cross compiling the system is pretty easy. > >>> > > > > > > However, how can one cross compile custom user application > or kernel drivers. > >>> > > > > > > > >>> > > > > > > Regards, > >>> > > > > > > Michał Kruszewski > >>> > > > > > > > >>> > > > > > > Sent with Proton Mail secure email. > >>> > > > > > > >>> > > > > > -- > >>> > > > > > () ascii ribbon campaign - for an encoding-agnostic world > >>> > > > > > /\ - against html email - against proprietary attachments > >>> > > > > >>> > > > === > >>> > > > Mark Millard > >>> > > > marklmi at yahoo.com > >>> > > >>> > > >>> > === > >>> > Mark Millard > >>> > marklmi at yahoo.com > >>> > >> > > > === > Mark Millard > marklmi at yahoo.com > >