Creating armv7 MACHINE_ARCH

Mon Jun 19 01:22:20 UTC 2017

FYI: I'll provide the following example of the types of issues
that can show up in trying to support old 32-bit code on an
armv8 via AArch32 state mode.

This is from OpenSuse's https://bugzilla.opensuse.org/show_bug.cgi?id=1029158 :

Tom Rini wrote on 2017-03-13 15:11:25 UTC :

> If one wants to use legacy aarch32 binaries, the CONFIG_COMPAT option needs to be set in the kernel, and today we do this.  This in turn allows some aarch32 binaries to run easily.  However, there is a large world of software that is "armhf" but rather than being tuned for ARMv7 is tuned for ARMv6 (and the RPi).  In order for this to run ARMV8_DEPRECATED and then at least CP15_BARRIER_EMULATION needs to be enabled (the other 2 options would also likely be helpful).
> 
> The real-world example here is that at least today, the Docker "armhf" scratch builds (and in turn lots of stuff is based on this) do not just run and you will get for example:
> [ 1961.636326] python[3661]: undefined instruction: pc=00000000f6965dbc
> [ 1961.636336] Code: e5821060 e3a01000 e5821064 e5821068 (ee070fba) 
> (this is also true of busybox).
> And then with the options enabled:
> [   37.605796] "ls" (1464) uses deprecated CP15 Barrier instruction at 0xc95dc
> ...
> [   72.888570] "python" (1881) uses deprecated CP15 Barrier instruction at 0xf6ab4dbc

So full support can involve software emulation of
some instructions that otherwise would be classified
as illegal instructions or possibly as deprecated
instructions.

OpenSuse had a fix merged into the kernel near the
end of 2017-Mar.

===
Mark Millard
markmi at dsl-only.net

On 2017-Jun-12, at 10:37 PM, Russell Haley <russ.haley at gmail.com> wrote:

Thanks Mark! I'll chew on that for a couple of days.

:D
Russ

On Mon, Jun 12, 2017 at 7:01 PM, Mark Millard <markmi at dsl-only.net> wrote:
> 
> On 2017-Jun-12, at 4:10 PM, Russell Haley <russ.haley at gmail.com> wrote:
> 
>> Okay, feel free to ignore me, I'm not going to get the time drill into
>> the source code for my own questions so I don't expect anyone else
>> too. However, I'll ask anyway. I'm too confused to try and inline
>> these questions. Lets see if I understand:
>> 
>> - armv7 does not support 64 bit instructions (according to Wikipedia?
>> I claim no expertise.)
> 
> It does not support AArch64 instructions but does support AArch32
> instructions (or is a good approximation).
> 
> AArch32 was actually created later to be armv7-a like.
> 
>> - FreeBSD has an armv6 "architecture" that is supports armv6 and armv7
>> on Pre-Cortex-A-53 processors that is not supported on A-53 through
>> the emulated AArch32.
> 
> There are actually architectures (by ARM's
> definitions). . .
> 
> ARMv6, ARMv6T2, ARMv6Z, ARMv6K, ARMV6-M
> ARMv7-M, ARMv7E-M, ARMv7-R, ARMv7-A
> ARMv8-A, ARMv8.1-A ARMv8.2-Am ARMV8.3-A, ARMv8-R, ARMv8-M
> 
> I'm going to largely ignore much of
> that variation in structure.
> 
> ARM has continued to produce/make new variations for both
> 32-bit and 64-bit in overlapping time frames. So the
> "Pre-Cortex-A53" presumes to much about relative timing.
> 
> Technically the old RPI2s (V1.1 and before) have
> Cortex-A7's and the new V1.2's have Cortex-A53's.
> RPI3 is also Cortex-A53 based.
> 
> Cortex-A7 includes an implementation of armv7-A architecture
> (but is not the only one).
> Cortex-A53 includes an implementation of armv8-A architecture
> (but is not the only one).
> (Both include other things as well. And the System On A Chip has even
> more stuff than the Cortex-A<?>'s in question.)
> 
> Cortex-A<?>'s are specific CPUs/cores that are also
> examples of the specific architecture(s) they implement.
> 
> If a kernel supports armv7-a it can support a variety of
> CPUs that all implement armv7-a architecture. It may not
> support things in the CPUs that are beyond that
> architecture.
> 
> If a kernel supports armv8-a it can support a variety of
> CPUs that all implement armv8-a architecture. It may not
> support things in the CPUs that are beyond that
> architecture.
> 
> Note there are also issues of support of the System On
> a Chip involved as well. More than ARM is involved
> overall.
> 
> 
> 
>> - Cortex A-53 can support armv8 (AArch64) and armv7 (AArch32) instructions
> 
> As I remember armv8-a specifies:
> 
> A) Optional support: AArch64
>   (Cortex-A32 is ARMv8-A archtecture but only 32-bit)
>   (ARMv8-R architecture is always only 32-bit as I understand)
>   (ARMv8-M architecture is always only 32-bit as I understand)
> B) Optional support: AArch32
>   (Cortex-A35/53/57/72/73 are ARMv8-A and have both)
>   (Cortex-A55/75 are ARMV8.2-A architecture and have both)
>   (But variants can be produced that omit AArch32.)
> C) Various things specific to armv8-a
>   might go beyond what AArch64 and
>   AArch32 specify.
> 
> Note: AArch32 has 2 instruction sets
> (A32/ARM and T32/Thumb), like armv7-a
> does. AArch64 has A64. (I'm not giving
> thumb version numbers here but there
> are versioned vintages.)
> 
> So 1 architecture (armv8-a) has at least
> 3 instruction sets when both AArch64
> and AArch32 are implemented.
> 
> AArch64 state: A64
> 
> AArch32 state: A32 and T32
> 
> There can be context switching between
> these states as I understand.
> 
> (I'll not get into the NEON distinctions
> and such so the above is simplified.)
> 
> armv8-a does not necessarily uniquely identify
> the interrupt controller or its software
> interface, as an example of variation that is
> not an instruction set issue: register
> interfaces to other hardware. ARM provides
> GIC-400 and GIC-500 and others. But the
> controller does not have to come from ARM.
> 
> While Cortex-A53 has a default/reference
> interrupt controller(s), it is possible to
> build variations that use other ones as
> I understand.
> 
> Similar points go for Cortex-A7.
> 
> The other parts of a System On a Chip are for non-arm
> aspects and may have some mixed degree on commonality
> independently of ARM. But the kernel may deal with
> such specifics as well.
> 
>> - The current proposal is to split the armv6 and armv7 into their own
>> "architectures"
> 
> ARM defines these as distinct architectures.
> That is not a FreeBSD specific terminology.
> But much of the armv6 is common with armv7.
> 
> armv7 architecture has more than armv6 architecture does:
> it is an update. Having armv7 implemented separately
> in FreeBSD means more completely/correctly identifying
> what is there compared to pretending it is an armv6.
> 
>> FreeBSD has an Arm 64 bit kernel build. I don't see what the
>> TARGET_ARCH settings in the wiki and know little about it, but will
>> conjecture that it doesn't have a TARGET_ARCH=armv8 (please correct me
>> if I'm wrong).
> 
> FreeBSD has TARGET=arm64 TARGET_ARCH=aarch64 . That combination
> is for armv8-A with AArch64 currently. Cortex-A32 is not
> supported because it lacks AArch64.
> 
> FreeBSD's TARGET=arm64 TARGET_ARCH=aarch64 may grow to span
> armv8.1 or such as well for all I know. Cortex-A53 is one
> example of an armv8 implementation that the kernel supports
> (presuming certain interrupt controllers and such). There are
> others.
> 
>> What I was trying to ask was: is the kernel development moving in a
>> direction that clearly indicates the differences in the instructions
>> vs the architectures (or have I grossly simplified the problem)? Will
>> it be possible to target a Cortex-A53 and build for 32 or 64 bit
>> support? Or is this just to fix RPi?
> 
> Instruction sets are part of an architecture but not all of it.
> They are not independent of architecture. Talking of an overall
> architecture includes covering the instruction set(s). One
> architecture can have multiple instruction sets as part of it.
> 
> TARGET=arm64 TARGET_ARCH=aarch64 may get a "lib32" (compat32)
> implementation someday if someone is motivated to do the
> work. That would enable AArch32 user processes.
> 
> What exists now and will in the future is mostly tied to
> choices of volunteers that say "I'm going to provide
> <whatever>" and then manage to complete whatever it was.
> (Clearly a "we are going to" is also an option.)
> 
> A similar points would go for TARGET_ARCH=armv7 supporting
> something like Cortex-A53 in AArch32 mode (Raspian like).
> 
> So at this point no one knows if or when as far as I know.
> 
> armv6 and armv7 architectures are strongly overlapping for what
> is in armv6. The GENERIC kernel for what is currently a
> TARGET_ARCH=armv6 build actually builds for armv7 and will not
> work on armv6. One has to use a different kernel configuration
> than GENERIC for armv6 (such as the one for the older RPI vintage
> that was armv6 based).
> 
> Identifying an armv7-a build as a armv6 build has odd
> consequences for software that tries to figure out what
> can be depended on.
> 
>> In terms of Raspbian, I had assumed they were just supporting Aarch32
>> across both processor models. Many of the drivers would be the same
>> source if they share components so I would think it would be "simple".
>> I didn't see anything in my brief look at it today to indicate
>> otherwise.
> 
> For Cortex-A53: Raspian supports just AArch32 state,
> not AArch64 state for rpi2 V1.2 and RPI3. (I do not
> know if early boot is temporarily AArch64 or not.)
> 
> For Cortex-A7: Cortex-A7 predates AArch32 and is armv7-a
> directly but AArch32 was designed to appear to be be
> nearly the same as armv7-a. So Raspian is supporting
> armv7-a directly for rpi2 V1.1 and V1.0.
> 
> ===
> Mark Millard
> markmi at dsl-only.net
> 
> On Mon, Jun 12, 2017 at 2:07 PM, Warner Losh <imp at bsdimp.com> wrote:
>> 
>> 
>> On Mon, Jun 12, 2017 at 2:35 PM, Mark Millard <markmi at dsl-only.net> wrote:
>>> 
>>> 
>>> On 2017-Jun-12, at 1:00 PM, Mark Millard <markmi at dsl-only.net> wrote:
>>> 
>>>> On Mon, Jun 12, 2017 at 1:00 PM, Mark Millard <markmi at dsl-only.net>
>>>> wrote:
>>>>> On 2017-Jun-12, at 12:16 PM, Russell Haley <russ.haley at gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> On Mon, Jun 12, 2017 at 10:36 AM, Mark Millard <markmi at
>>>>>> dsl-only.net> wrote:
>>>>>>> 
>>>>>>> On 2017-Jun-12, at 8:39 AM, Warner Losh <imp at bsdimp.com> wrote:
>>>>>>> 
>>>>>>>> . . .
>>>>>>>> 
>>>>>>>> Plus, we aren't quite doing what Ian wanted. He wanted a full
>>>>>>>> rename. The
>>>>>>>> proposal on the able is to add an armv7 TARGET_ARCH in 12. Not to
>>>>>>>> rename or
>>>>>>>> remove armv6. Sadly, that will still be there since the RPI
>>>>>>>> foundation
>>>>>>>> keeps finding new ways to repackage the rpi into new boards that are
>>>>>>>> just
>>>>>>>> too cheap to ignore.
>>>>>>> 
>>>>>>> On 2017-Jun-12, at 6:59 AM, Andrew Turner <andrew at fubar.geek.nz>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> I like this. My understanding is adding armv7 would also fix many of
>>>>>>>> the currently broken ports that assume they are being built for armv7 as
>>>>>>>> many Linux distros target ARMv7+.
>>>>>>>> 
>>>>>>>> It should also be noted the GENERIC kernel is likely to only ever
>>>>>>>> target ARMv7+ even without an armv7 TARGET_ARCH.
>>>>>>> 
>>>>>>> 
>>>>>>> Hopefully the choices related to TARGET and TARGET_ARCH
>>>>>>> for armv7 end up identifying the context to port builds
>>>>>>> so that many would just automatically do the right thing.
>>>>>>> 
>>>>>>> 
>>>>>>> As for GENERIC:
>>>>>>> 
>>>>>>> powerpc has. . .
>>>>>>> 
>>>>>>> TARGET=powerpc TARGET_ARCH=powerpc   and GENERIC
>>>>>>> TARGET=powerpc TARGET_ARCH=powerpc64 and GENERIC64
>>>>>>> 
>>>>>>> So there is precedent for more than one GENERIC*
>>>>>>> for a family, with which one being appropriate
>>>>>>> being based on TARGET_ARCH.
>>>>>>> 
>>>>>>> For powerpc TARGET=powerpc implicitly uses
>>>>>>> TARGET_ARCH=powerpc when TARGET_ARCH is not
>>>>>>> specified (if I remember right). Which should
>>>>>>> be the default for armv6 vs. armv7 might go
>>>>>>> the other direction (TARGET_ARCH=armv7 by
>>>>>>> default).
>>>>>>> 
>>>>>>> 
>>>>>>> Side note:
>>>>>>> 
>>>>>>> A caution about talking about "rpi2" as
>>>>>>> an example. . .
>>>>>>> 
>>>>>>> Raspberry Pi 2 Model B V1.2 is Cortex-A53 based
>>>>>>> (so arm64/aarch64). (A BCM2837, not a BCM2836.)
>>>>>>> This dates about to something like 2014 based
>>>>>>> on the pictures showing the (c) notice on the
>>>>>>> boards.
>>>>>>> 
>>>>>>> V1.1 and before were armv7 (BCM2836) based.
>>>>>>> 
>>>>>>> Unless a kernel and world are made that can
>>>>>>> also configure/handle a Cortex-A53 in a
>>>>>>> armv7-like manor there will be two different
>>>>>>> GENERIC builds in order to span the "rpi2"
>>>>>>> family, based on just V1.2+ vs. V1.1 and
>>>>>>> before.
>>>>>>> 
>>>>>>> (A single, modern distribution of the official
>>>>>>> Raspbian software for the rpi2 does support
>>>>>>> all the V1.x boards if I understand right.)
>>>>>> 
>>>>>> I am confused. I don't see any documentation about Raspbian supporting
>>>>>> 64 bit?
>>>>> 
>>>>> 64-bit Cortex-A53's can be configure to operate in a
>>>>> 32-bit mode (AArch32). Raspian does that for RPI2 V1.2
>>>>> and for RPI3.
>>>>> 
>>>>> Raspian does not (yet?) support a 64-bit mode (AArch64).
>>>>> 
>>>>> The Cortex-A53 can support either. As I understand it
>>>>> is possible for an OS to allow a user processes to be
>>>>> one or the other, different processes using the different
>>>>> modes. That does not mean that all operating systems
>>>>> bother to.
>>>>> 
>>>>> If I remember right the official Ubuntu for an ODroid-C2
>>>>> allows both AArch64 and AArch32 user programs (and
>>>>> so processes, with shared library types tracking).
>>>>> 
>>>>>> From Arm at
>>>>>> https://www.arm.com/products/processors/cortex-a/cortex-a53-processor.php:
>>>>>> "The Cortex-A53 supports the full ARMv8-A architecture. It not only
>>>>>> runs 64-bit applications also seamlessly and efficiently runs legacy
>>>>>> ARM 32-bit applications."
>>>>>> 
>>>>>> I assume that means it handles armv7-A without issue? (In fact, many
>>>>>> on this board know it does)
>>>>> 
>>>>> I've not gone through the details but targeting AArch32
>>>>> probably means targeting a subset of armv7. Or may be
>>>>> to support both requires targeting a common subset of both.
>>>>> (My guess is that AArch32 is the definition of a common
>>>>> subset for 32-bit, at least for user processes.)
>>>>> 
>>>>> Raspian targets just AArch32 on armv7 and Cortex-A53
>>>>> (user space). (If I've got the definition of AArch32
>>>>> right: otherwise a common subset.)
>>>>> 
>>>>> FreeBSD targets armv7 and AArch64 separately (via
>>>>> separate GENERIC kernels). I'm not aware of FreeBSD
>>>>> targeting AArch32 at all on cores capable of AArch64.
>>>>> FreeBSD possibly does not restrict itself to AArch32
>>>>> (user processes) on armv7 if AArch32 is really a
>>>>> subset.
>>>> 
>>>> I thought all 64 bit Arm instructions are defined in armv8?
>>> 
>>> (I assume you are not trying to indicate armv8.1, armv8.2
>>> and such. Cortex-A53 is older than those and so does not
>>> have the newer things involved.)
>>> 
>>> That Cortex-A53 allows armv8 64-bit arm code is not in
>>> dispute. But the operating system in involved in setting
>>> up what will actually work based on how it configures
>>> things and operates. Much of this is the kernel.
>> 
>> 
>> Correct.
>> 
>>> 
>>> Cortex-A53 also supports AArch32, i.e., 32 bit instructions.
>>> So that the 64-bit instructions all being there does not
>>> of itself prevent using a 32-bit mode instead.
>>> 
>>> (The kernel likely has to deal with more specifics of
>>> processor variations than user code does not. My notes
>>> are really about the user process model, not the all
>>> the kernel details.)
>>> 
>>> Raspian deals with armv7's that have no 64-bit support
>>> and with Cortex-A53's that do. It presents a user-process
>>> model that is 32-bit only, even on Cortex-A53's that allow
>>> for 64-bit (but do not required user programs to be AArch64
>>> code).
>>> 
>>> Ubuntu for ODroid-C2 does not deal with armv7's but does
>>> allow both 64-bit (AArch64) and 32-bit (AArch32) user
>>> processes as I remember, on its Cortex-A53's.
>>> 
>>> FreeBSD armv7 does not support Cortext-A53 or anything
>>> that allows 64-bit (that allows AArch64). This is a kernel
>>> level issue.
>> 
>> 
>> Not a hugely difficult issue to fix, but one nobody had fixed...
>> 
>>> 
>>> FreeBSD aarch64 does not support having AArch32 user
>>> processes. Nor does its kernel support processors that
>>> do not support aarch64 (so it does not support armv7).
>> 
>> 
>> Executing a 32-bit /bin/cat on aarch64 level support exists outside the
>> tree, according to the hallway track at BSDcan, so it will only be a matter
>> of time before compat32 exists there I think.
>> 
>> There's a further complication is that the aarch32 unit of aarch64
>> processors is optional. Not all of them have it, so that can be a problem...
>> IIRC, the early aarch64 targets didn't have this feature...
>> 
>>> 
>>> These are simply examples of different choices about
>>> what combinations of the technical possibilities to
>>> put effort into supporting in the kernels (and
>>> possibly elsewhere). None of the alternatives is
>>> automatic. None are independent of software choices
>>> that must be made by each operating system.
>> 
>> 
>> Yes. They all require somebody to be interested in doing the work.
>> 
>> Warner
>> 
>> 
>> 
> 