INTRNG (Was: svn commit: r301453....)

[Nathan Whitehorn]

On 08/06/16 09:58, Warner Losh wrote:
> On Sat, Aug 6, 2016 at 10:44 AM, Nathan Whitehorn
> <nwhitehorn at freebsd.org> wrote:
>
>> Fair enough! I don't think we need that for, e.g., GPIOs (see cases 1-2
>> above), just for bus enumeration schemes (ACPI, OFW are probably the only
>> ones) that usually require a ton of this kind of thing anyway. But,
>> fundamentally, it doesn't matter. There are three important things from my
>> end:
>> 1. That it is possible to, at bus enumeration time, permanently assign an
>> IRQ to an interrupt specifier from OFW/ACPI.
>> 2. That that assignment not depend on having the PIC attached yet.
>> 3. That the implementation details of that mechanism be reasonably
>> abstracted so that they can change later or vary platform to platform.
>>
>> Whether mapping tables are in some central place (subr_intr.c) or in the
>> parent bus, how the PIC API works, whether they are stored in that table in
>> the form of a union or in different tables, doesn't matter for those three
>> at all. And, with a constant API (3) we can even change our minds later
>> without a lot of hassle.
> First, I hate mapping tables at the nexus, unless they are created
> dynamically at run time. There's too much variation between boards,
> SoCs, etc to have that code live in the nexus otherwise. They simply
> don't scale. This board has interrupts 1-16 wired this way, but that
> board didn't do that and has an external PIC. This SoC based on
> Cortext A<whatever> uses the GPIC, while that one based on the
> same Cortext A<whatever> chose to use Atmel's PIC. Perhaps I'm
> misunderstanding something here as to what is meant by a table
> though.

The table in question is just a mapping of abstract IRQ numbers to the 
corresponding interrupt specifier and parent.

For example,
5: <&gicX, 7 2>

Where 5 is the (potentially arbitrary) assigned number for the system 
corresponding to whatever <7 2> means on gic0. The table is built on the 
fly so that OFW bus nodes can specify interrupts as something <&gicX, 7 
2> and get back scalars that work with rman and the PCI APIs and all the 
other places that want definitions of interrupts as scalars. It 
naturally handles all of your examples.

The discussion here is whether to:
a) keep an OFW/FDT-specific table like this in ofwbus.c (and analogs for 
ACPI, etc. in their respective places)
b) keep a table that maps numbers to some union of OFW/ACPI/whatever 
data in nexus or intr.c or some global place

It doesn't matter much from a functionality perspective (or an API 
perspective) either way. The question is more about which is easier to 
maintain and extend long-term. Since the API doesn't change either way, 
we're free to decide incorrectly and revisit it at will later with 
little consequence.

> Next, In your list there's another dependency that's implicit
> but maybe not called out. You can have PICs that cascade into
> other PICs, or GPIO controllers that need to enable external
> PIC-like things before they can route interrupts from things
> that are downstream (interrupt wise) from them. Maybe I'm
> just hung up on the phrase "the PIC" and it really means
> "whatever complex thing or things handles getting the
> interrupt routed to the CPU." I don't see this design so much
> on basic eval boards, but do see it in more complex boards
> that control complicated things.

Yes. We've supported this forever on PPC (where it is very common) and 
it works here too. The implementation is really simple.

First, here's an explanation of the general interrupt mapping system, 
for context:

Let's suppose you have some interrupt hierarchy like this:

CPU <- pic0 <- some device

PIC0 has a bunch of interrupt pins. As you enumerate the system, the 
code encounters interrupt specifiers like <&pic0, 3 1>, <&pic0, 4, 1>, 
<&pic1, 3, 2>. Through OFW_BUS_MAP_INTR(), the bus enumeration code 
turns these into some arbitrary scalar IRQs assigned by 
OFW_BUS_MAP_INTR(). The mapping from the (interrupt parent, specifier) 
tuple to that assigned scalar IRQ is stored somewhere (see above) for later.

When the bus devices that use those IRQs attach, they call 
bus_activate_resource() and bus_setup_intr(). At this point, some code 
close to the root of the hierarchy (again, see above for exactly where) 
looks up the corresponding vector specifier in the table, looks up the 
registered PIC corresponding to the interrupt parent key, and tells the 
driver attached to the interrupt parent to think of the scalar IRQ as 
meaning whatever string of numbers it saw earlier. It also tells the 
machine-dependent interrupt layer that the given scalar IRQ is handled 
by the device_t for the PIC so that it can ask the PIC to 
mask/unmask/bind/etc. the interrupt.

When PICs attach, they register themselves with whatever bus layer is 
doing this mapping to say that a given interrupt parent key (the phandle 
for OFW/FDT) corresponds to the device_t for the PIC. The PIC attached 
directly to the CPU's interrupt pin[s] (pic0 here) also registers itself 
somehow with the MD interrupt system. On PPC, there is a global device_t 
called "root_pic" that the driver sets.

When "some device" signals an interrupt, pic0 (the hardware) signals the 
CPU. The CPU signals the kernel. The MD interrupt layer notices and 
calls the driver for pic0 (the root PIC). That code inspects whatever 
registers on PIC0 give it the interrupt line and then signals the MD 
interrupt layer (let's say by calling a function called 
arm_dispatch_irq(int irq)) with the corresponding scalar IRQ. That then 
invokes the corresponding filters and/or ithreads.

So how does this change with cascaded PICs? Very little. Let's suppose 
you have some interrupt hierarchy like this:

CPU <- pic0 <- pic1 <- some device

PIC0 has a bunch of interrupt pins, one of which is connected to pic1, 
which provides a bunch more interrupt pins.
Both pic0 and pic1 register themselves with the bus layer by the 
appropriate handles and get their mask/unmask/bind functions called by 
the interrupt layer during interrupt setup for interrupts they handle. 
PIC0 registers itself as the root during attachment because it notices 
that it does not have any interrupts in its resource list. Child PICs, 
on the other hand, have interrupt properties in the FDT corresponding to 
the pin on pic0 that is attached to pic1. These get assigned as normal 
through newbus and the child PIC (pic1) calls bus_setup_intr and 
registers a filter handler.

When "some device" signals an interrupt, pic1 drives a line on pic0, 
which drives a line on the CPU. The driver for PIC0 calls 
arm_dispatch_irq() with the IRQ corresponding to pic1, which calls 
pic1's interrupt handler. Since filter handlers run in primary interrupt 
context, pic1's interrupt handler can look exactly like what pic0 does 
when signalled by the interrupt layer: it runs through its registers, 
finds any lines with active interrupts, then calls arm_dispatch_irq() 
with the appropriate corresponding scalar IRQ.

The nice thing here is that cascaded PICs require zero special handling 
by the system. You can just treat them as normal interrupts, with no 
special methods required in the bus layer, the interrupt system, or the 
PIC driver.
-Nathan

>
> Generally, though, I like the direction things are going.
>
> Warner
>