ocpbus(4)

[Marcel Moolenaar]

On Jan 1, 2008, at 5:49 PM, M. Warner Losh wrote:

> : > I disagree with you that it is a fundamental flaw in the system,  
> but
> : > am always open to better suggestions at how to implement this  
> stuff.
> : > The hints were my first, best guess, and they have served us well.
> :
> : Yes, but...
> :
> : In the beginning, hints were used to let drivers know where to look
> : for hardware. They were hints to the drivers so that drivers knew
> : where to probe. Possibly before probing a fixed list of I/O ports
> : where the ISA card could be configured to reside.
>
> In the beginning (when newbus was introduce circa 3.0) they were used
> by the busses to assign resources to the drivers they were adding.  It
> just turned out that the 'hints' were compiled into tables the drivers
> used.  In 4.x, the hints migrated from the config file to a text file
> called typically device.hints.  Throughout these underlying changes,
> the drivers have never really known about resource hints.  The bus is
> the only thing that parses and consumes them (the busses were changed
> from using the compiled in tables to using hints in 4.0).  The ISA bus
> used to be the only one that looked for the 'at' key with a value of
> 'isa' or 'isa0', and it used that to populate the children on the isa
> bus.  The drivers whose nodes are added to the tree as hints have no
> idea that the resources came from hints or from some other enumeration
> mechanism.  They just call bus_alloc_resource and they get the
> resource.

> The removal of probing a fixed list of I/O ports happened at the same
> time that newbus was introduced into the system.  There may have been
> one or two stragglers after we did the bulk of this work, but nearly
> all of it was done when newbus was committed.
>
> See devinfo -v on my system:
>          isa0
>            ppc0
>            sc0
>            sio1
>            sio2
>            sio3
>            vga0
>            orm0
>
> I do not have the sio or ppc drivers in my kernel at all, yet they
> appear to be bound to nodes in the tree by virtue of the hints that
> are present in device.hints.  This is what allows the drivers to
> attach to nodes in the system if I were to dynamically load them.

Let me use the above to better express my philosophical approach:

The fact that that devinfo shows you that non-existent drivers are
bound to newbus nodes is exactly the problem with hints. Pre-newbus
the kernel configuration defined the instantiations and as such the
list of drivers that were needed. Both (i.e. the drivers and their
instantiations) came from a single description and could never be
out of sync.
Nowadays we have the drivers coming from the kernel configuration
(or dynamically through module loading) and instantiations coming
from the hints file. These two are typically out of sync with each
other. If hints were tentative, then we wouldn't have newbus nodes
for driver instantiations if the driver in question didn't exist.

It's fundamentally wrong to instantiate a driver that doesn't exist.
I mean, what does that mean anyway?

> : The key point of that statement is that the drivers were already
> : there. They were compiled-in by virtue of the kernel configuration
> : that already included bus attachment information and resource lists.
> : The hints didn't cause the instantiation of a driver!
>
> Actually, they did.  You are confusing the code that's in the kernel
> to device nodes in the tree.  The hints will *ALWAYS* be there, even
> when the appropriate driver isn't compiled into the tree.  The device
> nodes in the device tree will be there, unattached.

Yes, you're right. I misspoke. The hints caused the instantiations.
See also above.

> : This is not what's happening now. Hints cause driver instantiations
> : and as such are much more significant than originally designed.
> : They stopped being hints and turned into something much less well-
> : defined.
>
> What is happening now is what has always happened.

As per above: previously the driver was guaranteed to exist, this is
not the case anymore, so the meaning of hints was changed in a subtle
way, not because we redefined them, but because we changed the
environment in which they were used unchanged. We never accounted
for that as far as I can see.

> : >  We
> : > are at the point now where we must expand them either as John has
> : > outlined, or we need to scrap them entirely in favor of something
> : > better.  What we cannot do is let the promise of something  
> better that
> : > isn't really being worked on stop us from expanding what we have.
> :
> : It has nothing to do with the promise of something better. It
> : has everything to do with understanding of the problems.
> :
> : Put differently: extending what you have because you don't want
> : to put in the effort of understanding the problem and working out
> : how such could be solved is ultimately worse, because you end up
> : with something that's not broken enough to fix but which will
> : always cause you problems -- problems you can only accept because
> : the alternative was not even considered.
>
> We can't let your aesthetic objections to the system that's basically
> working today block further efforts.  I hate to be so blunt about it,
> but we as a project have gone more with a rough consensus and working
> code over a pie in the sky perfect design with no implementation.

Again: it has nothing to do with pie in the sky. It has everything to
do with getting the concepts right and then deciding how we want to
implement.

I didn't write uart(4) because I liked doing it. I wrote uart(4) because
sio(4) is unportable and encumbered. It's been hacked to solve problems
it shouldn't have to solve. Likewise for geom_part. I don't like doing  
it,
but it's obvious that the minimal amount of effort put into solving the
problem right has resulted in an abundance of flaky tools and something
horrendous as libdisk. Whenever FreeBSD is ported to a new platform, the
biggest hurdle isn't so much the MD code; it's coding the partitioning
of disks so that you can install with sysinstall.

I'm saying this because sooner or later someone will have to overhaul
code to make it work in new environments and one of the problems of
working on non-i386 is that you end up being the one to do it. I'd like
to minimize the change of me having to overhaul this too...

All I'm saying is: define the problem. We can still decide to partly
solve the problem and extend hints. If anything, do it to prove me
wrong :-)

> Having said that, I'm willing to consider an alternative, but it has
> to handle the functionality that hints handle today.

Only fair.

> : > Do you have something specific in mind that we can start working  
> on
> : > some kind of design document against?
> :
> : I don't have a design document, but I have mentioned in various
> : emails what I think we should be doing. Unless there's sufficient
> : agreement on the rough course of action will I be working on
> : details.
> :
> : >  It is OK if that's a series of
> : > conversations, but we have to get off the inflection point we're  
> at
> : > now: No, you can't expand what's there, and there's no clearly
> : > articulated 'something better'.
> :
> : I have been conversing. I have been expression my point of view.
> : What I have not seen, until now, is an actual statement of intend
> : to just look at it from a different angle and see if we can come
> : up with some set of requirements. Everything so far has been
> : pivoting around hints. We have them and we're extending them.
> : What about letting the discussion pivot around the problems we're
> : trying to solve and once we have that fleshed out see if and how
> : we can use hints before we realize that it's better to start
> : from scratch?
>
> The reason for that is that hints are basically working.

I disagree. They're simply not causing problems big enough for people
to notice or care about. There are many little things that are
fundamentally flawed and that have been plastered over to sort-of
work. I think we can do much better than that, especially if we're
going to build another floor on this house of cards.

> : > So if you could take a few minutes and write up a straw-man  
> proposal
> : > for what you see as doing it right, I think that would get us  
> toward a
> : > solution...
> :
> : As said previously in many of the emails on the subject: We need to
> : seperate out the different uses or needs and come up with a layered
> : solution of some sorts:
> : The lowest layer is enumeration of the hardware. This, typically, is
> : a combination of hardware self-enumeration, firmware-based  
> enumeration
> : or pure software-based enumeration (i.e. the proverbial hardcoded
> : tables).
> :
> : On top of the hardware layer we have the newbus infrastructure layer
> : that, given a view of the hardware provided by the hardware layer,
> : constructs the newbus device tree and provides for wiring before the
> : generic probe/attach sequence starts.
> :
> : On top of the newbus layer are the devices themselves. They use the
> : newbus layer to obtain their resources and fetch device-driver
> : specific settings.
> :
> : For the hardware layer we need a description of the hardware. The
> : keying information is resource location or device path.
> :
> : The newbus layer needs a description of the wiring and/or routing.
> : The keying information is resource location or device path.
> :
> : The device layer needs a description of per-instance driver options.
> : The keying information is resource location or device path, combined
> : with driver and unit number.
>
> I think you are mixing layers here.  The newbus layer is in charge of
> assigning unit numbers.

You misunderstand. I'm not mixing things up. The newbus layer is  
assigning
the unit numbers, but it's not a key at that layer. It's a key at the
layer above. At the newbus layer it's just a property.

In fact, it's probably better that the unit number is not part of the  
key,
because the unit number doesn't mean anything. Its the device path with
driver combination that's important (see also below).

>  The driver shouldn't know anything at all
> about device paths or resources that bind it to a unit.

Correct. The newbus layer abstracts it. This is not to say that
the information specific to the device layer (or should I say
driver layer) is not to be keyed on the device path.

Take for example the following case:
In a computer are 2 serial ports. One is using a standard chips and
the other is using one with a non-standard FIFO size. The driver
needs to know the FIFO size to avoid overrunning the Tx FIFO. So, for
each instance of the driver there needs to be a property that describes
the FIFO size. Since the FIFO size is a property of the hardware and
not of the driver instance and since the need for a FIFO size property
is dependent upon the driver, the key is {device path, driver}. This
is conceptually. It does mean that the driver knows about device paths.

>  It is told
> 'probe this node and your unit number is X' and the driver has to like
> it.  There's a kludge in place now that I've removed in my tree for
> sio so that the serial port would get the right unit number, but that
> has always been considered an aberration that should be eliminated at
> the first change possible.

The need for the kluge is another sign that hints are misused: you
don't specify which driver instance is the console, because that can
change. You specify which physical port is to be used as the console
and have the driver instance that maps to that port do the right thing.
That's why uart(4) doesn't need to be kluged and why uart(4) works
when the console is defined by ACPI tables and DIG64 HCDP tables or
that it respects the OpenFirmware settings...

> : Concretely:
> :
> : A simplified hardware description could be (key=ioport):
> : 	ioport=0x3f8 -> iosize=8, type=UART, irq=4
> :
> : A description for prewiring could be: (key=ioport):
> : 	ioport=0x3f8 -> driver=sio, unit=0
> :
> : A description for the driver could be (key=ioport_driver+unit):
> : 	ioport=0x3f8, driver=sio, unit=0 -> fifosize=32
> :
> : A serial console is defined alongside the hardware description.
> : You don't define the hardware, so it shouldn't be part of it.
> : You do describe intended use of the hardware, so it's should
> : be in the hardware layer where you define it. As such, the
> : serial console is keyed off of the ioport in the example above:
> : 	ioport=0x3f8 -> what=console, baudrate=9600
> :
> : This all makes it possible to replace the sio driver with uart
> : and have a different set of driver descriptions for the same
> : hardware:
> : 	ioport=0x3f8, driver=uart, unit=any, clock=1843200
>
> I think you are mixing the levels of abstraction again.  We don't want
> the hardware locators to be used once the newbus bus has assigned a
> unit and driver to the device.

It's not mixed. It's conceptual. The newbus layer abstracts the
details from the driver so that the driver doesn't have to know.
Think IVARs. The driver doesn't know where they came from. It's
the newbus layer that tied the information to the device node.

>  This is the main problem that I have
> with this design: it blurs the line between the hardware location and
> other ways of locating a device.

No it doesn't. I described a concept. The problem is that it looks like
you interpret it in term of an implementation. In fact, that's the
whole problem in the discussion about hints: people don't rise above
the implementation. Everything I say, no matter how abstract, is
interpreted from an implementation point of view. No wonder we're not
getting anywhere :-)

Seriously: When a piece of information applies to a layer, it does not
mean that there's code that belongs to that layer that gropes in some
data structure that implements the information. It means that the infor-
mation has meaning to that layer and that access to that information is
abstracted by the layers underneath.

> : You also know that competely removing the newbus information
> : (i.e. the prewiring) will not affect the view of the hardware.
> : It will only affect which drivers get attached and which units
> : are being assigned.
> :
> : You also know that if the (view of the) hardware changes, your
> : newbus description will match where possible and be ignored
> : where appropriate. This mean that if you previously had a newbus
> : description that prewires a network interface to, say, bge2 and
> : you removed the interface, you simply wont use the prewiring.
> : It will later be used when the interface is added back.
> :
> : Apply RDB theory and you can start constructing the tables.
> :  From the tables you can derive a more suitable way of specifying
> : the information and having it used early on by a kernel.
>
> I need more details than this.  I hate to keep pressing for more
> details, but I think that your view of the world leads to the same
> blurring of abstraction levels that we have with hints today.

I don't mind explaining myself in more detail or more concretely.
Unfortunately, I do like to get something back: I would like people
to test various cases and let me know if there are cases I haven't
thought about or that I'm overlooking. The more feedback about the
concepts the better I can fill in the gaps...

> : Another thing to keep in mind is that any low-level console,
> : whether serial or otherwise and whether using legacy devices
> : or USB or firewire needs access to the hardware before we
> : have newbus up-and running. That's why the console specification
> : would be in the hardware decsription layer: you don't have
> : newbus, no prewiring and no assigned devices.
>
> If we can make consoles regular in the system, so much the better, but
> we shouldn't let their needs drive the design.  Consoles are a special
> case today, and if they remain a special case, then I'm OK with that.

uart(4) does this correctly already. It is a special case no matter
how you look at it, but it needs to be made a special case throughout
the layers to get the proper (abstract) support to avoid it being a
kluge. We have it in place already for uart(4) so the only thing
we need to avoid is breaking it.

> : Moving forward, we should take OFW or EFI as an example of
> : how you can describe hardware and morph that into something
> : that allows us to:
> : 	o  compile into some binary form for compactness
> : 	o  can be loaded as a module or specified at the
> : 	   loader prompt.
> :
> : Once we have that, newbus descriptions (for prewiring) are
> : trivially added. Driver descriptions (or more accurately
> : driver directives) should be more like var=value lines because
> : of the wide and unstructured nature of what drivers may need.
>
> So the only significant change from the current hints scheme is just
> the key lookup for the device?

The key lookup makes all the difference, but yes, if we get the
keying right and find that we don't have to abandon the hints
concept as it is, then all the better and I don't see a reason
for overhauls.

>  right now the only key you can use is
> the device name and unit for a given variable.  Your scheme would
> allow for more types of lookups than just the one key we currently
> provide.  Or put another way each device node in the newbus tree would
> have some attributes.  One would use those attributes, in addition to
> the name/number to look up other attributes of the device.

The information available at each level differs. At the hardware
level there's no notion of newbus or drivers. The hardware exists
independent of that. So, "hints" that apply to the hardware layer
have no correspondence to newbus nodes and/or driver instances.
As such, you cannot use driver names or unit numbers as a key. The
one thing that is known to be unique is the (primary) I/O address
(either port or memory mapped) or alternatively, a device path
of some sorts.

In any case, you cannot lookup the hardware information with a key
that consists of driver+unit, unless the hardware is attached by
the driver. After the attach of a driver, the driver+unit, unique
by definition, can be used as a secondary key. This secondary key
is removed and meaningless when the driver detaches again.

As such, the secondary key cannot be stored on disk. Only the I/O
resource or the device path is immutable and thus safe to be used
across reboots.

Wiring of devices can then be defined as a pre-determined mapping
between the primary key and secondary key, provided the driver
actually exists. Without wiring the secondary key is mapped by
virtue of the newbus probe and attach methods.

Drivers don't need to know the primary key, because the secondary key
is mapped to the primary key so you can get the hardware information
by using the secondary key as well. This of course is abstract, not
something that's done in actuality this way (I presume)...

> I'll be out in the bay area in a couple of weeks.  I think we're doing
> some talking past each other in this email exchange.  Maybe we could
> get together while I'm out there and go over your ideas and see how we
> can move to that kind of system.  I think this might help us get the
> ideas into a more concrete form than doing it just through email.  Are
> you up for that?  If so, I can send you the dates I'm available...

That sounds like an excellent plan. I'm sure we can find a couple
of hours to flesh this out a bit...

-- 
Marcel Moolenaar
xcllnt at mac.com