How's bus-space stuff supposed to work with superscalar MIPS?

[Warner Losh]

On Oct 5, 2013, at 5:51 PM, Adrian Chadd wrote:

> On 5 October 2013 16:06, Stanislav Sedov <stas at freebsd.org> wrote:
> 
>> 
>> On Oct 5, 2013, at 10:18 AM, Adrian Chadd <adrian at freebsd.org> wrote:
>> 
>>> Hi all,
>>> 
>>> I've been bringing up the AR9344 PHY and after a lot of digging, I
>>> discovered that I can fix things by changing ARGE_WRITE() (ie, write to
>> the
>>> ethernet space registers) to:
>>> 
>>> bus_write_4();
>>> bus_read_4();
>>> 
>>> .. to (what I'm guessing here) flush the write out before the next
>>> instruction is run.
>>> 
>>> So, given this particular hilarity has shown up, what's the story with
>>> doing IO accesses on a superscalar MIPS CPU? If it's going to kseg1, is
>> it
>>> somehow going to magically enforce ordering? Or am I right in thinking we
>>> will need explicit barriers here?
>>> 
>> 
>> I don't know specifics of mips74k, but usually one indeed needs memory
>> barriers
>> when performing read of write operation sequences that require ordering on
>> device I/O (e.g changing the ring and writing the new ring index
>> afterwards).  I would
>> not be surprised if the cpu reorders i/o bus memory access, especially a
>> multi-issue
>> one.
>> 
>> It is a good idea to have barriers where needed regardless.  We have
>> special macros
>> for them which are defined to nothing on the in-order platforms.
> 
> 
> Right. I know this stuff. I really though want to know this kind of stuff:
> 
> * What the specifics are for superscalar MIPS CPUs;

I believe they document that writes can be reordered unless there's an intervening read or memory barrier. I've not looked it up.

> * What the bus space stuff should be be providing by default (and I've been
> down this path once, with ath(4) bugs, PPC, and the bus space macros not
> enforcing flushes after IO operations, even though the API requires drivers
> do it themselves..);

It isn't so much flushes as barriers to prevent reordering. By doing the read after write, you are forcing an expensive memory barrier. Drivers that depend on a particular write ordering need to have explicit barriers.

> * Whether it should be enough to map space COHERENT - then it's up to the
> underlying bus implementation to implement enforcing ordering.

The question here is whether there should be an implied barrier in write operations. On x86 there is, but as you are discovering on other architectures there isn't. While it would be convenient to force a memory barrier between every write (something trivial to do with an explicit barrier in your driver), it is not very performant to do so, since most writes don't have an explicit ordering...

Warner