Partial cacheline flush problems on ARM and MIPS

[Warner Losh]

On Aug 27, 2012, at 9:20 AM, Adrian Chadd wrote:

> On 27 August 2012 08:12, Tim Kientzle <tim at kientzle.com> wrote:
>> But don't mbuf structures do pretty much what Hans is suggesting?
>> 
>> Why is mbuf okay?
> 
> Which part of mbufs?
> 
> I think the difference here is that there's no concurrent access. Ie,
> although you may have an mbuf header + data inside the same cache
> line, you wouldn't go fondling the mbuf once it's handed to the
> hardware.
> 
> But I was under the impression that mbuf + mbuf buffers are already
> correctly aligned.

Not quite.  mbufs are large enough and aligned enough so that no two mbufs are in the same cache line.  The data inside the mbuf on 32-bit platforms is 24 bytes from the start of the mbuf, so the mbuf data and mbuf header do share the same cache line, but as you say, nobody touches any part of the mbuf will the hardware has it. Recall that it isn't alignment of where the buffer starts that matters here, but rather alignment to ensure that there's no sharing of cache lines between structures.

> This all honestly looks like a very i386-centric interpretation of the
> busdma "intention", which I think illustrates a need to better
> document what assumptions busdma actually makes.

Yes.  x86 lets you play fast and loose with many of these things.  We've tried to accommodate this for a long time, but that is lame.

> That does remind me, I think the ath(4) driver does the same (since it
> allocates its own descriptor block and then treats it as an array of
> descriptors for the hardware to access) - I should ensure that
> sizeof(ath_desc) is aligned on the relevant architecture. It gets
> slightly scary - AR93xx TX descriptors are "L1 cache == 128 byte
> aligned" which is an enormous waste of memory compared to a 16 or 32
> byte aligned platform. Alas..

The problem is with cache line sharing, not necessarily with alignment.  If you are only ever using one of them at a time, or if you have perfect hygiene, you can cope with this situation without undue waste.  The perfect hygiene might be hard sometimes.

Warner