Partial cacheline flush problems on ARM and MIPS

[Warner Losh]

On Aug 27, 2012, at 9:12 AM, Tim Kientzle wrote:

> 
> On Aug 27, 2012, at 6:38 AM, Warner Losh wrote:
> 
>> 
>> On Aug 27, 2012, at 12:06 AM, Hans Petter Selasky wrote:
>> 
>>> Hi,
>>> Correct.
>>> 
>>>> We also need some rules about working with buffers obtained from
>>>> bus_dmamem_alloc() and external buffers passed to bus_dmamap_load().  I
>>>> think the rule should be that a buffer obtained from bus_dmamem_alloc(),
>>>> or more formally any region of memory mapped by a bus_dmamap_load(), is
>>>> a single logical object which can only be accessed by one entity at a
>>>> time.  That means that there cannot be two concurrent DMA operations
>>>> happening in different regions of the same buffer, nor can DMA and CPU
>>>> access be happening concurrently even if in different parts of the
>>>> buffer.  
>>> 
>>> Is this something which we can fix using a simple __align(USB_DMA_ALIGN) on elements in C-structures which are allowed to be DMA loaded.
>> 
>> No. I don't think so.  the reason is that you can't define USB_DMA_ALGIN to be a constant on MIPS, at least, or I think ARM because that's determined at run time.
> 
> But don't mbuf structures do pretty much what Hans is suggesting?

They kinda do, and kinda don't.

> Why is mbuf okay?

mbuf is OK because it is never changed while the DMA is pending to the buffers.  That is, m_hdr and m_ext are invariant while the device owns the memory. In addition, mbuf allocations are so large that no two mbufs share the same cache line (although it looks like 256 might be too small to avoid that on some MIPS processors). usb buffers do not obey these same rules, otherwise none of the stuff we're talking about would matter.

Warner