CPU Cache and busdma usage in USB

[Hans Petter Selasky]

Hi,

On Monday 29 June 2009 12:49:43 Piotr Zięcik wrote:
> Monday 29 June 2009 11:55:11 Hans Petter Selasky napisał(a):
> > Hi Piotr and Rafal,

>
> Look into ehci_check_transfer() function
> (http://fxr.watson.org/fxr/source/dev/usb/controller/ehci.c#L1294)
>
> usb_pc_cpu_invalidate() [bus_dmamap_sync()] is not used in this function
> correcly. It is not paired with usb_pc_cpu_flush() [opposite
> bus_dmamap_sync()] as busdma requires (see part of manpage cited above).
> The same problem is in part of code shown in previous mail.
>
> If usb_pc_cpu_invalidate()/usb_pc_cpu_flush() functions had been
> implemented without using busdma, for example as cpu_*() functions,
> ehci_check_transfer() would have been 100% correct. In current code busdma
> requirements are simply not met.

Good point. Unfortunately I cannot pair like you suggest, because then I can 
overwrite values updated by the hardware, if I first read the status for 
example, and the same 32-bits are updated by hardware, before the old value is 
flushed over the new one.. Also I cannot find the word "pair" in the busdma 
manpage? Where is this requirement stated?

I see some things:

You wrote earlier that the COHERENT flag was not set. That means your setup is 
using bounce pages for DMA? Right?

Then I see a problem, if I do several POST operations in a row then I can see 
that the real DMA memory can get cached:

In: "src/sys/arm/arm/busdma_machdep.c"

Is "vaddr_nocache" set or cleared?

                if (op & BUS_DMASYNC_POSTREAD) {
                        if (bpage->vaddr_nocache == 0) {
                                cpu_dcache_inv_range(bpage->vaddr,
                                    bpage->datacount);
                                cpu_l2cache_inv_range(bpage->vaddr,
                                    bpage->datacount);
                        }
                        bcopy((void *)(bpage->vaddr_nocache != 0 ? 
                            bpage->vaddr_nocache : bpage->vaddr),
                            (void *)bpage->datavaddr, bpage->datacount);
                        dmat->bounce_zone->total_bounced++;
                }

USB is currently _updating_ (!!) the PAGE offset part of "vaddr". If 
cpu_dcache_inv_range() is called with an address not starting at the cache 
line what will the cpu_dcache_inv_range() do? Will it skip to the next cache 
line? Or will it completely skip the whole cache sync operation?! In the 
function just above in the file I refer to, "bus_dmamap_sync_buf()", there is 
quite some more code to handle invalidation when the destination address is 
not properly aligned. I'm not an ARM expert. Maybe you can do an experiment 
for me:

Change: bpage->vaddr into (bpage->vaddr & ~arm_dcache_align_mask)
Change: bpage->datacount into (bpage->datacount + (bpage->vaddr & 
arm_dcache_align_mask) + arm_dcache_align_mask - 1) & ~arm_dcache_align_mask;

You don't need any memcpy there, because we own the complete memory page which 
vaddr is a part of!

>
> Ok. So if you use bus_dma(), please use it in correct way.

What has to be changed in busdma_machdep.c for ARM/MIPS so that the problem is 
resolved. I think there are something missing there and not in USB!

--HPS