CPU Cache and busdma usage in USB

[Piotr Zięcik]

Monday 29 June 2009 15:16:56 Hans Petter Selasky napisał(a):
>
> You want to change the flags passed to bus_dmamap_sync() so that the
> flush/invalidate mapping gets right. I understand why your patch makes it
> work. That's not the problem.
>
> In "src/sys/arm/arm/busdma_machdep.c" there is a function called
> "_bus_dmamap_sync_bp()". If you look at that function you see that it only
> triggers on the "BUS_DMASYNC_PREWRITE" and "BUS_DMASYNC_POSTREAD" flags.
> After your patching only the PREXXXX flags are used, so if bouce pages are
> used on ARM and x86 and amd64 +++, then only BUS_DMASYNC_PREWRITE will do
> anything. This indicates that your patch is not fully correct.

That is true. I have missed "bounce page" case. I can change flags passed to 
bus_dmamap_sync() to fix this on ARM, but this will break MIPS.

This clearly shows the problem - using side effect of busdma to manage CPU 
cache. Current USB implementation relies of this side effect assuming that 
bus_dmamap_sync() with given flags will do cpu cache flush/invalidation.
This is not true even on i386 !

This thread is not about my patch. It is only a fast and dirty hack
making USB working on platforms without hardware cache coherency
and showing the problem. I see two proper solutions:

1. Implement usb_pc_cpu_invalidate() / usb_pc_cpu_flush() with
cpu_*() functions realizing requested operation.

2. Using busdma in proper way:
        [... prepare data to transfer ...]
        bus_dmamap_sync(..., PREREAD | PREWRITE);
        [... do transfer ...]
        bus_dmamap_sync(..., POSTREAD | POSTWRITE);
        [... check results ...]
as manpage says:
	<cite>
	If read and write operations are not preceded and followed by the
	appropriate synchronization operations, behavior is undefined.
	</cite>
Requirement cited above is currently not met in USB stack. Funtion
shown in http://fxr.watson.org/fxr/source/dev/usb/controller/ehci.c#L1294
is just an example of improper usage of bus_dmamap_sync(), which is hidden 
under usb_pc_cpu_invalidate().

This thread started from my question about general usage of usb_pc_cpu_*() 
functions. So I am asking once again - why these functions relies on side 
effect of busdma instead of simply doing cache flush/invalidation through 
cpu_*() functions ?

> Grepping through the source code for ARM, I found a line like this:
> (...)
> You see that it only performs purge and no prior flush. This is what needs
> fixing! If semihalf could go through the "arm/arm/cpufunc.c" file and fix
> those flush and invalidate functions then many people would become happy!

My developement platform is sheeva. CPU functions for this platform are 
implemented correctly.

> Again, it is not a problem in USB firstly, it is a problem in "arm/xxx".

You are suggesting that the problem is in ARM busdma (and in MIPS busdma, as 
on this platform USB also does not work). Could you give me example of 
platform _without_ hardware coherency where new USB stack simply works ?

-- 
Best regards.
Piotr Ziecik