USB2 reveals cache sync problems on AT91RM9200

[Grzegorz Bernacki]

Stanislav Sedov wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Wed, 7 Jan 2009 18:19:25 +0100
> Hans Petter Selasky <hselasky at c2i.net> mentioned:
> 
>> Hi,
>>
>> I'm writing this e-mail to you because some of you have changed something in:
>>
>> http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/arm/arm/busdma_machdep.c
>>
>> I'm experiencing a problem where the CPU is not seeing the data written to 
>> memory by the OHCI using DMA. This does not happen all the time.
>>
>> Adding a "cpu_idcache_wbinv_all()" to my "cpu_invalidate" function solves the 
>> problem 100% reliably. I think Stanislav should be able to reproduce this.
>>
>> Any ideas?
>>
>> I feel pretty sure that this is not an USB2 problem. I have tested that data 
>> is correctly bounced on x86 and assume that if the data is correctly bounced, 
>> it will also be correctly synched. Using bounce buffers by lowering 
>> the "lowaddr" in the DMA tag, on my AT91RM9200, does not solve the problem.
>>
>> --HPS
>>
>> Hardware:
>>
>> ## Starting application at 0x200000E0 ...
>> KDB: debugger backends: ddb
>> KDB: current backend: ddb
>> Copyright (c) 1992-2009 The FreeBSD Project.
>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>>         The Regents of the University of California. All rights reserved.
>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>> FreeBSD 8.0-CURRENT #33: Wed Jan  7 17:52:20 CET 2009
>>     hans_other at server0.selasky.org:/usr/obj/arm/usr/8-current/src/sys/custom
>> CPU: ARM920T rev 0 (ARM9TDMI core)
>>   DC enabled IC enabled WB enabled LABT
>>   16KB/32B 64-way Instruction cache
>>   16KB/32B 64-way write-back-locking-A Data cache
>>
>> My patch that makes things work again:
>>
>> #include <machine/cpu.h>
>>
>> and:
>>
>> /*------------------------------------------------------------------------*
>>  *      usb2_pc_cpu_invalidate - invalidate CPU cache
>>  *------------------------------------------------------------------------*/
>> void
>> usb2_pc_cpu_invalidate(struct usb2_page_cache *pc)
>> {
>>         if (pc->page_offset_end == pc->page_offset_buf) {
>>                 /* nothing has been loaded into this page cache! */
>>                 return;
>>         }
>>  
>>         cpu_idcache_wbinv_all(); /* and this line */
>>
>>         bus_dmamap_sync(pc->tag, pc->map,
>>             BUS_DMASYNC_POSTWRITE | BUS_DMASYNC_POSTREAD);
>> }
>>
> 
> Well, further research showed that what is really not synchronizing
> correctly is ohci ed descriptors. I've added the following code into
> ohci2.c and clearly shows that ed descriptor contents differ before
> and after cpu_idcache_wbinv_all despite it should have been syncronized
> by usb2_pc_cpu_invalidate.
> 
> 	usb2_pc_cpu_invalidate(ed->page_cache);
>         ed_flags = le32toh(ed->ed_flags);
>         ed_headp = le32toh(ed->ed_headp);
>         ed_tailp = le32toh(ed->ed_tailp);
> +       printf("OHCI: ed flags was: %d %d %d\n", le32toh(ed->ed_flags), le32toh(ed->ed_headp), le32toh(ed->ed_tailp));
> +       cpu_idcache_wbinv_all();
> +       printf("OHCI: ed flags now: %d %d %d\n", le32toh(ed->ed_flags), le32toh(ed->ed_headp), le32toh(ed->ed_tailp));
> 
> Moving cpu_idcache_wbinv_all just after usb2_pc_cpu_invalidate before ed descriptor
> access fixes umass storage on arm for me.
> 
Hi,

Some time ago we had a problem with EHCI. The problem was that after transfer CSW was overwritten with some old data from CSW of previous transfer. We saw it only when write-allocate feature was turned off. 
The problem was caused by specific handling of BUS_DMASYNC_POSTREAD in ARM bus dma code. If you look at the code you see that if synced region is not cache aligned we copy it into temporary location, invalidate the region and copy back saved data. But if write-allocate is turned on, writing data back causes allocating cache line and store the data in it. So we end up with dirty cache line contains stale data which should be invalidated. And those data could be flushed back at random time whenever cache line is recycled.
We fixed the problem using BUS_DMASYNC_PREREAD command before using updated CSW.
I don't know if you use write-allocate, but if you do and if the problem disappears when you turn it off it's probably the same problem.
You can also check the thread when Rafal described our problem:
http://docs.freebsd.org/cgi/getmsg.cgi?fetch=198997+0+archive/2008/freebsd-usb/20080203.freebsd-usb
correct place for fix mentioned there is:
http://people.freebsd.org/~raj/patches/misc/committed/usbdi-bus_dma-fix.diff

pozdrawiam,
Grzesiek