USB2 reveals cache sync problems on AT91RM9200

[Stanislav Sedov]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 7 Jan 2009 18:19:25 +0100
Hans Petter Selasky <hselasky at c2i.net> mentioned:

> Hi,
> 
> I'm writing this e-mail to you because some of you have changed something in:
> 
> http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/arm/arm/busdma_machdep.c
> 
> I'm experiencing a problem where the CPU is not seeing the data written to 
> memory by the OHCI using DMA. This does not happen all the time.
> 
> Adding a "cpu_idcache_wbinv_all()" to my "cpu_invalidate" function solves the 
> problem 100% reliably. I think Stanislav should be able to reproduce this.
> 
> Any ideas?
> 
> I feel pretty sure that this is not an USB2 problem. I have tested that data 
> is correctly bounced on x86 and assume that if the data is correctly bounced, 
> it will also be correctly synched. Using bounce buffers by lowering 
> the "lowaddr" in the DMA tag, on my AT91RM9200, does not solve the problem.
> 
> --HPS
> 
> Hardware:
> 
> ## Starting application at 0x200000E0 ...
> KDB: debugger backends: ddb
> KDB: current backend: ddb
> Copyright (c) 1992-2009 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>         The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 8.0-CURRENT #33: Wed Jan  7 17:52:20 CET 2009
>     hans_other at server0.selasky.org:/usr/obj/arm/usr/8-current/src/sys/custom
> CPU: ARM920T rev 0 (ARM9TDMI core)
>   DC enabled IC enabled WB enabled LABT
>   16KB/32B 64-way Instruction cache
>   16KB/32B 64-way write-back-locking-A Data cache
> 
> My patch that makes things work again:
> 
> #include <machine/cpu.h>
> 
> and:
> 
> /*------------------------------------------------------------------------*
>  *      usb2_pc_cpu_invalidate - invalidate CPU cache
>  *------------------------------------------------------------------------*/
> void
> usb2_pc_cpu_invalidate(struct usb2_page_cache *pc)
> {
>         if (pc->page_offset_end == pc->page_offset_buf) {
>                 /* nothing has been loaded into this page cache! */
>                 return;
>         }
>  
>         cpu_idcache_wbinv_all(); /* and this line */
> 
>         bus_dmamap_sync(pc->tag, pc->map,
>             BUS_DMASYNC_POSTWRITE | BUS_DMASYNC_POSTREAD);
> }
> 

Well, further research showed that what is really not synchronizing
correctly is ohci ed descriptors. I've added the following code into
ohci2.c and clearly shows that ed descriptor contents differ before
and after cpu_idcache_wbinv_all despite it should have been syncronized
by usb2_pc_cpu_invalidate.

	usb2_pc_cpu_invalidate(ed->page_cache);
        ed_flags = le32toh(ed->ed_flags);
        ed_headp = le32toh(ed->ed_headp);
        ed_tailp = le32toh(ed->ed_tailp);
+       printf("OHCI: ed flags was: %d %d %d\n", le32toh(ed->ed_flags), le32toh(ed->ed_headp), le32toh(ed->ed_tailp));
+       cpu_idcache_wbinv_all();
+       printf("OHCI: ed flags now: %d %d %d\n", le32toh(ed->ed_flags), le32toh(ed->ed_headp), le32toh(ed->ed_tailp));

Moving cpu_idcache_wbinv_all just after usb2_pc_cpu_invalidate before ed descriptor
access fixes umass storage on arm for me.

- -- 
Stanislav Sedov
ST4096-RIPE
-----BEGIN PGP SIGNATURE-----

iEYEARECAAYFAkllI6gACgkQK/VZk+smlYG0ewCcD5VlJMoLas6hrx9sUC2y7/e5
cfQAnjBKaOwvmMBxt1nMecjB3pvemk7a
=86fe
-----END PGP SIGNATURE-----

!DSPAM:49652316967001025050683!