CPU Cache and busdma usage in USB

Thu Jul 16 19:56:32 UTC 2009

On Tuesday 14 July 2009 19:20:22 Rafal Jaworowski wrote:

> >> Please note these problems should be considered as a showstopper
> >> for the release since USB is currently broken on at least three ARM
> >> platforms in the tree (Marvell).

Hi,

First of all I'm not able to boot into userland on my ARM board. Maybe I 
missed something. I'm using a custom built userland image.

Trying to mount root from ufs:/dev/md0                                                                                              
warning: no time-of-day clock registered, system time will not be set 
accurately                                                    
Jul 16 18:53:22 init: login_getclass: unknown class 'daemon'                                                                        
Fatal kernel mode data abort: 'Alignment Fault 3'                                                                                   
trapframe: 0xc622cc60                                                                                                               
FSR=00000003, FAR=6e75203a, spsr=600000d3                                                                                           
r0 =60000093, r1 =c10594a0, r2 =c10594a0, r3 =00000002                                                                              
r4 =c10acbd0, r5 =00000000, r6 =6e75203a, r7 =c10594a0                                                                              
r8 =c622cd74, r9 =00000000, r10=c0ab081c, r11=c622cccc                                                                              
r12=c622ccac, ssp=c622ccac, slr=c020a144, pc =c00c02c8                                                                              

[thread pid 4 tid 100008 ]                                                                                                          
Stopped at      _thread_lock_flags+0x24:        ldr     r4, [r6]                                                                    
db>                            

With the following patch in arm/cpufunc.c my UMASS device gets detected at 
boot time (KB9202B ARM board). Else I get a USB request timeout followed by a 
panic. But data transfers of more than 16KByte are likely to be broken.

struct cpu_functions arm9_cpufuncs = {
...
- /*XXX*/	arm9_dcache_wbinv_range,	/* dcache_inv_range	*/
+ /*XXX*/	arm9_dcache_inv_range,	/* dcache_inv_range	*/

};

Looking at the assembly code behind "arm9_dcache_inv_range" in 
"cpufunc_asm_arm9.S" I see that "arm9_dcache_wbinv_all" gets called if the 
area to invalidate is greater equal to 16KByte. This is wrong. When asked to 
invalidate, then no memory must be flushed in that region. Because USB works 
like this:

invalidate (TD)
read field from (TD)

Next IRQ

invalidate (TD)
read field from (TD)

If the invalidate call flushes any data, you clearly see from the pseudo code 
above that there is a chance that the last read TD field in the cache gets 
written back before the next invalidate at the next IRQ, when the invalidate 
function is implemented like a writeback+invalidate!??

Piotr and Rafal: How does ARM work? Are there two caches? One read cache and 
one write cache. Or are they the same. What happens if you read a value from 
RAM into cache, then writeback+invalidate it. Does it get written back to RAM 
or does it get discarded? Sorry, I have some holes in my ARM knowledge which 
you need to fill in.

Reference:

ENTRY(arm9_dcache_inv_range)
	ldr	ip, .Larm9_line_size
	cmp	r1, #0x4000
	bcs	.Larm9_dcache_wbinv_all

Anything unclear?

Working version of kernel with my patch having stick plugged in at boot:

ugen1.2: <Kingston> at usbus1
umass0: <Kingston DataTraveler 2.0, class 0/0, rev 2.00/1.10, addr 2> on 
usbus1
umass0:  SCSI over Bulk-Only; quirks = 0x0000
umass0:0:0:-1: Attached to scbus0
(probe0:umass-sim0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0 
(probe0:umass-sim0:0:0:0): CAM Status: SCSI Status Error
(probe0:umass-sim0:0:0:0): SCSI Status: Check Condition
(probe0:umass-sim0:0:0:0): UNIT ATTENTION asc:28,0
(probe0:umass-sim0:0:0:0): Not ready to ready change, medium may have changed
(probe0:umass-sim0:0:0:0): Retrying Command (per Sense Data)
da0 at umass-sim0 bus 0 target 0 lun 0
da0: <Kingston DataTraveler 2.0 PMAP> Removable Direct Access SCSI-0 device 
da0: 1.000MB/s transfers
da0: 3842MB (7868416 512 byte sectors: 255H 63S/T 489C)

Stock 8-current code:

ugen1.2: <Kingston> at usbus1
umass0: <Kingston DataTraveler 2.0, class 0/0, rev 2.00/1.10, addr 2> on 
usbus1
umass0:  SCSI over Bulk-Only; quirks = 0x0000
umass0:0:0:-1: Attached to scbus0
(probe0:umass-sim0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0 
(probe0:umass-sim0:0:0:0): CAM Status: SCSI Status Error
(probe0:umass-sim0:0:0:0): SCSI Status: Check Condition
(probe0:umass-sim0:0:0:0): UNIT ATTENTION asc:28,0
(probe0:umass-sim0:0:0:0): Not ready to ready change, medium may have changed
(probe0:umass-sim0:0:0:0): Retrying Command (per Sense Data)
da0 at umass-sim0 bus 0 target 0 lun 0
da0: <Kingston DataTraveler 2.0 PMAP> Removable Direct Access SCSI-0 device 
da0: 1.000MB/s transfers
da0: 3842MB (7868416 512 byte sectors: 255H 63S/T 489C)
ugen1.2: <Kingston> at usbus1 (disconnected)
umass0: at uhub1, port 1, addr 2 (disconnected)
(da0:umass-sim0:0:0:0): lost device
(da0:umass-sim0:0:0:0): removing device entry
ugen1.2: <Kingston> at usbus1
umass0: <Kingston DataTraveler 2.0, class 0/0, rev 2.00/1.10, addr 2> on 
usbus1
umass0:  SCSI over Bulk-Only; quirks = 0x0000
umass0:0:0:-1: Attached to scbus0
Sleeping thread (tid 100031, pid 19) owns a non-sleepable lock
sched_switch() at sched_switch+0x10
scp=0xc00e7830 rlv=0xc00d2228 (mi_switch+0x108)
        rsp=0xc6271ddc rfp=0xc6271e04
        r7=0xc11306f0 r6=0x00000000
        r5=0x000000c0 r4=0x00000000

...
umass0: <Kingston DataTraveler 2.0, class 0/0, rev 2.00/1.10, addr 2> on 
usbus1
umass0:  SCSI over Bulk-Only; quirks = 0x0000
umass0:0:0:-1: Attached to scbus0
(probe0:umass-sim0:0:0:0): AutoSense Failed
usb_alloc_device:1586: set address 3 failed (USB_ERR_IOERROR, ignored)
usb_alloc_device:1624: getting device descriptor at addr 3 failed, 
USB_ERR_IOERROR!
usbd_req_re_enumerate:1539: addr=3, set address failed! (USB_ERR_IOERROR, 
ignored)
usbd_req_re_enumerate:1553: getting device descriptor at addr 3 failed, 
USB_ERR_IOERROR!
usbd_req_re_enumerate:1539: addr=3, set address failed! (USB_ERR_IOERROR, 
ignored)
usbd_req_re_enumerate:1553: getting device descriptor at addr 3 failed, 
USB_ERR_IOERROR!
ugen1.3: <(null)> at usbus1 (disconnected)
uhub_reattach_port:435: could not allocate new device!
(da0:umass-sim0:0:0:0): got CAM status 0x4
(da0:umass-sim0:0:0:0): fatal error, failed to attach to device
(da0:umass-sim0:0:0:0): lost device
(da0:umass-sim0:0:0:0): removing device entry
ugen1.2: <Kingston> at usbus1 (disconnected)
umass0: at uhub1, port 1, addr 2 (disconnected)
ugen1.2: <Kingston> at usbus1
umass0: <Kingston DataTraveler 2.0, class 0/0, rev 2.00/1.10, addr 2> on 
usbus1
umass0:  SCSI over Bulk-Only; quirks = 0x0000

vm_fault(0xc0ac65f4, 0, 1, 0) -> 1
Fatal kernel mode data abort: 'Translation Fault (S)'
trapframe: 0xc6271a38
FSR=00000005, FAR=00000014, spsr=a00000d3
r0 =c0accdcc, r1 =00000000, r2 =c11306f0, r3 =00000004
r4 =00000000, r5 =00000000, r6 =cc05f270, r7 =c1168d78
r8 =c6271b80, r9 =0000000d, r10=00000030, r11=c6271a98
r12=c6271a6c, ssp=c6271a84, slr=c0209ee8, pc =c00fd518

[thread pid 19 tid 100031 ]
Stopped at      turnstile_broadcast+0x30:       ldr     r2, [r1, #0x014]

Are there any pending patches which I should have put into the code before 
building?

--HPS