From hselasky at c2i.net  Sun Jul  5 10:36:48 2009
From: hselasky at c2i.net (Hans Petter Selasky)
Date: Sun Jul  5 10:36:56 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <200906301128.26046.hselasky@c2i.net>
References: <200906231035.43096.kosmo@semihalf.com>
	<200906301037.49367.kosmo@semihalf.com>
	<200906301128.26046.hselasky@c2i.net>
Message-ID: <200907051236.22783.hselasky@c2i.net>

On Tuesday 30 June 2009 11:28:23 Hans Petter Selasky wrote:
> On Tuesday 30 June 2009 10:37:48 Piotr Zi?cik wrote:
> > Monday 29 June 2009 15:16:56 Hans Petter Selasky napisa?(a):
> > > You want to change the flags passed to bus_dmamap_sync() so that the
> > > flush/invalidate mapping gets right. I understand why your patch makes
> > > it work. That's not the problem.
> > >
> > > In "src/sys/arm/arm/busdma_machdep.c" there is a function called
> > > "_bus_dmamap_sync_bp()". If you look at that function you see that it
> > > only triggers on the "BUS_DMASYNC_PREWRITE" and "BUS_DMASYNC_POSTREAD"
> > > flags. After your patching only the PREXXXX flags are used, so if bouce
> > > pages are used on ARM and x86 and amd64 +++, then only
> > > BUS_DMASYNC_PREWRITE will do anything. This indicates that your patch
> > > is not fully correct.
>
> Hi,
>
> > That is true. I have missed "bounce page" case. I can change flags passed
> > to bus_dmamap_sync() to fix this on ARM, but this will break MIPS.
>
> Right, so there is inconsistency among the platforms in how the BUSDMA is
> implemented, which should be fixed.
>
> > This clearly shows the problem - using side effect of busdma to manage
> > CPU cache. Current USB implementation relies of this side effect assuming
> > that bus_dmamap_sync() with given flags will do cpu cache
> > flush/invalidation. This is not true even on i386 !
>
> i386 handles this differently.
>
> > This thread is not about my patch. It is only a fast and dirty hack
> > making USB working on platforms without hardware cache coherency
> > and showing the problem. I see two proper solutions:
>
> Ok.
>
> > 1. Implement usb_pc_cpu_invalidate() / usb_pc_cpu_flush() with
> > cpu_*() functions realizing requested operation.
> >
> > 2. Using busdma in proper way:
> >         [... prepare data to transfer ...]
> >         bus_dmamap_sync(..., PREREAD | PREWRITE);
> >         [... do transfer ...]
> >         bus_dmamap_sync(..., POSTREAD | POSTWRITE);
> >         [... check results ...]
> > as manpage says:
> > 	<cite>
> > 	If read and write operations are not preceded and followed by the
> > 	appropriate synchronization operations, behavior is undefined.
> > 	</cite>
> > Requirement cited above is currently not met in USB stack. Funtion
> > shown in http://fxr.watson.org/fxr/source/dev/usb/controller/ehci.c#L1294
> > is just an example of improper usage of bus_dmamap_sync(), which is
> > hidden under usb_pc_cpu_invalidate().
>
> I'm not violating any rules in busdma. The busdma manual is silent about
> doing multiple sync operations of the same kind in a row. If the sheeva
> hardware you are using works differently, you have to fix it.
>
> > This thread started from my question about general usage of
> > usb_pc_cpu_*() functions. So I am asking once again - why these functions
> > relies on side effect of busdma instead of simply doing cache
> > flush/invalidation through cpu_*() functions ?
>
> Because busdma is the interface for doing these kinds of operations and not
> cpu_xxx(). The DMA memory is allocated using busdma and loaded using
> busdma, and cpu_xxx() is not a part of the busdma interface.
>
> > > Grepping through the source code for ARM, I found a line like this:
> > > (...)
> > > You see that it only performs purge and no prior flush. This is what
> > > needs fixing! If semihalf could go through the "arm/arm/cpufunc.c" file
> > > and fix those flush and invalidate functions then many people would
> > > become happy!
> >
> > My developement platform is sheeva. CPU functions for this platform are
> > implemented correctly.
> >
> > > Again, it is not a problem in USB firstly, it is a problem in
> > > "arm/xxx".
> >
> > You are suggesting that the problem is in ARM busdma (and in MIPS busdma,
> > as on this platform USB also does not work). Could you give me example of
> > platform _without_ hardware coherency where new USB stack simply works ?
>
> If I find time later today I will fix the cache sync operations for
> AT91RM9200 which has a built in OHCI, which has been working fine with my
> new USB stack.
>
> Can you please add some prints to the "bus_dmamap_sync_buf()" code on ARM,
> and verify which cases the CPU is entering, without your patch and with
> your patch. From my analysis your patch will just change the execution
> path:
>
> With your patch it calls:
>
>                 cpu_dcache_wb_range((vm_offset_t)buf, len);
>                 cpu_l2cache_wb_range((vm_offset_t)buf, len);
>
> Without your patch it calls:
>
>                         cpu_dcache_wbinv_range((vm_offset_t)buf, len);
>                         cpu_l2cache_wbinv_range((vm_offset_t)buf, len);
>
> In my view these execution paths are equivalent with regard to clean or
> cache flush. They should both perform a cache flush, which is what the USB
> code wants to do, but obviously the latter case is not implemented
> correctly, most likely because it doesn't flush the buffer like expected!
>
> Try adding:
>
>                 cpu_dcache_wb_range((vm_offset_t)buf, len);
>                 cpu_l2cache_wb_range((vm_offset_t)buf, len);
>
> Before:
>
>                         cpu_dcache_wbinv_range((vm_offset_t)buf, len);
>                         cpu_l2cache_wbinv_range((vm_offset_t)buf, len);
>
> In: sys/arm/arm/busdma_machdep.c
>

Hi,

I did not find time yet to test on my AT91RM9200 board. I hope to do a test 
this week. Did you at semihalf find anything?

--HPS

From mih at semihalf.com  Mon Jul  6 09:29:33 2009
From: mih at semihalf.com (Michal Hajduk)
Date: Mon Jul  6 09:29:40 2009
Subject: pmap problem in FreeBSD current 
Message-ID: <4A51BFF6.4090401@semihalf.com>

Hello Mark,

While testing EHCI (FreeBSD 8-current with new USB-stack) on
ARM machine I have found some problem probably related to
svn commit *194459* 
http://svn.freebsd.org/viewvc/base?view=revision&revision=194459
With compiled USB-stack I had vm_fault:

ehci0: <Marvell Integrated USB 2.0 controller> at mem 
0xf1050000-0xf1050fff irq 48,19 on mbus0
lock order reversal: (sleepable after non-sleepable)
1st 0xc0f55a88 256 (UMA zone) @ 
/home/mih/git/marvell-current/sys/vm/uma_core.c:2036
2nd 0xc0bf2fbc user map (user map) @ 
/home/mih/git/marvell-current/sys/vm/vm_map.c:3520
KDB: stack backtrace:
db_trace_thread() at db_trace_thread+0x10
scp=0xc0b28458 rlv=0xc0916b78 ($a+0x34)
rsp=0xc0e46764 rfp=0xc0e46880
r10=0xc36236c0 r9=0x00000dc0
r8=0xc0d3504c r7=0xc3623930 r6=0xffffffff r5=0xc0b52180
r4=0xc0e4676c
$a() at $a+0x10
scp=0xc0916b54 rlv=0xc09f6ae0 (kdb_backtrace+0x3c)
rsp=0xc0e46884 rfp=0xc0e46894
r4=0xc0c05d60
kdb_backtrace() at kdb_backtrace+0x10
scp=0xc09f6ab4 rlv=0xc0a05e80 (_witness_debugger+0x5c)
rsp=0xc0e46898 rfp=0xc0e468ac
r4=0x00000001
_witness_debugger() at _witness_debugger+0x14
scp=0xc0a05e38 rlv=0xc0a069a8 (witness_checkorder+0x5f4)
rsp=0xc0e468b0 rfp=0xc0e468f8
r5=0xc0bf2fbc r4=0x00000000
witness_checkorder() at witness_checkorder+0x10
scp=0xc0a063c4 rlv=0xc09d5cdc (_sx_slock+0x30)
rsp=0xc0e468fc rfp=0xc0e46920
r10=0xc0e46b40 r9=0xc0e46a40
r8=0x00000000 r7=0x00000000 r6=0x00000dc0 r5=0xc0b89084
r4=0xc0bf2fbc
_sx_slock() at _sx_slock+0x10
scp=0xc09d5cbc rlv=0xc0b1171c (vm_map_lookup+0x3c)
rsp=0xc0e46924 rfp=0xc0e46964
r8=0x00000001 r7=0x00000001
r6=0xc0bf2f74 r5=0x00000000 r4=0xc0e46a68
vm_map_lookup() at vm_map_lookup+0x10
scp=0xc0b116f0 rlv=0xc0b0b61c (vm_fault+0x90)
rsp=0xc0e46968 rfp=0xc0e46a98
r10=0xc0e46b40 r9=0xc0e46ef8
r8=0x00000001 r7=0xc0bf2d40 r6=0x00000000 r5=0x00000000
r4=0xc0e46a68
vm_fault() at vm_fault+0x10
scp=0xc0b0b59c rlv=0xc0b37cf0 (data_abort_handler+0x1ec)
rsp=0xc0e46a9c rfp=0xc0e46b3c
r10=0xc0e46b40 r9=0xc0e46ef8
r8=0x00000001 r7=0xc0bf2d40 r6=0x00000000 r5=0x00000000
r4=0xc0bf2b00
data_abort_handler() at data_abort_handler+0x10
scp=0xc0b37b14 rlv=0xc0b29ee0 (exception_exit)
rsp=0xc0e46b40 rfp=0xc0e46bac
r10=0xc0f4a1a4 r9=0x0000001f
r8=0xc0f54c80 r7=0x00000000 r6=0xc0f54c80 r5=0xffff1004
r4=0xc0f55a80
$a() at $a+0x10
scp=0xc0b08b6c rlv=0xc0b08ef4 ($a+0x3c)
rsp=0xc0e46bb0 rfp=0xc0e46bc8
r7=0x0000000f r6=0xc0f54c80
r5=0x00000003 r4=0xc0f55a80
$a() at $a+0x10
scp=0xc0b08ec8 rlv=0xc0b0a13c (uma_zalloc_arg+0x32c)
rsp=0xc0e46bcc rfp=0xc0e46c10
r6=0xc3b5efa8 r5=0x0000000f
r4=0x0000000f
uma_zalloc_arg() at uma_zalloc_arg+0x10
scp=0xc0b09e20 rlv=0xc09bfefc (malloc+0x5c)
rsp=0xc0e46c14 rfp=0xc0e46c34
r10=0xc0e46c80 r9=0xc0bebeac
r8=0xc0bda080 r7=0x00000002 r6=0x00000100 r5=0xc0f54c80
r4=0x00000004
malloc() at malloc+0x14
scp=0xc09bfeb4 rlv=0xc0b26674 (bus_dmamem_alloc+0x258)
rsp=0xc0e46c38 rfp=0xc0e46c70
r8=0x00000004 r7=0x00000002
r6=0xc3621800 r5=0xc0d6a234 r4=0x00000080
bus_dmamem_alloc() at bus_dmamem_alloc+0x10
scp=0xc0b2642c rlv=0xc0966fc4 (usb_pc_alloc_mem+0xa4)
rsp=0xc0e46c74 rfp=0xc0e46cb0
r10=0xc3b4de28 r9=0xc0b64844
r8=0xc3b4d808 r7=0xc3b4ddfc r6=0x00000100 r5=0xc3b4bf20
r4=0x00000080
usb_pc_alloc_mem() at usb_pc_alloc_mem+0x10
scp=0xc0966f30 rlv=0xc0962748 ($a+0x2c)
rsp=0xc0e46cb4 rfp=0xc0e46cc4
r10=0x00000000 r9=0xc0b64844
r8=0xc095ac30 r7=0xc096271c r6=0xc3b4dc50 r5=0xc3b4a000
r4=0xc3b4dc50
$a() at $a+0x10
scp=0xc096272c rlv=0xc095ad00 (ehci_iterate_hw_softc+0xd0)
rsp=0xc0e46cc8 rfp=0xc0e46ce8
r4=0x00000078
ehci_iterate_hw_softc() at ehci_iterate_hw_softc+0x10
scp=0xc095ac40 rlv=0xc09626bc (usb_bus_mem_alloc_all+0xb8)
rsp=0xc0e46cec rfp=0xc0e46d18
r7=0xc3b4dc50 r6=0x00000000
r5=0xc3b4ddac r4=0x00000008
usb_bus_mem_alloc_all() at usb_bus_mem_alloc_all+0x10
scp=0xc0962614 rlv=0xc0b49494 ($a+0x60)
rsp=0xc0e46d1c rfp=0xc0e46d5c
r8=0xc3b4dc50 r7=0xc09f08fc
r6=0xc3620000 r5=0xc3b4a000 r4=0xc3620480
$a() at $a+0x10
scp=0xc0b49444 rlv=0xc09f1fc8 (device_attach+0x70)
rsp=0xc0e46d60 rfp=0xc0e46da0
r10=0x00000000 r9=0xc0b64844
r8=0xc3620000 r7=0xc09f08fc r6=0xc362004c r5=0xc36bd344
r4=0xc3620480
device_attach() at device_attach+0x10
scp=0xc09f1f68 rlv=0xc09f3060 (bus_generic_attach+0x20)
rsp=0xc0e46da4 rfp=0xc0e46db4
r10=0x00000000 r9=0xc0b64844
r8=0xc3620480 r7=0xc0bed508 r6=0xc0bed510 r5=0x00000001
r4=0xc3620000
bus_generic_attach() at bus_generic_attach+0x10
scp=0xc09f3050 rlv=0xc0b40864 ($a+0x1fc)
rsp=0xc0e46db8 rfp=0xc0e46de0
r4=0x00000020
$a() at $a+0x10
scp=0xc0b40678 rlv=0xc09f1fc8 (device_attach+0x70)
rsp=0xc0e46de4 rfp=0xc0e46e24
r8=0xc3620480 r7=0xc09f08fc
r6=0xc36204cc r5=0xc361b204 r4=0xc3620600
device_attach() at device_attach+0x10
scp=0xc09f1f68 rlv=0xc09f3060 (bus_generic_attach+0x20)
rsp=0xc0e46e28 rfp=0xc0e46e38
r10=0x00000000 r9=0xc0b64844
r8=0xc3620600 r7=0xc09f08fc r6=0xc362064c r5=0xc361b465
r4=0xc3620480
bus_generic_attach() at bus_generic_attach+0x10
scp=0xc09f3050 rlv=0xc0b2cdcc ($a+0x1c)
rsp=0xc0e46e3c rfp=0xc0e46e4c
r4=0xc3620600
$a() at $a+0x10
scp=0xc0b2cdc0 rlv=0xc09f1fc8 (device_attach+0x70)
rsp=0xc0e46e50 rfp=0xc0e46e90
r4=0x80000000
device_attach() at device_attach+0x10
scp=0xc09f1f68 rlv=0xc09f32a0 (bus_generic_new_pass+0x110)
rsp=0xc0e46e94 rfp=0xc0e46eb4
r10=0x00000000 r9=0x0100029c
r8=0xc0c05b68 r7=0xc3620a00 r6=0xc0bec458 r5=0xc0bd38d0
r4=0xc3620600
bus_generic_new_pass() at bus_generic_new_pass+0x10
scp=0xc09f31a0 rlv=0xc09efb28 (bus_set_pass+0x98)
rsp=0xc0e46eb8 rfp=0xc0e46ed4
r8=0x0103cc50 r7=0xc0c05b68
r6=0x7fffffff r5=0xc3620a00 r4=0xc361c820
bus_set_pass() at bus_set_pass+0x10
scp=0xc09efaa0 rlv=0xc0997a1c (mi_startup+0x8c)
rsp=0xc0e46ed8 rfp=0xc0e46ef4
r7=0xc0bf345c r6=0xc0bf3460
r5=0xc0bf3464 r4=0xc0b8f2f4
mi_startup() at mi_startup+0x10
scp=0xc09979a0 rlv=0xc09001e4 (virt_done+0x14)
rsp=0xc0e46ef8 rfp=0x00000000
r7=0x00900140 r6=0x0103e020
r5=0x0090014c r4=0x00900224

vm_fault(0xc0bf2f74, 0, 1, 0) -> 1
Fatal kernel mode data abort: 'Translation Fault (S)'
trapframe: 0xc0e46b40
FSR=00000005, FAR=00000004, spsr=600000d3
r0 =c0f55a80, r1 =c0f54c80, r2 =00000003, r3 =fffffffb
r4 =c0f55a80, r5 =00000003, r6 =c0f54c80, r7 =00000000
r8 =c0f54c80, r9 =0000001f, r10=c0f4a1a4, r11=c0e46bac
r12=c0e46bb0, ssp=c0e46b8c, slr=c0b08ef4, pc =c0b08c74

[thread pid 0 tid 100000 ]
Stopped at $a+0x118: ldr r2, [r7, #0x004]
db>

Without new-usb stack with booting from NFS I have a LOR on
the line which was added in svn commit *194459* 
<http://svn.freebsd.org/viewvc/base?view=revision&revision=194459> :
mge0: link state changed to UP
NFS ROOT: 10.0.0.201:/nfsroot/rd6281/
lock order reversal:
1st 0xc36050b0 pmap (pmap) @ 
/home/mih/git/marvell-current/sys/arm/arm/pmap.c:971
2nd 0xc0d26038 vm object (uma object) @ 
/home/mih/git/marvell-current/sys/vm/uma_core.c:1011
KDB: stack backtrace:
db_trace_thread() at db_trace_thread+0x10
scp=0xc0af6ce8 rlv=0xc0907798 ($a+0x34)
rsp=0xcf404880 rfp=0xcf40499c
r10=0xc35c84b8 r9=0x000003f3
r8=0xc0cea35c r7=0xc35c8a00 r6=0xffffffff r5=0xc0b1e610
r4=0xcf404888
$a() at $a+0x10
scp=0xc0907774 rlv=0xc09c5370 (kdb_backtrace+0x3c)
rsp=0xcf4049a0 rfp=0xcf4049b0
r4=0xc0bbb1d0
kdb_backtrace() at kdb_backtrace+0x10
scp=0xc09c5344 rlv=0xc09d4710 (_witness_debugger+0x5c)
rsp=0xcf4049b4 rfp=0xcf4049c8
r4=0x00000001
_witness_debugger() at _witness_debugger+0x14
scp=0xc09d46c8 rlv=0xc09d5238 (witness_checkorder+0x5f4)
rsp=0xcf4049cc rfp=0xcf404a14
r5=0xc0d26038 r4=0x00000000
witness_checkorder() at witness_checkorder+0x10
scp=0xc09d4c54 rlv=0xc09912b4 (_mtx_lock_flags+0x34)
rsp=0xcf404a18 rfp=0xcf404a40
r10=0xc0d26b80 r9=0xc0ef86e0
r8=0x000003f3 r7=0xc0b46f74 r6=0x00000000 r5=0x00000000
r4=0xc0d26038
_mtx_lock_flags() at _mtx_lock_flags+0x10
scp=0xc0991290 rlv=0xc0ad56b0 ($a+0x3c)
rsp=0xcf404a44 rfp=0xcf404a80
r10=0x00001000 r8=0x00000101
r7=0x00000000 r6=0xc0ef86e0 r5=0xc0ef9600 r4=0xc0ef9600
$a() at $a+0x10
scp=0xc0ad5684 rlv=0xc0ad6f44 ($a+0x70)
rsp=0xcf404a84 rfp=0xcf404ac0
r10=0xc0ad5674 r9=0xc0ef86e0
r8=0x00000101 r7=0x00000000 r6=0xc0ef86e0 r5=0x00000001
r4=0xc0ef9600
$a() at $a+0x10
scp=0xc0ad6ee4 rlv=0xc0ad74c0 ($a+0xd4)
rsp=0xcf404ac4 rfp=0xcf404ae4
r10=0xc0ee4c48 r9=0x00000080
r8=0xc0ef86e0 r7=0x00000000 r6=0xc0ef86e0 r5=0x00000201
r4=0xc0ef9600
$a() at $a+0x10
scp=0xc0ad73fc rlv=0xc0ad7784 ($a+0x3c)
rsp=0xcf404ae8 rfp=0xcf404b00
r7=0x0000006f r6=0xc0ef86e0
r5=0x00000001 r4=0xc0ef9600
$a() at $a+0x10
scp=0xc0ad7758 rlv=0xc0ad89cc (uma_zalloc_arg+0x32c)
rsp=0xcf404b04 rfp=0xcf404b48
r6=0xc1888de8 r5=0x0000006f
r4=0x0000006f
uma_zalloc_arg() at uma_zalloc_arg+0x10
scp=0xc0ad86b0 rlv=0xc0b00ca4 ($a+0x870)
rsp=0xcf404b4c rfp=0xcf404bd0
r10=0x01f9a55e r9=0x0000003c
r8=0xc36050b0 r7=0x00000000 r6=0x00000000 r5=0x01f9a552
r4=0xc3cbe0b8
$a() at $a+0x10
scp=0xc0b00444 rlv=0xc0b0169c (pmap_enter+0x70)
rsp=0xcf404bd4 rfp=0xcf404c04
r10=0x00000007 r9=0xc0d1b948
r8=0xc0f951b8 r7=0x7ffff000 r6=0x00000000 r5=0xc36050b0
r4=0xc0b4a4bc
pmap_enter() at pmap_enter+0x10
scp=0xc0b0163c rlv=0xc0adaff0 ($a+0x17c)
rsp=0xcf404c08 rfp=0xcf404d38
r10=0xcf404de0 r9=0xcf404ef8
r8=0x00000002 r7=0x00000000 r6=0x00000000 r5=0x00000001
r4=0x00000000
vm_fault() at vm_fault+0x10
scp=0xc0ad9e2c rlv=0xc0b065f0 (data_abort_handler+0x1ec)
rsp=0xcf404d3c rfp=0xcf404ddc
r10=0xcf404de0 r9=0xcf404ef8
r8=0x00000002 r7=0xc3603000 r6=0x00000000 r5=0x7ffff000
r4=0xc3601088
data_abort_handler() at data_abort_handler+0x10
scp=0xc0b06414 rlv=0xc0af8770 (exception_exit)
rsp=0xcf404de0 rfp=0xcf404e80
r10=0xc0965f04 r9=0xc3601000
r8=0xc0b892c8 r7=0xc0b892d2 r6=0xcf404eac r5=0xffff1004
r4=0x7fffffff
$a() at $a+0x10
scp=0xc0965f14 rlv=0xc097c78c (fork_exit+0x64)
rsp=0xcf404e84 rfp=0xcf404ea8
r10=0xc0965f04 r9=0x00000000
r8=0x00000000 r7=0xc3601000 r6=0xcf404eac r5=0xc0d26b80
r4=0xc3603000
fork_exit() at fork_exit+0x10
scp=0xc097c738 rlv=0xc0b05c5c (fork_trampoline+0x14)
rsp=0xcf404eac rfp=0x00000000
r10=0x00000000 r8=0x00000000
r7=0xc0b29cb8 r6=0xcf45feac r5=0x00000000 r4=0xc0965f04
lock order reversal:
1st 0xc36050b0 pmap (pmap) @ 
/home/mih/git/marvell-current/sys/arm/arm/pmap.c:971
2nd 0xc0d1b948 vm page queue mutex (vm page queue mutex) @ 
/home/mih/git/marvell-current/sys/arm/arm/pmap.c:1696
KDB: stack backtrace:
db_trace_thread() at db_trace_thread+0x10
scp=0xc0af6ce8 rlv=0xc0907798 ($a+0x34)
rsp=0xcf4047e8 rfp=0xcf404904
r10=0xc35c84b8 r9=0x000006a0
r8=0xc0ce9ee4 r7=0xc35c8178 r6=0xffffffff r5=0xc0b1e610
r4=0xcf4047f0
$a() at $a+0x10
scp=0xc0907774 rlv=0xc09c5370 (kdb_backtrace+0x3c)
rsp=0xcf404908 rfp=0xcf404918
r4=0xc0bbb1d0
kdb_backtrace() at kdb_backtrace+0x10
scp=0xc09c5344 rlv=0xc09d4710 (_witness_debugger+0x5c)
rsp=0xcf40491c rfp=0xcf404930
r4=0x00000001
_witness_debugger() at _witness_debugger+0x14
scp=0xc09d46c8 rlv=0xc09d5238 (witness_checkorder+0x5f4)
rsp=0xcf404934 rfp=0xcf40497c
r5=0xc0d1b948 r4=0x00000000
witness_checkorder() at witness_checkorder+0x10
scp=0xc09d4c54 rlv=0xc09912b4 (_mtx_lock_flags+0x34)
rsp=0xcf404980 rfp=0xcf4049a8
r10=0xc0d26b80 r9=0x00000012
r8=0x000006a0 r7=0xc0b4a4bc r6=0x00000000 r5=0x00000000
r4=0xc0d1b948
_mtx_lock_flags() at _mtx_lock_flags+0x10
scp=0xc0991290 rlv=0xc0afc600 ($a+0x110)
rsp=0xcf4049ac rfp=0xcf4049d8
r10=0x00000000 r8=0xc0f95494
r7=0xc36054dc r6=0xc36050b0 r5=0xc0f95464 r4=0xc1889150
$a() at $a+0x10
scp=0xc0afc500 rlv=0xc0b00f90 ($a+0xb5c)
rsp=0xcf4049dc rfp=0xcf404a60
r10=0x01fa300e r9=0x00000000
r8=0xc36050b0 r7=0x00000000 r6=0x00000000 r5=0x01fa3002
r4=0xc3cbde90
$a() at $a+0x10
scp=0xc0b00444 rlv=0xc0b015c8 (pmap_enter_object+0x88)
rsp=0xcf404a64 rfp=0xcf404a98
r10=0x00000005 r9=0xc36050b0
r8=0xc0f95464 r7=0x00000000 r6=0x00000097 r5=0xc0f95464
r4=0x00000001
pmap_enter_object() at pmap_enter_object+0x10
scp=0xc0b01550 rlv=0xc0ade254 (vm_map_pmap_enter+0x294)
rsp=0xcf404a9c rfp=0xcf404af0
r10=0x00000000 r9=0x00000000
r8=0x00000000 r7=0x00000097 r6=0x00000000 r5=0x00000000
r4=0x0000000f
vm_map_pmap_enter() at vm_map_pmap_enter+0x14
scp=0xc0addfd4 rlv=0xc0adef74 (vm_map_insert+0x390)
rsp=0xcf404af4 rfp=0xcf404b44
r10=0x00008000 r9=0x00000000
r8=0x00000000 r7=0xc3cc1110 r6=0x00000000 r5=0x00000000
r4=0x00000000
vm_map_insert() at vm_map_insert+0x10
scp=0xc0adebf4 rlv=0xc0963f9c ($a+0x258)
rsp=0xcf404b48 rfp=0xcf404b8c
r10=0x0009f000 r9=0x00008000
r8=0xc3605000 r7=0x00000000 r6=0x00000000 r5=0x00000005
r4=0x00000000
$a() at $a+0x10
scp=0xc0963d54 rlv=0xc09640b0 ($a+0xe0)
rsp=0xcf404b90 rfp=0xcf404bd0
r10=0x00096208 r9=0x00008000
r8=0xc3cc1110 r7=0x00000000 r6=0x00096208 r5=0xfffff000
r4=0x0009f000
$a() at $a+0x10
scp=0xc0963fe0 rlv=0xc0965300 ($a+0x448)
rsp=0xcf404bd4 rfp=0xcf404c64
r10=0x00000000 r9=0xc0ba7eb4
r8=0xc0ba19a4 r7=0xcf404d58 r6=0xcf408000 r5=0x00000000
r4=0xcf408034
$a() at $a+0x10
scp=0xc0964ec8 rlv=0xc09799d0 (kern_execve+0x258)
rsp=0xcf404c68 rfp=0xcf404de4
r10=0x00000000 r9=0xc3603000
r8=0xcf404d58 r7=0xffffffff r6=0xc0ba8bb4 r5=0x00000000
r4=0x00000002
kern_execve() at kern_execve+0x10
scp=0xc0979788 rlv=0xc097a58c (execve+0x48)
rsp=0xcf404de8 rfp=0xcf404e28
r10=0x7ffffffd r9=0xc3601000
r8=0xc0b892c8 r7=0xc0b892d2 r6=0x7ffffff2 r5=0xcf404df0
r4=0xc3603000
execve() at execve+0x10
scp=0xc097a554 rlv=0xc09660f0 ($a+0x1ec)
rsp=0xcf404e2c rfp=0xcf404e80
r5=0x7fffffe4 r4=0x7fffffe8
$a() at $a+0x10
scp=0xc0965f14 rlv=0xc097c78c (fork_exit+0x64)
rsp=0xcf404e84 rfp=0xcf404ea8
r10=0xc0965f04 r9=0x00000000
r8=0x00000000 r7=0xc3601000 r6=0xcf404eac r5=0xc0d26b80
r4=0xc3603000
fork_exit() at fork_exit+0x10
scp=0xc097c738 rlv=0xc0b05c5c (fork_trampoline+0x14)
rsp=0xcf404eac rfp=0x00000000
r10=0x00000000 r8=0x00000000
r7=0xc0b29cb8 r6=0xcf45feac r5=0x00000000 r4=0xc0965f04

When I reverted changes in pmap from this svn commit I didn't have LORs 
nor panics.
I will be grateful for any help or advice.

Best regards,
Micha? Hajduk
From stas at FreeBSD.org  Mon Jul  6 09:46:22 2009
From: stas at FreeBSD.org (Stanislav Sedov)
Date: Mon Jul  6 09:46:28 2009
Subject: pmap problem in FreeBSD current
In-Reply-To: <4A51BFF6.4090401@semihalf.com>
References: <4A51BFF6.4090401@semihalf.com>
Message-ID: <20090706134612.5cf860e4.stas@FreeBSD.org>

On Mon, 06 Jul 2009 11:12:22 +0200
Michal Hajduk <mih@semihalf.com> mentioned:

> Hello Mark,
> 
> While testing EHCI (FreeBSD 8-current with new USB-stack) on
> ARM machine I have found some problem probably related to
> svn commit *194459* 
> http://svn.freebsd.org/viewvc/base?view=revision&revision=194459
> With compiled USB-stack I had vm_fault:
> 

Hi, Michal!

This is the known problem and we currently working on fixing it.  It
looks like that r194459, while certainly good, uncovered a memory
corruption problem in ARM code.  As a workaround you can revert
r194459 for now.

-- 
Stanislav Sedov
ST4096-RIPE
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 801 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-arm/attachments/20090706/a9a14334/attachment.pgp
From mih at semihalf.com  Mon Jul  6 10:35:46 2009
From: mih at semihalf.com (Michal Hajduk)
Date: Mon Jul  6 10:35:53 2009
Subject: pmap problem in FreeBSD current
In-Reply-To: <20090706134612.5cf860e4.stas@FreeBSD.org>
References: <4A51BFF6.4090401@semihalf.com>
	<20090706134612.5cf860e4.stas@FreeBSD.org>
Message-ID: <4A51D398.9070104@semihalf.com>

Stanislav Sedov wrote:
> On Mon, 06 Jul 2009 11:12:22 +0200
> Michal Hajduk <mih@semihalf.com> mentioned:
>
>   
>> Hello Mark,
>>
>> While testing EHCI (FreeBSD 8-current with new USB-stack) on
>> ARM machine I have found some problem probably related to
>> svn commit *194459* 
>> http://svn.freebsd.org/viewvc/base?view=revision&revision=194459
>> With compiled USB-stack I had vm_fault:
>>
>>     
>
> Hi, Michal!
>
> This is the known problem and we currently working on fixing it.  It
> looks like that r194459, while certainly good, uncovered a memory
> corruption problem in ARM code.  As a workaround you can revert
> r194459 for now.
>
>   
Hi, Stanislav!

Thank you for answering.
Could you tell something more about this memory
corruption problem in ARM code? Have you found
which part of FreeBSD code is responsible for this,
and what do you think about LORs related to
r194459?

Best regards,
Micha? Hajduk
From bugmaster at FreeBSD.org  Mon Jul  6 11:06:55 2009
From: bugmaster at FreeBSD.org (FreeBSD bugmaster)
Date: Mon Jul  6 11:07:36 2009
Subject: Current problem reports assigned to freebsd-arm@FreeBSD.org
Message-ID: <200907061106.n66B6rYE010691@freefall.freebsd.org>

Note: to view an individual PR, use:
  http://www.freebsd.org/cgi/query-pr.cgi?pr=(number).

The following is a listing of current problems submitted by FreeBSD users.
These represent problem reports covering all versions including
experimental development code and obsolete releases.


S Tracker      Resp.      Description
--------------------------------------------------------------------------------
o arm/134368   arm        [patch] nslu2_led driver for the LEDs on the NSLU2
o arm/134338   arm        [patch] Lock GPIO accesses on ixp425
o arm/134092   arm        [patch] NSLU.hints contains wrong hints for on board n

3 problems total.

From tinguely at casselton.net  Mon Jul  6 17:14:56 2009
From: tinguely at casselton.net (Mark Tinguely)
Date: Mon Jul  6 17:15:02 2009
Subject: pmap problem in FreeBSD current
In-Reply-To: <20090706134612.5cf860e4.stas@FreeBSD.org>
Message-ID: <200907061714.n66HEqsV063372@casselton.net>

>  Michal Hajduk <mih@semihalf.com> mentioned:
>
>  > Hello Mark,
>  >=20
>  > While testing EHCI (FreeBSD 8-current with new USB-stack) on
>  > ARM machine I have found some problem probably related to
>  > svn commit *194459*=20
>  > http://svn.freebsd.org/viewvc/base?view=3Drevision&revision=3D194459
>  > With compiled USB-stack I had vm_fault:
>  >=20
>
>  Hi, Michal!
>
>  This is the known problem and we currently working on fixing it.  It
>  looks like that r194459, while certainly good, uncovered a memory
>  corruption problem in ARM code.  As a workaround you can revert
>  r194459 for now.
>
>  --=20
>  Stanislav Sedov
>  ST4096-RIPE

Sorry about that guys. I will walk through the pmap logic again to see
if there is a problem with the patch.

--Mark Tinguely.
From tinguely at casselton.net  Mon Jul  6 18:36:29 2009
From: tinguely at casselton.net (Mark Tinguely)
Date: Mon Jul  6 18:36:52 2009
Subject: pmap problem in FreeBSD current
In-Reply-To: <20090706134612.5cf860e4.stas@FreeBSD.org>
Message-ID: <200907061836.n66IaRSc068048@casselton.net>

One quick observation:
in pmap_nuke_pv(), lower if statement cleans out the pv_list when there
is no other mapping besides the kernel mapping. If I remember correctly,
this was put in at the last minute. we had to remove the kernel mapping
because the page went back to the one of the queues and the kernel mapping
caused a panic.

When the kernel mapping is removed from the other-wise empty pv_list,
the PG_WRITABLE is turned off like is done when the pv_entry list is
normally emptied.

But at this point, we know that there still is a kernel mapping and the
pmap_enter() should have enabled the PG_WRITABLE flag because kernel mappings
should be writable. In my opinion, that flag should be enabled.

Does it make sense that we should leave the page modified bit
or will it panic on the freeing of the page?

	if (pv != NULL && (pv->pv_flags & PVF_UNMAN) &&
	    TAILQ_NEXT(pv, pv_list) == NULL) {
		pg->md.pv_kva = pv->pv_va;
			/* a recursive pmap_nuke_pv */
		TAILQ_REMOVE(&pg->md.pv_list, pv, pv_list);
		TAILQ_REMOVE(&pm->pm_pvlist, pv, pv_plist);
		if (pv->pv_flags & PVF_WIRED)
			--pm->pm_stats.wired_count;
		pg->md.pvh_attrs &= ~PVF_REF;
-		pg->md.pvh_attrs &= ~PVF_MOD;
-		vm_page_flag_clear(pg, PG_WRITEABLE);
		pmap_free_pv_entry(pv);
	}


From kosmo at semihalf.com  Tue Jul  7 13:50:31 2009
From: kosmo at semihalf.com (Piotr =?iso-8859-2?q?Zi=EAcik?=)
Date: Tue Jul  7 13:50:38 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <200907051236.22783.hselasky@c2i.net>
References: <200906231035.43096.kosmo@semihalf.com>
	<200906301128.26046.hselasky@c2i.net>
	<200907051236.22783.hselasky@c2i.net>
Message-ID: <200907071550.28781.kosmo@semihalf.com>

>
> I did not find time yet to test on my AT91RM9200 board. I hope to do a test
> this week. Did you at semihalf find anything?
>

I had Checked USB behaviour on PowerPC without hardware cache coherency.
The problem also exists here and patch helps.

I also generated some logs which you asked for:

---- Original USB code -----

ugen0.1: <Marvell> at usbus0
uhub0: <Marvell EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <Marvell> at usbus1
uhub1: <Marvell EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ugen2.1: <Marvell> at usbus2
uhub2: <Marvell EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
Root mount waiting for: usbus2 usbus1 usbus0
uhub0: 1 port with 1 removable, self powered
uhub1: 1 port with 1 removable, self powered
uhub2: 1 port with 1 removable, self powered
=> bus_dmamap_sync(0xC3A9E780, 0xC3B6F540, [PREREAD PREWRITE ________ 
_________])
        -> Writeback Invalidate 0xC3B9E168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E500, 0xC3AB8C40, [PREREAD PREWRITE ________ 
_________])
        -> Writeback Invalidate 0xC3B9D168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E500, 0xC3AB8C00, [_______ ________ POSTREAD 
POSTWRITE])
        -> Invalidate 0xC3B9D170 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E500, 0xC3AB8C00, [_______ ________ POSTREAD 
POSTWRITE])
        -> Invalidate 0xC3B9D170 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [PREREAD PREWRITE ________ 
_________])
        -> Writeback Invalidate 0xC3B9C168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [_______ ________ POSTREAD 
POSTWRITE])
        -> Invalidate 0xC3B9C170 (size: 0x00000012)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [_______ ________ POSTREAD 
POSTWRITE])
        -> Invalidate 0xC3B9C170 (size: 0x00000012)
=> bus_dmamap_sync(0xC3A9E080, 0xC3B6F400, [PREREAD PREWRITE ________ 
_________])
        -> Writeback Invalidate 0xC3B9B168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E080, 0xC3B6F280, [_______ ________ POSTREAD 
POSTWRITE])
        -> Invalidate 0xC3B9B170 (size: 0x00000002)
=> bus_dmamap_sync(0xC3A9E080, 0xC3B6F280, [_______ ________ POSTREAD 
POSTWRITE])
        -> Invalidate 0xC3B9B170 (size: 0x00000002)
=> bus_dmamap_sync(0xC3A9E080, 0xC3B6F400, [PREREAD PREWRITE ________ 
_________])
        -> Writeback Invalidate 0xC3B9B168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E080, 0xC3B6F280, [_______ ________ POSTREAD 
POSTWRITE])
        -> Invalidate 0xC3B9B170 (size: 0x00000009)
=> bus_dmamap_sync(0xC3A9E080, 0xC3B6F280, [_______ ________ POSTREAD 
POSTWRITE])
        -> Invalidate 0xC3B9B170 (size: 0x00000009)
Root mount waiting for: usbus0
=> bus_dmamap_sync(0xC3A9E300, 0xC3AB8D00, [PREREAD PREWRITE ________ 
_________])
        -> Writeback Invalidate 0xC3B9A168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E580, 0xC3AB8E40, [PREREAD PREWRITE ________ 
_________])
        -> Writeback Invalidate 0xC3B99168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E580, 0xC3B6F380, [_______ ________ POSTREAD 
POSTWRITE])
        -> Invalidate 0xC3B99170 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E580, 0xC3B6F380, [_______ ________ POSTREAD 
POSTWRITE])
        -> Invalidate 0xC3B99170 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E800, 0xC3B6F480, [PREREAD PREWRITE ________ 
_________])
        -> Writeback Invalidate 0xC3B98168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E800, 0xC3AB8E00, [_______ ________ POSTREAD 
POSTWRITE])
        -> Invalidate 0xC3B98170 (size: 0x00000012)
=> bus_dmamap_sync(0xC3A9E800, 0xC3AB8E00, [_______ ________ POSTREAD 
POSTWRITE])
        -> Invalidate 0xC3B98170 (size: 0x00000012)
=> bus_dmamap_sync(0xC35DC700, 0xC3AB8B40, [PREREAD PREWRITE ________ 
_________])
        -> Writeback Invalidate 0xC3B97168 (size: 0x00000008)
=> bus_dmamap_sync(0xC35DC700, 0xC3B6F500, [_______ ________ POSTREAD 
POSTWRITE])
        -> Invalidate 0xC3B97170 (size: 0x00000009)
=> bus_dmamap_sync(0xC35DC700, 0xC3B6F500, [_______ ________ POSTREAD 
POSTWRITE])
        -> Invalidate 0xC3B97170 (size: 0x00000009)
usb_alloc_device:1729: Failure selecting configuration index 0: USB_ERR_INVAL, 
port 1, addr 2 (ignored)
ugen0.2: <PQI> at usbus0

----- USB Stack with my patch -----

ugen0.1: <Marvell> at usbus0
uhub0: <Marvell EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <Marvell> at usbus1
uhub1: <Marvell EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ugen2.1: <Marvell> at usbus2
uhub2: <Marvell EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
Root mount waiting for: usbus2 usbus1 usbus0
uhub0: 1 port with 1 removable, self powered
uhub1: 1 port with 1 removable, self powered
uhub2: 1 port with 1 removable, self powered
=> bus_dmamap_sync(0xC3A9E780, 0xC3B6F540, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3B9E168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E500, 0xC3AB8C40, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3B9D168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E500, 0xC3AB8C00, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9D170 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E500, 0xC3AB8C00, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9D170 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3B9C168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000012)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000012)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3B9C168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000002)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000002)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3B9C168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000004)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000004)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3B9C168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000002)
Root mount waiting for: usbus0
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000002)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3B9C168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000022)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000022)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3B9C168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000002)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000002)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3B9C168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3B9C168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000002)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000002)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3B9C168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000022)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000022)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3B9C168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000009)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000009)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3B9C168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000020)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000020)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3B9C168 (size: 0x00000008)
ugen0.2: <pqi> at usbus0
umass0: <pqi IntelligentStick, class 0/0, rev 2.00/11.00, addr 2> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0x0000
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F080, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3B9C168 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000001)
=> bus_dmamap_sync(0xC3A9E280, 0xC3B6F140, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3B9C170 (size: 0x00000001)
Root mount waiting for: usbus0
umass0:0:0:-1: Attached to scbus0
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3BB23A8 (size: 0x0000001F)
Trying to mount root from nfs:10.0.0.201:/nfsroot/mv78100-4/
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB6098 (size: 0x00000024)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB6098 (size: 0x00000024)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3BB23A8 (size: 0x0000001F)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________ 
_________])
        -> Invalidate 0xC3BC0500 (size: 0x00000100)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________ 
_________])
        -> Invalidate 0xC3BC0500 (size: 0x00000100)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3BB23A8 (size: 0x0000001F)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3BB23A8 (size: 0x0000001F)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BBC0A0 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BBC0A0 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
da0 at umass-sim0 bus 0 target 0 lun 0
da0: <pqi IntelligentStick 1100> Removable Direct Access SCSI-0 device
da0: 40.000MB/s transfers
da0: 1912MB (3915776 512 byte sectors: 255H 63S/T 243C)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3BB23A8 (size: 0x0000001F)
NFS ROOT: 10.0.0.201:/nfsroot/mv78100-4/
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3AB9C80 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3AB9C80 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3BB23A8 (size: 0x0000001F)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3BB23A8 (size: 0x0000001F)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BBAC70 (size: 0x00000020)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BBAC70 (size: 0x00000020)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3BB23A8 (size: 0x0000001F)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________ 
_________])
        -> Invalidate 0xC3B6D800 (size: 0x00000200)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________ 
_________])
        -> Invalidate 0xC3B6D800 (size: 0x00000200)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3BB23A8 (size: 0x0000001F)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________ 
_________])
        -> Invalidate 0xC3C2A000 (size: 0x00002000)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________ 
_________])
        -> Invalidate 0xC3C2A000 (size: 0x00002000)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3BB23A8 (size: 0x0000001F)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3BB23A8 (size: 0x0000001F)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3BB23A8 (size: 0x0000001F)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3AB9B50 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3AB9B50 (size: 0x00000008)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3BB23A8 (size: 0x0000001F)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3BB23A8 (size: 0x0000001F)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BBAC70 (size: 0x00000020)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB00C0, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BBAC70 (size: 0x00000020)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3BB23A8 (size: 0x0000001F)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB0380, [_______ PREWRITE ________ 
_________])
        -> Writeback 0xC3BB23A8 (size: 0x0000001F)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)
=> bus_dmamap_sync(0xC3A9E480, 0xC3BB8080, [PREREAD ________ ________ 
_________])
        -> Writeback Invalidate 0xC3BB2770 (size: 0x0000000D)

-- 
Best regards,
Piotr Zi?cik
From tinguely at casselton.net  Tue Jul  7 14:02:34 2009
From: tinguely at casselton.net (Mark Tinguely)
Date: Tue Jul  7 14:02:40 2009
Subject: pmap problem in FreeBSD current
In-Reply-To: <200907061714.n66HEqsV063372@casselton.net>
Message-ID: <200907071402.n67E2WgH010354@casselton.net>


In arm_remap_nocache()/arm_unmap_nocache() a kernel shadow mapping is
made for BUS_DMA_COHERENT pages but is not removed. It is probably one
place that causes undeleted mappings required us to add the tail "if"
statement in pmap_nuke_pv() that I mentioned yesterday.

FYI future ideas: we can get rid of the shadow KVA for the BUS_DMA_COHERENT
by setting a bit in the page flags to denote a no_cache situation. The pmap
code can see that bit and refraim from turning caching on in pmap_fix_cache().

Speaking of pmap_nuke_pv() and in regards to my yesterday email, the more I
think of it, the more I believe the line "pg->md.pvh_attrs &= ~PVF_MOD;"
should remain in the code. The line, "vm_page_flag_clear(pg, PG_WRITEABLE);"
should be remove to avoid corruption.

It would be nice to remove all these unremoved mappings so that "if"
statement in pmap_nuke_pv() is unecessary. If we can't find all of
these unremoved mappings, we may need to clean out md.pv_kva when a
page is freed (or allocated) because we may be tricked into thinking the
page is shared when it really is not. This would not cause corruption,
but the cache will be turned off unnecessarily on a really unshared page.

vm_machdep.c:
void
arm_unmap_nocache(void *addr, vm_size_t size)
{
	vm_offset_t raddr = (vm_offset_t)addr;
	int i;

	size = round_page(size);
	i = (raddr - arm_nocache_startaddr) / (PAGE_SIZE);
-	for (; size > 0; size -= PAGE_SIZE, i++)
+	for (; size > 0; size -= PAGE_SIZE, i++) {
		arm_nocache_allocated[i / BITS_PER_INT] &= ~(1 << (i %
		    BITS_PER_INT));
+		pmap_kremove(raddr); 
+		raddr += PAGE_SIZE;
}

--Mark Tinguely
From tinguely at casselton.net  Tue Jul  7 14:37:33 2009
From: tinguely at casselton.net (Mark Tinguely)
Date: Tue Jul  7 14:37:40 2009
Subject: pmap problem in FreeBSD current
In-Reply-To: <200907071402.n67E2WgH010354@casselton.net>
Message-ID: <200907071437.n67EbVnX012439@casselton.net>


sf_buf_alloc()/sf_buf_free() in vm_machdep.c also leaves dangling kernel
mappings.

/*
 * Detatch mapped page and release resources back to the system.
 */
void
sf_buf_free(struct sf_buf *sf)
{
#ifndef ARM_USE_SMALL_ALLOC
	mtx_lock(&sf_buf_lock);
	sf->ref_count--;
	if (sf->ref_count == 0) {
		TAILQ_INSERT_TAIL(&sf_buf_freelist, sf, free_entry);
		nsfbufsused--;
+		pmap_kremove(sf->kva);
		if (sf_buf_alloc_want > 0)
			wakeup_one(&sf_buf_freelist);
	}
	mtx_unlock(&sf_buf_lock);
#endif

--Mark Tinguely.
From tinguely at casselton.net  Tue Jul  7 15:54:52 2009
From: tinguely at casselton.net (Mark Tinguely)
Date: Tue Jul  7 15:54:59 2009
Subject: pmap problem in FreeBSD current
In-Reply-To: <200907071437.n67EbVnX012439@casselton.net>
Message-ID: <200907071554.n67Fsouv016804@casselton.net>


Apologies for the noise. I am not familiar with the sf_buf_alloc/sf_buf_free
code and sent a bad code sequence. If we remove the mapping, it has to
come off the active list also:

/*
 * Detatch mapped page and release resources back to the system.
 */
void
sf_buf_free(struct sf_buf *sf)
{
#ifndef ARM_USE_SMALL_ALLOC
	mtx_lock(&sf_buf_lock);
	sf->ref_count--;
	if (sf->ref_count == 0) {
		TAILQ_INSERT_TAIL(&sf_buf_freelist, sf, free_entry);
		nsfbufsused--;
+		pmap_kremove(sf->kva);
+		sf->m = NULL;
+		LIST_REMOVE(sf, list_entry);
		if (sf_buf_alloc_want > 0)
			wakeup_one(&sf_buf_freelist);
	}
	mtx_unlock(&sf_buf_lock);
#endif

--Mark Tinguely.

From hselasky at c2i.net  Tue Jul  7 16:47:08 2009
From: hselasky at c2i.net (Hans Petter Selasky)
Date: Tue Jul  7 16:47:21 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <200907071550.28781.kosmo@semihalf.com>
References: <200906231035.43096.kosmo@semihalf.com>
	<200907051236.22783.hselasky@c2i.net>
	<200907071550.28781.kosmo@semihalf.com>
Message-ID: <200907071846.42086.hselasky@c2i.net>

On Tuesday 07 July 2009 15:50:28 Piotr Zi?cik wrote:
> > I did not find time yet to test on my AT91RM9200 board. I hope to do a
> > test this week. Did you at semihalf find anything?
>

Hi,

> I had Checked USB behaviour on PowerPC without hardware cache coherency.
> The problem also exists here and patch helps.

In my source code view the busdma sync function is empty for power-pc. Your 
patch should have no effect at all for sync operations?

>
> I also generated some logs which you asked for:

Thanks. I will try to make a build for AT91RM9200 tonight.

Did you test my suggested patch:

> Try adding:
>
>                 cpu_dcache_wb_range((vm_offset_t)buf, len);
>                 cpu_l2cache_wb_range((vm_offset_t)buf, len);
>
> Before:
>
>                         cpu_dcache_wbinv_range((vm_offset_t)buf, len);
>                         cpu_l2cache_wbinv_range((vm_offset_t)buf, len);
>
> In: sys/arm/arm/busdma_machdep.c

Or have you got any comments on that?

--HPS

From hselasky at c2i.net  Wed Jul  8 09:04:10 2009
From: hselasky at c2i.net (Hans Petter Selasky)
Date: Wed Jul  8 09:05:25 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <37C51279-42D8-49DE-8249-0DA386EBB062@semihalf.com>
References: <200906231035.43096.kosmo@semihalf.com>
	<200907071846.42086.hselasky@c2i.net>
	<37C51279-42D8-49DE-8249-0DA386EBB062@semihalf.com>
Message-ID: <200907081103.45388.hselasky@c2i.net>

On Wednesday 08 July 2009 10:58:17 Rafal Jaworowski wrote:
> On 2009-07-07, at 18:46, Hans Petter Selasky wrote:
> >> I had Checked USB behaviour on PowerPC without hardware cache
> >> coherency.
> >> The problem also exists here and patch helps.
> >
> > In my source code view the busdma sync function is empty for power-
> > pc. Your
> > patch should have no effect at all for sync operations?
>
> This was about the PPC4xx PowerPC port, not committed to the tree yet.
> Unlike most PowerPC this system has a de-coherent DMA and therefore
> its busdma sync is non empty, but needs to enforce coherency by
> software.
>
> The point is this is another platform on which USB stack (usb_pc_cpu_*
> functions) shows similar issues as reported for ARM and MIPS.

And what about my patch suggestion in my previous e-mail having the same 
subject. Does it work?

Regarding my testing on the AT91RM9200 I was short of time yesterday and will 
try to get it done today.

--HPS

From raj at semihalf.com  Wed Jul  8 09:14:27 2009
From: raj at semihalf.com (Rafal Jaworowski)
Date: Wed Jul  8 09:14:33 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <200907071846.42086.hselasky@c2i.net>
References: <200906231035.43096.kosmo@semihalf.com>
	<200907051236.22783.hselasky@c2i.net>
	<200907071550.28781.kosmo@semihalf.com>
	<200907071846.42086.hselasky@c2i.net>
Message-ID: <37C51279-42D8-49DE-8249-0DA386EBB062@semihalf.com>


On 2009-07-07, at 18:46, Hans Petter Selasky wrote:

>> I had Checked USB behaviour on PowerPC without hardware cache  
>> coherency.
>> The problem also exists here and patch helps.
>
> In my source code view the busdma sync function is empty for power- 
> pc. Your
> patch should have no effect at all for sync operations?

This was about the PPC4xx PowerPC port, not committed to the tree yet.  
Unlike most PowerPC this system has a de-coherent DMA and therefore  
its busdma sync is non empty, but needs to enforce coherency by  
software.

The point is this is another platform on which USB stack (usb_pc_cpu_*  
functions) shows similar issues as reported for ARM and MIPS.

Rafal

From kosmo at semihalf.com  Wed Jul  8 10:16:43 2009
From: kosmo at semihalf.com (Piotr =?iso-8859-2?q?Zi=EAcik?=)
Date: Wed Jul  8 10:16:49 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <200907081103.45388.hselasky@c2i.net>
References: <200906231035.43096.kosmo@semihalf.com>
	<37C51279-42D8-49DE-8249-0DA386EBB062@semihalf.com>
	<200907081103.45388.hselasky@c2i.net>
Message-ID: <200907081216.40100.kosmo@semihalf.com>

Wednesday 08 July 2009 11:03:43 Hans Petter Selasky napisa?(a):
>
> And what about my patch suggestion in my previous e-mail having the same
> subject. Does it work?
>

I have tested it and it does not work. By the way Writeback before Writeback 
Invalidate did not change cache behaviour too much. Writeback invalidate 
means flush all modified cache lines and then invalidate cache.

However looking into logs which I have sent you yesterdat I see one difference 
which may be significant. My patch changes Invalidate into Writeback 
Invalidate. In original code if driver write something to memory and then 
invalidate cache, the write will be lost. With my patch after change will
be written to memory and then cache will be invalidated. What do you think ?

-- 
Best Regards,
Piotr Ziecik
From sebastian.huber at embedded-brains.de  Wed Jul  8 10:40:58 2009
From: sebastian.huber at embedded-brains.de (Sebastian Huber)
Date: Wed Jul  8 10:41:05 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <200907081216.40100.kosmo@semihalf.com>
References: <200906231035.43096.kosmo@semihalf.com>	<37C51279-42D8-49DE-8249-0DA386EBB062@semihalf.com>	<200907081103.45388.hselasky@c2i.net>
	<200907081216.40100.kosmo@semihalf.com>
Message-ID: <4A547556.4050801@embedded-brains.de>

Hi!

The general approach for systems with no hardware cache coherence protocol
should be:

Memory to device DMA transfer of a buffer:

1. write all modified cache lines of the buffer back to memory (cache flush for
buffer)
2. start DMA transfer

Device to memory DMA transfer of a buffer:

1. invalidate the cache of the buffer, here you have to make sure that this
buffer does not share a cache line with other buffers (!)
2. start DMA transfer
3. all accesses to the buffer via the cache must wait until the DMA has finished

CU

-- 
Sebastian Huber, Embedded Brains GmbH

Address : Obere Lagerstr. 30, D-82178 Puchheim, Germany
Phone   : +49 89 18 90 80 79-6
Fax     : +49 89 18 90 80 79-9
E-Mail  : sebastian.huber@embedded-brains.de
PGP     : Public key available on request

Diese Nachricht ist keine gesch?ftliche Mitteilung im Sinne des EHUG.
From hselasky at c2i.net  Wed Jul  8 10:51:22 2009
From: hselasky at c2i.net (Hans Petter Selasky)
Date: Wed Jul  8 10:51:34 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <200907081216.40100.kosmo@semihalf.com>
References: <200906231035.43096.kosmo@semihalf.com>
	<200907081103.45388.hselasky@c2i.net>
	<200907081216.40100.kosmo@semihalf.com>
Message-ID: <200907081250.58151.hselasky@c2i.net>

Hi,

On Wednesday 08 July 2009 12:16:39 Piotr Zi?cik wrote:
> Wednesday 08 July 2009 11:03:43 Hans Petter Selasky napisa?(a):
> > And what about my patch suggestion in my previous e-mail having the same
> > subject. Does it work?
>
> I have tested it and it does not work. By the way Writeback before
> Writeback Invalidate did not change cache behaviour too much. Writeback
> invalidate means flush all modified cache lines and then invalidate cache.

By flush you mean write from CPU cache to RAM, right. And nothing else? You 
don't mean discard CPU cache by "flush" ???

>
> However looking into logs which I have sent you yesterdat I see one
> difference which may be significant. My patch changes Invalidate into
> Writeback Invalidate. In original code if driver write something to memory
> and then invalidate cache

If that is the case I'm very surprised. Could you make another printout for 
me. Compile the kernel with KDB, and add a call to "kdb_backtrace()" in 
bus_dmamap_sync() before printing out the writeback and/or invalidate, so that 
I can see the backtrace.

> , the write will be lost.

I'm aware about that.

> With my patch after
> change will be written to memory and then cache will be invalidated. What
> do you think ?

I still cannot see that I'm using the wrong flags when calling 
bus_dmamap_sync().

You are absoultely sure that your USB code is indentical to the now stock 8-
current USB code, and that there are no integration issues lurking?


In the USB "usb_pc_cpu_flush()" is used after CPU write operations to flush 
the CPU cache to RAM. "usb_pc_cpu_invalidate()" is used before CPU read 
operations. So far so good?

You are arguing that:

BUS_DMASYNC_POSTWRITE | BUS_DMASYNC_POSTREAD

flags do not cause a CPU invalidate?

Whilst the busdma manual is clear, that these flags are used after "an update 
of host memory by the device and prior to CPU access to host memory".

             BUS_DMASYNC_POSTREAD   Perform any synchronization required after
                                    an update of host memory by the device and
                                    prior to CPU access to host memory.

             BUS_DMASYNC_POSTWRITE  Perform any synchronization required after
                                    device access to host memory.

And that:

             BUS_DMASYNC_PREREAD    Perform any synchronization required prior
                                    to an update of host memory by the device.

             BUS_DMASYNC_PREWRITE   Perform any synchronization required after
                                    an update of host memory by the CPU and
                                    prior to device access to host memory.

flags do not cause a CPU flush prior to device acccess.


/*------------------------------------------------------------------------*
 *      usb_pc_cpu_invalidate - invalidate CPU cache
 *------------------------------------------------------------------------*/
void
usb_pc_cpu_invalidate(struct usb_page_cache *pc)
{
        if (pc->page_offset_end == pc->page_offset_buf) {
                /* nothing has been loaded into this page cache! */
                return;
        }
        bus_dmamap_sync(pc->tag, pc->map,
            BUS_DMASYNC_POSTWRITE | BUS_DMASYNC_POSTREAD);
}

/*------------------------------------------------------------------------*
 *      usb_pc_cpu_flush - flush CPU cache
 *------------------------------------------------------------------------*/
void
usb_pc_cpu_flush(struct usb_page_cache *pc)
{
        if (pc->page_offset_end == pc->page_offset_buf) {
                /* nothing has been loaded into this page cache! */
                return;
        }
        bus_dmamap_sync(pc->tag, pc->map,
            BUS_DMASYNC_PREWRITE | BUS_DMASYNC_PREREAD);
}

Thanks for being patient. If there is a bug, it will be fixed.

--HPS
From mih at semihalf.com  Wed Jul  8 14:01:01 2009
From: mih at semihalf.com (Michal Hajduk)
Date: Wed Jul  8 14:01:07 2009
Subject: pmap problem in FreeBSD current
In-Reply-To: <200907071554.n67Fsouv016804@casselton.net>
References: <200907071554.n67Fsouv016804@casselton.net>
Message-ID: <4A54A6B2.1020400@semihalf.com>

Mark Tinguely wrote:
> Apologies for the noise. I am not familiar with the sf_buf_alloc/sf_buf_free
> code and sent a bad code sequence. If we remove the mapping, it has to
> come off the active list also:
>
> /*
>  * Detatch mapped page and release resources back to the system.
>  */
> void
> sf_buf_free(struct sf_buf *sf)
> {
> #ifndef ARM_USE_SMALL_ALLOC
> 	mtx_lock(&sf_buf_lock);
> 	sf->ref_count--;
> 	if (sf->ref_count == 0) {
> 		TAILQ_INSERT_TAIL(&sf_buf_freelist, sf, free_entry);
> 		nsfbufsused--;
> +		pmap_kremove(sf->kva);
> +		sf->m = NULL;
> +		LIST_REMOVE(sf, list_entry);
> 		if (sf_buf_alloc_want > 0)
> 			wakeup_one(&sf_buf_freelist);
> 	}
> 	mtx_unlock(&sf_buf_lock);
> #endif
>
> --Mark Tinguely.
>
>   
Hello Mark,
Thank you for looking again on r194459.

I've tried to compile kernel using your patch with
changes regarding PG_WRITABLE and arm_unmap_nocache(),
but it didn't help.

My problems are rather related to memory corruption, which
I've observed in uma_core (pointers had bad values and
count of free items in slab had negative value).

I am still concerned about LOR, which has appeared in
svn commit r194459 (I wrote about it in my first mail).
Could you take a look at this LOR?

Thanks for your help.

Best regards,
Micha? Hajduk

From imp at bsdimp.com  Wed Jul  8 14:29:26 2009
From: imp at bsdimp.com (M. Warner Losh)
Date: Wed Jul  8 14:29:33 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <200907081103.45388.hselasky@c2i.net>
References: <200907071846.42086.hselasky@c2i.net>
	<37C51279-42D8-49DE-8249-0DA386EBB062@semihalf.com>
	<200907081103.45388.hselasky@c2i.net>
Message-ID: <20090708.082343.-1343671115.imp@bsdimp.com>

In message: <200907081103.45388.hselasky@c2i.net>
            Hans Petter Selasky <hselasky@c2i.net> writes:
: On Wednesday 08 July 2009 10:58:17 Rafal Jaworowski wrote:
: > On 2009-07-07, at 18:46, Hans Petter Selasky wrote:
: > >> I had Checked USB behaviour on PowerPC without hardware cache
: > >> coherency.
: > >> The problem also exists here and patch helps.
: > >
: > > In my source code view the busdma sync function is empty for power-
: > > pc. Your
: > > patch should have no effect at all for sync operations?
: >
: > This was about the PPC4xx PowerPC port, not committed to the tree yet.
: > Unlike most PowerPC this system has a de-coherent DMA and therefore
: > its busdma sync is non empty, but needs to enforce coherency by
: > software.
: >
: > The point is this is another platform on which USB stack (usb_pc_cpu_*
: > functions) shows similar issues as reported for ARM and MIPS.
: 
: And what about my patch suggestion in my previous e-mail having the same 
: subject. Does it work?
: 
: Regarding my testing on the AT91RM9200 I was short of time yesterday and will 
: try to get it done today.

I think that the root cause of this issue is two fold.

First, The 4 operations for busdma are being collapsed to only 2
operations.  There are four for a reason (since you have to do
different things for each case).

Second, and I think this is more important, I think that we're seeing
some cache-line poisoning.  We're flushing the entire busdma tag
rather than just one cache line since we don't have the API to do
that.  In addition, I don't think that the USB code is being careful
enough to ensure that we don't have buffers that live in the same
cache line that are simultaneously being used for read and write.  The
corruption we're seeing is a classic signal that this may be going on,
although I must admit that I've not done the detailed analysis to
prove it.

Warner
From tinguely at casselton.net  Wed Jul  8 15:07:33 2009
From: tinguely at casselton.net (Mark Tinguely)
Date: Wed Jul  8 15:07:39 2009
Subject: pmap problem in FreeBSD current
In-Reply-To: <4A54A6B2.1020400@semihalf.com>
Message-ID: <200907081507.n68F7Vsu070524@casselton.net>


>  Hello Mark,
>  Thank you for looking again on r194459.
>
>  I've tried to compile kernel using your patch with
>  changes regarding PG_WRITABLE and arm_unmap_nocache(),
>  but it didn't help.
>
>  My problems are rather related to memory corruption, which
>  I've observed in uma_core (pointers had bad values and
>  count of free items in slab had negative value).
>
>  I am still concerned about LOR, which has appeared in
>  svn commit r194459 (I wrote about it in my first mail).
>  Could you take a look at this LOR?
>
>  Thanks for your help.
>
>  Best regards,
>  Michał Hajduk


I don't think this is a corruption issue but an allocation of a l2_bucket
issue. The kernel map is locked to do the kenter. A new bucket is needed,
UMA is called and the new page is kenter into the map. In pmap_enter_pv()
before we allocate a pv_entry, we turn off the kmap lock. I am not sure if
the kmap lock should be turned off in the pmap_alloc_l2_bucket(), or the
caller, but here is a quick manually made  kmap lock disable patch:

I still think the last "vm_page_flag_clear(pg, PG_WRITEABLE);" needs
to be removed from pmap_nuke_pv(). All dangling kenters need to be
kremoved ...
		------------------------

/*
 * Returns a pointer to the L2 bucket associated with the specified pmap
 * and VA.
 *
 * If no L2 bucket exists, perform the necessary allocations to put an L2
 * bucket/page table in place.
 *
 * Note that if a new L2 bucket/page was allocated, the caller *must*
 * increment the bucket occupancy counter appropriately *before* 
 * releasing the pmap's lock to ensure no other thread or cpu deallocates
 * the bucket/page in the meantime.
 */
static struct l2_bucket *
pmap_alloc_l2_bucket(pmap_t pm, vm_offset_t va)
{
	struct l2_dtable *l2;
	struct l2_bucket *l2b;
	u_short l1idx;
+	int km = 0;

	l1idx = L1_IDX(va);

	PMAP_ASSERT_LOCKED(pm);
+	if ((pm != pmap_kernel()) && PMAP_OWNED(pmap_kernel()) {
+		km = 1;
+		PMAP_UNLOCK(pmap_kernel());
+	}
	mtx_assert(&vm_page_queue_mtx, MA_OWNED);
	if ((l2 = pm->pm_l2[L2_IDX(l1idx)]) == NULL) {
		/*
		 * No mapping at this address, as there is
		 * no entry in the L1 table.
		 * Need to allocate a new l2_dtable.
		 */
again_l2table:
		PMAP_UNLOCK(pm);
		vm_page_unlock_queues();
		if ((l2 = pmap_alloc_l2_dtable()) == NULL) {
			vm_page_lock_queues();
			PMAP_LOCK(pm);
+			if (km)
+				PMAP_LOCK(pmap_kernel());
			return (NULL);
		}
		vm_page_lock_queues();
		PMAP_LOCK(pm);
		if (pm->pm_l2[L2_IDX(l1idx)] != NULL) {
			PMAP_UNLOCK(pm);
			vm_page_unlock_queues();
			uma_zfree(l2table_zone, l2);
			vm_page_lock_queues();
			PMAP_LOCK(pm);
			l2 = pm->pm_l2[L2_IDX(l1idx)];
			if (l2 == NULL)
				goto again_l2table;
			/*
			 * Someone already allocated the l2_dtable while
			 * we were doing the same.
			 */
		} else {
			bzero(l2, sizeof(*l2));
			/*
			 * Link it into the parent pmap
			 */
			pm->pm_l2[L2_IDX(l1idx)] = l2;
		}
	} 

	l2b = &l2->l2_bucket[L2_BUCKET(l1idx)];

	/*
	 * Fetch pointer to the L2 page table associated with the address.
	 */
	if (l2b->l2b_kva == NULL) {
		pt_entry_t *ptep;

		/*
		 * No L2 page table has been allocated. Chances are, this
		 * is because we just allocated the l2_dtable, above.
		 */
again_ptep:
		PMAP_UNLOCK(pm);
		vm_page_unlock_queues();
		ptep = (void*)uma_zalloc(l2zone, M_NOWAIT|M_USE_RESERVE);
		vm_page_lock_queues();
		PMAP_LOCK(pm);
		if (l2b->l2b_kva != 0) {
			/* We lost the race. */
			PMAP_UNLOCK(pm);
			vm_page_unlock_queues();
			uma_zfree(l2zone, ptep);
			vm_page_lock_queues();
			PMAP_LOCK(pm);
			if (l2b->l2b_kva == 0)
				goto again_ptep;
			return (l2b);
		}
		l2b->l2b_phys = vtophys(ptep);
		if (ptep == NULL) {
			/*
			 * Oops, no more L2 page tables available at this
			 * time. We may need to deallocate the l2_dtable
			 * if we allocated a new one above.
			 */
			if (l2->l2_occupancy == 0) {
				pm->pm_l2[L2_IDX(l1idx)] = NULL;
				pmap_free_l2_dtable(l2);
			}
+			if (km)
+				PMAP_LOCK(pmap_kernel());
			return (NULL);
		}

		l2->l2_occupancy++;
		l2b->l2b_kva = ptep;
		l2b->l2b_l1idx = l1idx;
	}

+	if (km)
+		PMAP_LOCK(pmap_kernel());
	return (l2b);
}
From tinguely at casselton.net  Wed Jul  8 15:51:17 2009
From: tinguely at casselton.net (Mark Tinguely)
Date: Wed Jul  8 15:51:23 2009
Subject: pmap problem in FreeBSD current
In-Reply-To: <200907081507.n68F7Vsu070524@casselton.net>
Message-ID: <200907081551.n68FpFeM073177@casselton.net>

I forgot to CC the rest. pmap_get_pv_entry() is called from pmap_enter_locked
and can have the same UMA call that can happen in pmap_kenter_internal.
move the kmap lock protection from pmap_enter_pv() to pmap_get_pv_entry()

static void
pmap_enter_pv(struct vm_page *pg, struct pv_entry *pve, pmap_t pm,
    vm_offset_t va, u_int flags)
{

-	int km;

	mtx_assert(&vm_page_queue_mtx, MA_OWNED);

	if (pg->md.pv_kva) {
		/* PMAP_ASSERT_LOCKED(pmap_kernel()); */
		pve->pv_pmap = pmap_kernel();
		pve->pv_va = pg->md.pv_kva;
		pve->pv_flags = PVF_WRITE | PVF_UNMAN;
		pg->md.pv_kva = 0;

		TAILQ_INSERT_HEAD(&pg->md.pv_list, pve, pv_list);
		TAILQ_INSERT_HEAD(&pm->pm_pvlist, pve, pv_plist);
-		if ((km = PMAP_OWNED(pmap_kernel())))
-			PMAP_UNLOCK(pmap_kernel());
		vm_page_unlock_queues();
		if ((pve = pmap_get_pv_entry()) == NULL)
			panic("pmap_kenter_internal: no pv entries");
		vm_page_lock_queues();
-		if (km)
-			PMAP_LOCK(pmap_kernel());
	}

	PMAP_ASSERT_LOCKED(pm);
	pve->pv_pmap = pm;
	pve->pv_va = va;
	pve->pv_flags = flags;

	TAILQ_INSERT_HEAD(&pg->md.pv_list, pve, pv_list);
	TAILQ_INSERT_HEAD(&pm->pm_pvlist, pve, pv_plist);
	pg->md.pvh_attrs |= flags & (PVF_REF | PVF_MOD);
	if (pve->pv_flags & PVF_WIRED)
		++pm->pm_stats.wired_count;
	vm_page_flag_set(pg, PG_REFERENCED);
}

static void
pmap_free_pv_entry(pv_entry_t pv)
{
+	int km;
	pv_entry_count--;
+	if ((km = PMAP_OWNED(pmap_kernel())))
+		PMAP_UNLOCK(pmap_kernel());
	uma_zfree(pvzone, pv);
+	if (km)
+		PMAP_LOCK(pmap_kernel());
}


/*
 * get a new pv_entry, allocating a block from the system
 * when needed.
 * the memory allocation is performed bypassing the malloc code
 * because of the possibility of allocations at interrupt time.
 */
static pv_entry_t
pmap_get_pv_entry(void)
{
	pv_entry_t ret_value;
+	int km;
	
	pv_entry_count++;
+	if ((km = PMAP_OWNED(pmap_kernel())))
+		PMAP_UNLOCK(pmap_kernel());
	if (pv_entry_count > pv_entry_high_water)
		pagedaemon_wakeup();
	ret_value = uma_zalloc(pvzone, M_NOWAIT);
+	if (km)
+		PMAP_LOCK(pmap_kernel());
	return ret_value;
}
From vassilis.laganakos at yahoo.com  Thu Jul  9 05:55:37 2009
From: vassilis.laganakos at yahoo.com (Vassilis Laganakos)
Date: Thu Jul  9 05:55:43 2009
Subject: ARMv7 - EABI - Cross Compiler
In-Reply-To: <200906180305.n5I35qci045136@casselton.net>
References: <200906180305.n5I35qci045136@casselton.net>
Message-ID: <462282.87449.qm@web59413.mail.ac4.yahoo.com>


Hi Mark,

Sorry for the delayed reply, I was in transitional period, moving things around.


> >  I don't know what are the changes from previous versions of GCC with the  trunk.
> >  Maybe the release notes have more info on this....
> 
> FreeBSD compiler has a variable defined into it that the kernel sources
> expect to find, it has extended format commands, and it modifies
> gcc/gcc.c to look for the support programs in normal places.
> 
Oh ok,  I assume the info is in the handbook (or in the sources :) )

> >  > I made changes to the gcc-4.5-20090528 directory and the build directory.
> >  > I will look at the libraries too.
> >  > 
> >  Cool!
> 
> I am still using modified GNU make file. Static libraries can cross
> compile an application to ARM but I gave up on configure/gmake on
> libmudflap. My library compile process became a hacking in the configure
> files. I was just walking around the problems as they popped up. Had
> I read and understood all the configure options, maybe it would have
> gone smoother. The "-Werror" complaints and "xgcc -E" as a pre-processor
> command are the two biggest problem in the library compilation process.
> 
libmudflap gave me trouble too. Went through quite a few web pages to 
get my head around, still I'm not sure what to do to config things the way I want...
I guess the -Werror would be kind of expected in such transitions. 

> Once we have a cross compiler that can make a kernel, it seems a waste
> of time to keep going with the configure/gmake route. Your idea of
> moving the sources over to the head/contrib/gcc and editing the BSD
> make files in head/gnu/usr.bin/cc is the way to go because making
> the compiler from sources is part of the cross build world.
> 
Thanks, let's hope it works, and it is not crazy difficult to do...

> I diff-ed the sources from the GNU original (4.2.1 if memory serves me
> right) sources and filled in the missing pieces into the GCC 4.5 files.
> Some things have already been put into GCC 4.5, some areas seemed re-written.
> I was mostly interested in getting the FreeBSD extensions in. I can give you
> diff of my sources against GCC 4.5 original code or tar up both the source
> and build directory.  The changes would be simular for GCC 4.4.
> 
That would be very helpful! Is the number of missing pieces huge?

> There is nothing special about GCC 4.5, I just took the most recent source
> that is available. Also, I just blindly copied the freebsd* header files
> from head source into GCC 4.5. The real way of doing this is to edit up
> the existing GCC 4.5 header files and add the new items. I did that with
> a couple that are used in the ARM build.
> 
True, I think we have to merge the current header files with the new header
files to have the new items in. Also, some things in the 4.4 header arm files
seem to be doing some things in a way different than the linux-gnueabi way
that I'm more familiar with. Trying to understand :)  (once I have my stuff working
again I will *finally* put the info in a wiki...)

> The sources in the gcc support library files are not changed; it would
> just take some BSD make file editing to get them to compile under the
> head directory.
> 
Ok. 

> --Mark.

Once I get my hands back on this, I'll post here the progress and keep you informed.

Many thanks,
Vassilis L.



      

From kosmo at semihalf.com  Thu Jul  9 15:21:03 2009
From: kosmo at semihalf.com (Piotr =?iso-8859-2?q?Zi=EAcik?=)
Date: Thu Jul  9 15:21:14 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <200907081250.58151.hselasky@c2i.net>
References: <200906231035.43096.kosmo@semihalf.com>
	<200907081216.40100.kosmo@semihalf.com>
	<200907081250.58151.hselasky@c2i.net>
Message-ID: <200907091721.01272.kosmo@semihalf.com>

Wednesday 08 July 2009 12:50:56 Hans Petter Selasky napisa?(a):
> By flush you mean write from CPU cache to RAM, right. And nothing else? You
> don't mean discard CPU cache by "flush" ???

Yes. By flush I mean write all valid and modified cache lines to RAM.

> > However looking into logs which I have sent you yesterdat I see one
> > difference which may be significant. My patch changes Invalidate into
> > Writeback Invalidate. In original code if driver write something to
> > memory and then invalidate cache
>
> If that is the case I'm very surprised. Could you make another printout for
> me. Compile the kernel with KDB, and add a call to "kdb_backtrace()" in
> bus_dmamap_sync() before printing out the writeback and/or invalidate, so
> that I can see the backtrace.
>
> > , the write will be lost.
>
> I'm aware about that.

Full log with backtraces is here: 
http://people.freebsd.org/~raj/logs/usb-cache.log

> > With my patch after
> > change will be written to memory and then cache will be invalidated. What
> > do you think ?
>
> You are absoultely sure that your USB code is indentical to the now stock
> 8- current USB code, and that there are no integration issues lurking?
>

I am using current from 2009.06.15. In later versions is bug in ARM pmap
causing segmentation fault in ehci atatch code (for more details loot at
http://www.nabble.com/pmap-problem-in-FreeBSD-current-td24352351.html).


-- 
Pozdrawiam.
Piotr Zi?cik
From kosmo at semihalf.com  Thu Jul  9 15:23:14 2009
From: kosmo at semihalf.com (Piotr =?iso-8859-2?q?Zi=EAcik?=)
Date: Thu Jul  9 15:23:25 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <4A547556.4050801@embedded-brains.de>
References: <200906231035.43096.kosmo@semihalf.com>
	<200907081216.40100.kosmo@semihalf.com>
	<4A547556.4050801@embedded-brains.de>
Message-ID: <200907091723.13121.kosmo@semihalf.com>

Wednesday 08 July 2009 12:30:46 Sebastian Huber napisa?(a):
> Device to memory DMA transfer of a buffer:
> (...)
> 3. all accesses to the buffer via the cache must wait until the DMA has
> finished

And this is the problem in my oppinion - USB stack does not wait.

-- 
Best Regards.
Piotr Zi?cik
From hselasky at c2i.net  Thu Jul  9 16:35:08 2009
From: hselasky at c2i.net (Hans Petter Selasky)
Date: Thu Jul  9 16:35:20 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <200907091721.01272.kosmo@semihalf.com>
References: <200906231035.43096.kosmo@semihalf.com>
	<200907081250.58151.hselasky@c2i.net>
	<200907091721.01272.kosmo@semihalf.com>
Message-ID: <200907091834.42462.hselasky@c2i.net>

Hi Piotr,

On Thursday 09 July 2009 17:21:00 Piotr Zi?cik wrote:
> Wednesday 08 July 2009 12:50:56 Hans Petter Selasky napisa?(a):
> > By flush you mean write from CPU cache to RAM, right. And nothing else?
> > You don't mean discard CPU cache by "flush" ???
>
> Yes. By flush I mean write all valid and modified cache lines to RAM.
>
> > > However looking into logs which I have sent you yesterdat I see one
> > > difference which may be significant. My patch changes Invalidate into
> > > Writeback Invalidate. In original code if driver write something to
> > > memory and then invalidate cache
> >
> > If that is the case I'm very surprised. Could you make another printout
> > for me. Compile the kernel with KDB, and add a call to "kdb_backtrace()"
> > in bus_dmamap_sync() before printing out the writeback and/or invalidate,
> > so that I can see the backtrace.
> >
> > > , the write will be lost.
> >
> > I'm aware about that.
>
> Full log with backtraces is here:
> http://people.freebsd.org/~raj/logs/usb-cache.log

1) My analysis: Only the data areas are being flushed/invalidated. No transfer 
descriptors are flushed/invalidated. I see no cache operations happening on 
any DMA control structures, even though there are calls from EHCI to 
xxx_pc_flush() and xxx_pc_invalidate().

One patch you can try is to add an additional unload call to 
"usbd_transfer_setup_sub_malloc()":

==== //depot/projects/usb/src/sys/dev/usb/usb_transfer.c#159 - 
/home/hans_other/usb.p4/src/sys/dev/usb/usb_transfer.c ====
@@ -268,6 +268,10 @@
                    pg, z, align)) {
                        return (1);     /* failure */
                }
+
+               bus_dmamap_unload(parm->dma_page_cache_ptr->tag,
+                       parm->dma_page_cache_ptr->map);
+
                /* Set beginning of current buffer */
                buf = parm->dma_page_cache_ptr->buffer;
                /* Make room for one DMA page cache and one page */

This will avoid the same memory area being loaded twice. Not sure if there is 
still a bug in pmap about this!

2) You can also try to remove the "BUS_DMA_COHERENT" flag from "usb_busdma.c".

My time is limited today, because I have to work on libusb v1.0 support.

Could you provide a new trace, showing an enumeration failure. In your 
previous trace there was no error, because the printouts probably caused to 
CPU to flush out its cache.

Debug details: /boot/loader.conf

hw.usb.debug=15
hw.usb.ehci.debug=15
hw.usb.ohci.debug=15
hw.usb.uhci.debug=15

> I am using current from 2009.06.15. In later versions is bug in ARM pmap
> causing segmentation fault in ehci atatch code (for more details loot at
> http://www.nabble.com/pmap-problem-in-FreeBSD-current-td24352351.html).

Ok.

I have a feeling we will soon figure out the problem.

--HPS

From hselasky at c2i.net  Thu Jul  9 16:46:04 2009
From: hselasky at c2i.net (Hans Petter Selasky)
Date: Thu Jul  9 16:46:10 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <200907091834.42462.hselasky@c2i.net>
References: <200906231035.43096.kosmo@semihalf.com>
	<200907091721.01272.kosmo@semihalf.com>
	<200907091834.42462.hselasky@c2i.net>
Message-ID: <200907091845.40515.hselasky@c2i.net>

On Thursday 09 July 2009 18:34:40 Hans Petter Selasky wrote:
> > Full log with backtraces is here:
> > http://people.freebsd.org/~raj/logs/usb-cache.log
>
> 1) My analysis: Only the data areas are being flushed/invalidated.

I also see that the flushing/invalidating is being done correctly.

--HPS
From mih at semihalf.com  Fri Jul 10 08:27:07 2009
From: mih at semihalf.com (Michal Hajduk)
Date: Fri Jul 10 08:27:14 2009
Subject: pmap problem in FreeBSD current
In-Reply-To: <200907081551.n68FpFeM073177@casselton.net>
References: <200907081551.n68FpFeM073177@casselton.net>
Message-ID: <4A56FB72.9040807@semihalf.com>

Hi Mark,
We've diagnosed the problem and it was related to arm_nocache_remap().
When the page was remaped into non-cacheable virtual memory region
there was no dcache write-back operation, so count of free items in
slab had differ value then should had.
This write-back operation should have been done in pmap_set_cache_entry()
but PVF_REF flag was missing so this part of code was omitted.

I think that we should enable PVF_REF flag in pmap_enter_pv().

================================================

diff --git a/sys/arm/arm/pmap.c b/sys/arm/arm/pmap.c
index 3cdab65..133dc6d 100644
--- a/sys/arm/arm/pmap.c
+++ b/sys/arm/arm/pmap.c
@@ -1683,7 +1683,7 @@ pmap_enter_pv(struct vm_page *pg, struct pv_entry 
*pve, pmap_t pm,
/* PMAP_ASSERT_LOCKED(pmap_kernel()); */
pve->pv_pmap = pmap_kernel();
pve->pv_va = pg->md.pv_kva;
- pve->pv_flags = PVF_WRITE | PVF_UNMAN;
+ pve->pv_flags = PVF_WRITE | PVF_UNMAN | PVF_REF;
pg->md.pv_kva = 0;

TAILQ_INSERT_HEAD(&pg->md.pv_list, pve, pv_list);

=================================================

With this patch I've had no panics.

Many thanks,
Micha? Hajduk
From tinguely at casselton.net  Fri Jul 10 13:07:50 2009
From: tinguely at casselton.net (Mark Tinguely)
Date: Fri Jul 10 13:07:56 2009
Subject: pmap problem in FreeBSD current
In-Reply-To: <4A56FB72.9040807@semihalf.com>
Message-ID: <200907101307.n6AD7lZT081110@casselton.net>


>  Hi Mark,
>  We've diagnosed the problem and it was related to arm_nocache_remap().
>  When the page was remaped into non-cacheable virtual memory region
>  there was no dcache write-back operation, so count of free items in
>  slab had differ value then should had.
>  This write-back operation should have been done in pmap_set_cache_entry()
>  but PVF_REF flag was missing so this part of code was omitted.
>
>  I think that we should enable PVF_REF flag in pmap_enter_pv().
>
>  ================================================
>
>  diff --git a/sys/arm/arm/pmap.c b/sys/arm/arm/pmap.c
>  index 3cdab65..133dc6d 100644
>  --- a/sys/arm/arm/pmap.c
>  +++ b/sys/arm/arm/pmap.c
>  @@ -1683,7 +1683,7 @@ pmap_enter_pv(struct vm_page *pg, struct pv_entry 
>  *pve, pmap_t pm,
>  /* PMAP_ASSERT_LOCKED(pmap_kernel()); */
>  pve->pv_pmap = pmap_kernel();
>  pve->pv_va = pg->md.pv_kva;
>  - pve->pv_flags = PVF_WRITE | PVF_UNMAN;
>  + pve->pv_flags = PVF_WRITE | PVF_UNMAN | PVF_REF;
>  pg->md.pv_kva = 0;
>
>  TAILQ_INSERT_HEAD(&pg->md.pv_list, pve, pv_list);
>
>  =================================================
>
>  With this patch I've had no panics.
>
>  Many thanks,
>  Michał Hajduk
>

Good job. This strange, your inclusion writes back the whole page of the
original buffer. This writeback of just the original buffer size should be
done in bus_dmamem_alloc() in busdma_machdep.c after the arm_remap_nocache()
with the line "cpu_idcache_wbinv_range(vm_offset_t)*vaddr, dmat->maxsize)".

I wonder if your cpu_idcache_wbinv_range() implementation correctly
wb/invalidates when the buffer is not cache line aligned or is smaller
than a cache-line? The other situation that would do this is if your
cache line larger than you think. I know the ARMv7 use 64 byte caches
instead of the more traditional 32 bytes. This should not be your case
because last week I looked up your processor information and thought it
said it used a 32 byte cache line.

Did http://www.casselton.net/~tinguely/arm_pmap_unmanaged.diff fix the lock
problems?

Thank-you for the test information and the code scouting.

--Mark Tinguely.
From tinguely at casselton.net  Fri Jul 10 14:54:08 2009
From: tinguely at casselton.net (Mark Tinguely)
Date: Fri Jul 10 14:54:14 2009
Subject: pmap problem in FreeBSD current - PS
In-Reply-To: <200907101307.n6AD7lZT081110@casselton.net>
Message-ID: <200907101454.n6AEs7nJ087492@casselton.net>


PS. sorry to even question your cpu_(l2)idcache_wbinv_range() implemetation.
    the cpu_idcache_wbinv_range() in bus_dmamem_alloc() is wrong.
    arm_remap_nocache() should wb/inv the entire page especially if the
    buffer that uses the page is less than a page size or non-page aligned. 
    The cache lines NOT in the buffer is the data that can be lost if
    used while the nocache shadow mapping is in effect.

   As an open question to the group: should we add an explict page
   cpu_(l2)idcache_wbinv_range() in the arm_remap_nocache() routine,
   so we know that it happens rather than a side effect of the new
   managing of kernel pages?

   For freebsd-9, I will rewrite arm_remap_nocache() to get rid of the
   shadow map.

Did your lock problems go away?

Good work.

--Mark Tinguely
From Toby at IACmusic.com  Sat Jul 11 08:52:59 2009
From: Toby at IACmusic.com (Toby@IACmusic.com)
Date: Sat Jul 11 08:53:54 2009
Subject: Finally, a song contest that is free to enter.  $27,
	000+ in prizes too!
Message-ID: <CHILKAT-MID-feda9ffe-c6aa-ab69-61a6-1b3b570d6dcc@FT-PC>


   
 


   Hi,

   




   


   IACmusic.com has started a major song contest and this one you can
   enter for free! &n   throwing a party that     Got a 
song   be a lot of 
   




   


   It's our [1]YEAR 
OF THE I   "Indiependents" Day July 
4th!&nb   categories to choose from to enter your song in, 
including Songwri   ting.  The Grand Prize is a huge package that 
includes $1000 wor   th of musical equipment (whatever you need), 2 
weeks stay in a c   ondo suite at your choice of a number of US 
vacation spots, an i   Pod Shuffle, and a IAC Prime Perpetual Lifetime 
membership.  But   there are also 3 nice prizes in each of 16 
categories and you can
   en   exposur   
competition.
   He   will hit 
   




   


   Go [3]here 

   




   


   Good luc
   




   


   The Staff at IACmusic.com (the Ind
   
 
 


References

   1. file://localhost/tmp/3D"htt   2. 3D"http://iacmusic.com/quickSignup.aspx"
   3. file://localhost/tmp/3D
From mih at semihalf.com  Mon Jul 13 08:39:31 2009
From: mih at semihalf.com (Michal Hajduk)
Date: Mon Jul 13 08:39:38 2009
Subject: pmap problem in FreeBSD current - PS
In-Reply-To: <200907101454.n6AEs7nJ087492@casselton.net>
References: <200907101454.n6AEs7nJ087492@casselton.net>
Message-ID: <4A5AF2DA.9050101@semihalf.com>

Hi Mark,

I've corrected busdma_machdep instead of adding this PVF_REF flag to arm
pmap code and it works good.
My patch:
===========================================================
diff --git a/sys/arm/arm/busdma_machdep.c b/sys/arm/arm/busdma_machdep.c
index a8b2de9..b55a714 100644
--- a/sys/arm/arm/busdma_machdep.c
+++ b/sys/arm/arm/busdma_machdep.c
@@ -631,10 +631,10 @@ bus_dmamem_alloc(bus_dma_tag_t dmat, void** vaddr, 
int flags,
((vm_offset_t)*vaddr & PAGE_MASK));
newmap->origbuffer = *vaddr;
newmap->allocbuffer = tmpaddr;
- cpu_idcache_wbinv_range((vm_offset_t)*vaddr,
- dmat->maxsize);
- cpu_l2cache_wbinv_range((vm_offset_t)*vaddr,
- dmat->maxsize);
+ cpu_idcache_wbinv_range((vm_offset_t)*vaddr &
+ ~PAGE_MASK, PAGE_SIZE);
+ cpu_l2cache_wbinv_range((vm_offset_t)*vaddr &
+ ~PAGE_MASK, PAGE_SIZE);
*vaddr = tmpaddr;
} else
newmap->origbuffer = newmap->allocbuffer = NULL;

============================================================

While debugging this problem we've found another in pmap_kremove(). 
There was an
invalidation on va + PAGE_SIZE instead of page which contains va address.
My patch:

=====================================================
diff --git a/sys/arm/arm/pmap.c b/sys/arm/arm/pmap.c
index 3cdab65..7988f40 100644
--- a/sys/arm/arm/pmap.c
+++ b/sys/arm/arm/pmap.c
@@ -2984,8 +2984,8 @@ pmap_kremove(vm_offset_t va)
pmap_free_pv_entry(pve);
PMAP_UNLOCK(pmap_kernel());
vm_page_unlock_queues();
- cpu_dcache_wbinv_range(va, PAGE_SIZE);
- cpu_l2cache_wbinv_range(va, PAGE_SIZE);
+ cpu_dcache_wbinv_range(va & ~PAGE_MASK, PAGE_SIZE);
+ cpu_l2cache_wbinv_range(va & ~PAGE_MASK, PAGE_SIZE);
cpu_tlb_flushD_SE(va);
cpu_cpwait();
*pte = 0;
=====================================================

I suggest, that maybe for now we should only change the 
cpu_(l2)idcache_wbinv_range()
routines by adding simple check:
If the va address is page aligned and size is equal to PAGE_SIZE we 
should write-back
and invalidate whole page. Otherwise we should invalidate line by line.

I've also checked your patch and it didn't help (I had a panic)
...
uhub0: 1 port with 1 removable, self powered
Root mount waiting for: usbus0
panic: blockable sleep lock (sleep mutex) vm page queue mutex @ 
/home/mih/git/marvell-current/sys/arm/arm/pmap.c:1947
KDB: enter: panic
[thread pid 15 tid 100028 ]
Stopped at $d: ldrb r15, [r15, r15, ror r15]!

Many thanks,
Micha? Hajduk
From bugmaster at FreeBSD.org  Mon Jul 13 11:06:51 2009
From: bugmaster at FreeBSD.org (FreeBSD bugmaster)
Date: Mon Jul 13 11:07:42 2009
Subject: Current problem reports assigned to freebsd-arm@FreeBSD.org
Message-ID: <200907131106.n6DB6oKV040522@freefall.freebsd.org>

Note: to view an individual PR, use:
  http://www.freebsd.org/cgi/query-pr.cgi?pr=(number).

The following is a listing of current problems submitted by FreeBSD users.
These represent problem reports covering all versions including
experimental development code and obsolete releases.


S Tracker      Resp.      Description
--------------------------------------------------------------------------------
o arm/134368   arm        [patch] nslu2_led driver for the LEDs on the NSLU2
o arm/134338   arm        [patch] Lock GPIO accesses on ixp425
o arm/134092   arm        [patch] NSLU.hints contains wrong hints for on board n

3 problems total.

From mih at semihalf.com  Mon Jul 13 12:57:09 2009
From: mih at semihalf.com (Michal Hajduk)
Date: Mon Jul 13 12:57:16 2009
Subject: pmap problem in FreeBSD current - PS
In-Reply-To: <4A5AF2DA.9050101@semihalf.com>
References: <200907101454.n6AEs7nJ087492@casselton.net>
	<4A5AF2DA.9050101@semihalf.com>
Message-ID: <4A5B2F3C.4030501@semihalf.com>

Michal Hajduk wrote:
> Hi Mark,
>
> I've corrected busdma_machdep instead of adding this PVF_REF flag to arm
> pmap code and it works good.
> My patch:
> ===========================================================
> diff --git a/sys/arm/arm/busdma_machdep.c b/sys/arm/arm/busdma_machdep.c
> index a8b2de9..b55a714 100644
> --- a/sys/arm/arm/busdma_machdep.c
> +++ b/sys/arm/arm/busdma_machdep.c
> ....
I've found another problem in busdma_machdep.c in function 
bus_dmamem_alloc.
After write-back invalidate operation there is no tlb flush.
Below there is complete patch:

============================================================
diff --git a/sys/arm/arm/busdma_machdep.c b/sys/arm/arm/busdma_machdep.c
index a8b2de9..9562615 100644
--- a/sys/arm/arm/busdma_machdep.c
+++ b/sys/arm/arm/busdma_machdep.c
@@ -631,10 +631,11 @@ bus_dmamem_alloc(bus_dma_tag_t dmat, void** vaddr, 
int flags,
((vm_offset_t)*vaddr & PAGE_MASK));
newmap->origbuffer = *vaddr;
newmap->allocbuffer = tmpaddr;
- cpu_idcache_wbinv_range((vm_offset_t)*vaddr,
- dmat->maxsize);
- cpu_l2cache_wbinv_range((vm_offset_t)*vaddr,
- dmat->maxsize);
+ cpu_dcache_wbinv_range((vm_offset_t)*vaddr &
+ ~PAGE_MASK, PAGE_SIZE);
+ cpu_l2cache_wbinv_range((vm_offset_t)*vaddr &
+ ~PAGE_MASK, PAGE_SIZE);
+ cpu_tlb_flushID_SE((vm_offset_t)*vaddr);
*vaddr = tmpaddr;
} else
newmap->origbuffer = newmap->allocbuffer = NULL;

============================================================

> I've also checked your patch and it didn't help (I had a panic)
> ...
> uhub0: 1 port with 1 removable, self powered
> Root mount waiting for: usbus0
> panic: blockable sleep lock (sleep mutex) vm page queue mutex @ 
> /home/mih/git/marvell-current/sys/arm/arm/pmap.c:1947
> KDB: enter: panic
> [thread pid 15 tid 100028 ]
> Stopped at $d: ldrb r15, [r15, r15, ror r15]!
>
> Many thanks,
> Micha? Hajduk
>
Can we move vm_page_unlock_queues() (line 1693) and vm_page_lock_queues to
pmap_get_pv_entry() ? Is it added only because of pagedaemon_wakeup()?

If this is correct, I've prepared a patch:
===========================================================
diff --git a/sys/arm/arm/pmap.c b/sys/arm/arm/pmap.c
index 3cdab65..b4ffbbf 100644
--- a/sys/arm/arm/pmap.c
+++ b/sys/arm/arm/pmap.c
@@ -1690,10 +1690,8 @@ pmap_enter_pv(struct vm_page *pg, struct pv_entry 
*pve, pmap_t pm,
TAILQ_INSERT_HEAD(&pm->pm_pvlist, pve, pv_plist);
if ((km = PMAP_OWNED(pmap_kernel())))
PMAP_UNLOCK(pmap_kernel());
- vm_page_unlock_queues();
if ((pve = pmap_get_pv_entry()) == NULL)
panic("pmap_kenter_internal: no pv entries");
- vm_page_lock_queues();
if (km)
PMAP_LOCK(pmap_kernel());
}
@@ -2904,10 +2902,8 @@ pmap_kenter_internal(vm_offset_t va, vm_offset_t 
pa, int flags)
vm_page_lock_queues();
if (!TAILQ_EMPTY(&m->md.pv_list) || m->md.pv_kva) {
/* release vm_page lock for pv_entry UMA */
- vm_page_unlock_queues();
if ((pve = pmap_get_pv_entry()) == NULL)
panic("pmap_kenter_internal: no pv entries");
- vm_page_lock_queues();
PMAP_LOCK(pmap_kernel());
pmap_enter_pv(m, pve, pmap_kernel(), va,
PVF_WRITE | PVF_UNMAN);
@@ -2984,8 +2980,8 @@ pmap_kremove(vm_offset_t va)
pmap_free_pv_entry(pve);
PMAP_UNLOCK(pmap_kernel());
vm_page_unlock_queues();
- cpu_dcache_wbinv_range(va, PAGE_SIZE);
- cpu_l2cache_wbinv_range(va, PAGE_SIZE);
+ cpu_dcache_wbinv_range(va & ~PAGE_MASK, PAGE_SIZE);
+ cpu_l2cache_wbinv_range(va & ~PAGE_MASK, PAGE_SIZE);
cpu_tlb_flushD_SE(va);
cpu_cpwait();
*pte = 0;
@@ -3935,8 +3931,11 @@ pmap_get_pv_entry(void)
pv_entry_t ret_value;
pv_entry_count++;
- if (pv_entry_count > pv_entry_high_water)
+ if (pv_entry_count > pv_entry_high_water) {
+ vm_page_unlock_queues();
pagedaemon_wakeup();
+ vm_page_lock_queues();
+ }
ret_value = uma_zalloc(pvzone, M_NOWAIT);
return ret_value;
}
==========================================================

With this patch I didn't observe LORs (as well as with your + mine patch).

Thanks for your help.

Best regards,
Micha? Hajduk
From krassi at bulinfo.net  Tue Jul 14 08:28:05 2009
From: krassi at bulinfo.net (Krassimir Slavchev)
Date: Tue Jul 14 08:28:12 2009
Subject: ohci not detected on at91rm9200?
Message-ID: <4A5C3BF5.9030200@bulinfo.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

The ohci controller is not detected by FreeBSD 8.0-BETA1 r195653.
Last working kernel I have is r187522 (~ Jan 2009) with old USB stack.
I have only removed the 'ugen' device from my kernel config file.

Can someone confirm this? Or may be I miss something?

Best regards
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (FreeBSD)

iD8DBQFKXDv1xJBWvpalMpkRAmmNAJ4q6n4nIkLhXwQc8FlI1hthMW5G0QCgrbBi
feMkpQcoM3/a5wVXfJMUak8=
=Mdqp
-----END PGP SIGNATURE-----
From kosmo at semihalf.com  Tue Jul 14 08:31:13 2009
From: kosmo at semihalf.com (Piotr =?iso-8859-2?q?Zi=EAcik?=)
Date: Tue Jul 14 08:31:19 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <200907091834.42462.hselasky@c2i.net>
References: <200906231035.43096.kosmo@semihalf.com>
	<200907091721.01272.kosmo@semihalf.com>
	<200907091834.42462.hselasky@c2i.net>
Message-ID: <200907141031.11185.kosmo@semihalf.com>

>
> 1) My analysis: Only the data areas are being flushed/invalidated. No
> transfer descriptors are flushed/invalidated. I see no cache operations
> happening on any DMA control structures, even though there are calls from
> EHCI to xxx_pc_flush() and xxx_pc_invalidate().
>

This is 100% correct if control structures are marked as BUS_DMA_COHERENT.

> One patch you can try is to add an additional unload call to
> "usbd_transfer_setup_sub_malloc()":
>
> ==== //depot/projects/usb/src/sys/dev/usb/usb_transfer.c#159 -
> /home/hans_other/usb.p4/src/sys/dev/usb/usb_transfer.c ====
> @@ -268,6 +268,10 @@
>                     pg, z, align)) {
>                         return (1);     /* failure */
>                 }
> +
> +               bus_dmamap_unload(parm->dma_page_cache_ptr->tag,
> +                       parm->dma_page_cache_ptr->map);
> +
>                 /* Set beginning of current buffer */
>                 buf = parm->dma_page_cache_ptr->buffer;
>                 /* Make room for one DMA page cache and one page */
>
> This will avoid the same memory area being loaded twice. Not sure if there
> is still a bug in pmap about this!

I checked this patch and I did not saw any changes in USB behaviour.
The problem sill was visible.

>
> Could you provide a new trace, showing an enumeration failure. In your
> previous trace there was no error, because the printouts probably caused to
> CPU to flush out its cache.
>

I have tried many times and I was unable to generate trace showing enumeration
faliure. Even little amount of debugging code caused proper enumeration. 
However access to attached device (pendrive) was not possible - SCSI layer 
reported errors.


Probaby you see more on your AT91 device as you know USB stack internals. Have 
you tried to bring up OHCI on you ARM board ?

-- 
Best Regards,
Piotr Ziecik
From hselasky at c2i.net  Tue Jul 14 08:37:05 2009
From: hselasky at c2i.net (Hans Petter Selasky)
Date: Tue Jul 14 08:37:17 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <200907141031.11185.kosmo@semihalf.com>
References: <200906231035.43096.kosmo@semihalf.com>
	<200907091834.42462.hselasky@c2i.net>
	<200907141031.11185.kosmo@semihalf.com>
Message-ID: <200907141036.44652.hselasky@c2i.net>

On Tuesday 14 July 2009 10:31:10 Piotr Zi?cik wrote:
> > 1) My analysis: Only the data areas are being flushed/invalidated. No
> > transfer descriptors are flushed/invalidated. I see no cache operations
> > happening on any DMA control structures, even though there are calls from
> > EHCI to xxx_pc_flush() and xxx_pc_invalidate().
>

> Probaby you see more on your AT91 device as you know USB stack internals.
> Have you tried to bring up OHCI on you ARM board ?

Not yet. I'm terribly busy with some LibUSB stuff headed for the 8-current 
release. As soon as I find time I will fire off a build and debug.

BTW: Has pmap been fixed for ARM in 8-current?

--HPS
From raj at semihalf.com  Tue Jul 14 09:21:42 2009
From: raj at semihalf.com (Rafal Jaworowski)
Date: Tue Jul 14 09:21:53 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <200907141036.44652.hselasky@c2i.net>
References: <200906231035.43096.kosmo@semihalf.com>
	<200907091834.42462.hselasky@c2i.net>
	<200907141031.11185.kosmo@semihalf.com>
	<200907141036.44652.hselasky@c2i.net>
Message-ID: <FB60465B-D714-4534-B07F-890C72A9CB33@semihalf.com>


On 2009-07-14, at 10:36, Hans Petter Selasky wrote:

> On Tuesday 14 July 2009 10:31:10 Piotr Zi?cik wrote:
>>> 1) My analysis: Only the data areas are being flushed/invalidated.  
>>> No
>>> transfer descriptors are flushed/invalidated. I see no cache  
>>> operations
>>> happening on any DMA control structures, even though there are  
>>> calls from
>>> EHCI to xxx_pc_flush() and xxx_pc_invalidate().
>>
>
>> Probaby you see more on your AT91 device as you know USB stack  
>> internals.
>> Have you tried to bring up OHCI on you ARM board ?
>
> Not yet. I'm terribly busy with some LibUSB stuff headed for the 8- 
> current
> release. As soon as I find time I will fire off a build and debug.

Please note these problems should be considered as a showstopper for  
the release since USB is currently broken on at least three ARM  
platforms in the tree (Marvell).

> BTW: Has pmap been fixed for ARM in 8-current?

Seems like the most critical problems (panics) are resolved and will  
be pushed into SVN shortly. In case you'd like to apply the fix  
directly, see: http://people.freebsd.org/~raj/patches/arm/pmap-fixes.diff

Rafal

From hselasky at c2i.net  Tue Jul 14 09:31:19 2009
From: hselasky at c2i.net (Hans Petter Selasky)
Date: Tue Jul 14 09:31:26 2009
Subject: ohci not detected on at91rm9200?
In-Reply-To: <4A5C3BF5.9030200@bulinfo.net>
References: <4A5C3BF5.9030200@bulinfo.net>
Message-ID: <200907141030.57784.hselasky@c2i.net>

On Tuesday 14 July 2009 10:04:05 Krassimir Slavchev wrote:
> Hi,
>
> The ohci controller is not detected by FreeBSD 8.0-BETA1 r195653.
> Last working kernel I have is r187522 (~ Jan 2009) with old USB stack.
> I have only removed the 'ugen' device from my kernel config file.
>
> Can someone confirm this? Or may be I miss something?
>

Does your kernel contain: "device ohci". Also diff the example config files 
for AT91RM9200 with yours under /sys/arm/conf .

--HPS

From krassi at bulinfo.net  Tue Jul 14 10:07:51 2009
From: krassi at bulinfo.net (Krassimir Slavchev)
Date: Tue Jul 14 10:07:58 2009
Subject: ohci not detected on at91rm9200?
In-Reply-To: <200907141030.57784.hselasky@c2i.net>
References: <4A5C3BF5.9030200@bulinfo.net>
	<200907141030.57784.hselasky@c2i.net>
Message-ID: <4A5C58F0.5050904@bulinfo.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Petter Selasky wrote:
> On Tuesday 14 July 2009 10:04:05 Krassimir Slavchev wrote:
>> Hi,
>>
>> The ohci controller is not detected by FreeBSD 8.0-BETA1 r195653.
>> Last working kernel I have is r187522 (~ Jan 2009) with old USB stack.
>> I have only removed the 'ugen' device from my kernel config file.
>>
>> Can someone confirm this? Or may be I miss something?
>>
> 
> Does your kernel contain: "device ohci". Also diff the example config files 
> for AT91RM9200 with yours under /sys/arm/conf .

Yes, it does:
kldstat -v | grep ohci
                24 ohci/usbus

With last working kernel from Jan:
...
ohci0: <AT91 integrated ohci controller> mem 0xdfe00000-0xdfefffff irq
23 on atmelarm0
ohci0: [GIANT-LOCKED]
ohci0: [ITHREAD]
usb0: OHCI version 1.0
usb0 on ohci0
usb0: USB revision 1.0
uhub0: <Atmel OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhub0: device problem (IOERROR), disabling port 2
...

I just upgraded the source tree and removed the ugen device from the
previous kernel config file.


> 
> --HPS
> 
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (FreeBSD)

iD8DBQFKXFjvxJBWvpalMpkRAlcYAJ9ThGnoLB1CVNtQtBR1xJpm2NoEJACfZyGR
vfuHD4i+U7IEl14OROb72Yk=
=YJ4r
-----END PGP SIGNATURE-----
From krassi at bulinfo.net  Tue Jul 14 11:36:18 2009
From: krassi at bulinfo.net (Krassimir Slavchev)
Date: Tue Jul 14 11:36:25 2009
Subject: ohci not detected on at91rm9200?
In-Reply-To: <200907141030.57784.hselasky@c2i.net>
References: <4A5C3BF5.9030200@bulinfo.net>
	<200907141030.57784.hselasky@c2i.net>
Message-ID: <4A5C6DAC.6030103@bulinfo.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Petter Selasky wrote:
> On Tuesday 14 July 2009 10:04:05 Krassimir Slavchev wrote:
>> Hi,
>>
>> The ohci controller is not detected by FreeBSD 8.0-BETA1 r195653.
>> Last working kernel I have is r187522 (~ Jan 2009) with old USB stack.
>> I have only removed the 'ugen' device from my kernel config file.
>>
>> Can someone confirm this? Or may be I miss something?
>>
> 
> Does your kernel contain: "device ohci". Also diff the example config files 
> for AT91RM9200 with yours under /sys/arm/conf .
> 
> --HPS
> 
> 

This fixes the problem:

- --- files.at91  (revision 195653)
+++ files.at91  (working copy)
@@ -18,6 +18,7 @@
 arm/at91/uart_bus_at91usart.c  optional        uart
 arm/at91/uart_cpu_at91rm9200usart.c    optional        uart
 arm/at91/uart_dev_at91usart.c  optional        uart
+dev/usb/controller/ohci_atmelarm.c     optional        ohci
 #
 # All the boards we support
 #

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (FreeBSD)

iD4DBQFKXG2sxJBWvpalMpkRAsPLAKCg3LhSgdFav00A0VkAO28rYNGuEQCXSq7F
j3wjjctKON3dmnORo023+w==
=1zRQ
-----END PGP SIGNATURE-----
From hselasky at c2i.net  Tue Jul 14 11:49:20 2009
From: hselasky at c2i.net (Hans Petter Selasky)
Date: Tue Jul 14 11:49:26 2009
Subject: ohci not detected on at91rm9200?
In-Reply-To: <4A5C6DAC.6030103@bulinfo.net>
References: <4A5C3BF5.9030200@bulinfo.net>
	<200907141030.57784.hselasky@c2i.net>
	<4A5C6DAC.6030103@bulinfo.net>
Message-ID: <200907141349.00740.hselasky@c2i.net>

On Tuesday 14 July 2009 13:36:12 Krassimir Slavchev wrote:
> Hans Petter Selasky wrote:
> > On Tuesday 14 July 2009 10:04:05 Krassimir Slavchev wrote:
> >> Hi,
> >>
> >> The ohci controller is not detected by FreeBSD 8.0-BETA1 r195653.
> >> Last working kernel I have is r187522 (~ Jan 2009) with old USB stack.
> >> I have only removed the 'ugen' device from my kernel config file.
> >>
> >> Can someone confirm this? Or may be I miss something?
> >
> > Does your kernel contain: "device ohci". Also diff the example config
> > files for AT91RM9200 with yours under /sys/arm/conf .
> >
> > --HPS
>
> This fixes the problem:
>
> --- files.at91  (revision 195653)
> +++ files.at91  (working copy)
> @@ -18,6 +18,7 @@
>  arm/at91/uart_bus_at91usart.c  optional        uart
>  arm/at91/uart_cpu_at91rm9200usart.c    optional        uart
>  arm/at91/uart_dev_at91usart.c  optional        uart
> +dev/usb/controller/ohci_atmelarm.c     optional        ohci
>  #
>  # All the boards we support
>  #

Hi,

%grep -d recurse ohci_atmelarm.c .
./files:dev/usb/controller/ohci_atmelarm.c      optional ohci at91rm9200

The option is already there, but you need to add:

device at91rm9200

To your kernel config file.

Maybe you also want to test the device port:

./files:dev/usb/controller/at91dci_atmelarm.c   optional at91dci at91rm9200

Requires usb_template module and ustorage module.

BTW: Does the OHCI part work like expected?

--HPS
From krassi at bulinfo.net  Tue Jul 14 14:14:56 2009
From: krassi at bulinfo.net (Krassimir Slavchev)
Date: Tue Jul 14 14:15:01 2009
Subject: ohci not detected on at91rm9200?
In-Reply-To: <200907141349.00740.hselasky@c2i.net>
References: <4A5C3BF5.9030200@bulinfo.net>
	<200907141030.57784.hselasky@c2i.net>
	<4A5C6DAC.6030103@bulinfo.net>
	<200907141349.00740.hselasky@c2i.net>
Message-ID: <4A5C92DB.7030701@bulinfo.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Petter Selasky wrote:
> On Tuesday 14 July 2009 13:36:12 Krassimir Slavchev wrote:
>> Hans Petter Selasky wrote:
>>> On Tuesday 14 July 2009 10:04:05 Krassimir Slavchev wrote:
>>>> Hi,
>>>>
>>>> The ohci controller is not detected by FreeBSD 8.0-BETA1 r195653.
>>>> Last working kernel I have is r187522 (~ Jan 2009) with old USB stack.
>>>> I have only removed the 'ugen' device from my kernel config file.
>>>>
>>>> Can someone confirm this? Or may be I miss something?
>>> Does your kernel contain: "device ohci". Also diff the example config
>>> files for AT91RM9200 with yours under /sys/arm/conf .
>>>
>>> --HPS
>> This fixes the problem:
>>
>> --- files.at91  (revision 195653)
>> +++ files.at91  (working copy)
>> @@ -18,6 +18,7 @@
>>  arm/at91/uart_bus_at91usart.c  optional        uart
>>  arm/at91/uart_cpu_at91rm9200usart.c    optional        uart
>>  arm/at91/uart_dev_at91usart.c  optional        uart
>> +dev/usb/controller/ohci_atmelarm.c     optional        ohci
>>  #
>>  # All the boards we support
>>  #
> 
> Hi,
> 
> %grep -d recurse ohci_atmelarm.c .
> ./files:dev/usb/controller/ohci_atmelarm.c      optional ohci at91rm9200
> 
> The option is already there, but you need to add:
> 
> device at91rm9200
> 
> To your kernel config file.

Ahh, where this is documented? This device is missing in other
at91rm9200 kernel config files in sys/arm/conf .

> 
> Maybe you also want to test the device port:
> 
> ./files:dev/usb/controller/at91dci_atmelarm.c   optional at91dci at91rm9200
> 
> Requires usb_template module and ustorage module.

You mean:
device	at91dci
device	usb_template
device	usfs

ugen0.2: <FreeBSD foundation> at usbus0
ugen0.2: <FreeBSD foundation> at usbus0 (disconnected)
ugen0.2: <FreeBSD foundation> at usbus0
ustorage_fs0: <USB Mass Storage Interface> on usbus0

umass0: FreeBSD foundation USB Memory Stick, rev 1.10/1.00, addr 5
da0 at umass-sim0 bus 0 target 0 lun 0
da0: <FreeBSD File-Stor Gadget 0101> Removable Direct Access SCSI-2 device
da0: 1.000MB/s transfers
da0: 4MB (8192 512 byte sectors: 64H 32S/T 4C)

Reading produces only zeros but I am not sure whether device port is
initialized or not, most probably not.

> 
> BTW: Does the OHCI part work like expected?

At least the most memory sticks I have work. Only one does not work:
ugen1.2: <Corsair> at usbus1
umass0: <Corsair Flash Voyager, class 0/0, rev 2.00/11.00, addr 2> on usbus1
umass0:  SCSI over Bulk-Only; quirks = 0x0000
umass0:0:0:-1: Attached to scbus0

This memory stick works on i386 6.x and 7.x
umass0: Corsair Flash Voyager, rev 2.00/11.00, addr 2
da0 at umass-sim0 bus 0 target 0 lun 0
da0: <Corsair Flash Voyager 1100> Removable Direct Access SCSI-0 device
da0: 40.000MB/s transfers
da0: 967MB (1981440 512 byte sectors: 64H 32S/T 967C)


Great work! Thanks.

Best Regards

> 
> --HPS
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (FreeBSD)

iD8DBQFKXJLbxJBWvpalMpkRAhh3AJ9lEaeWi1oP5iascpNy754zBmpAFACfTHeP
zHgTf6+R0seIZfMjh9qGllQ=
=+/2x
-----END PGP SIGNATURE-----
From imp at bsdimp.com  Tue Jul 14 15:04:05 2009
From: imp at bsdimp.com (M. Warner Losh)
Date: Tue Jul 14 15:04:12 2009
Subject: ohci not detected on at91rm9200?
In-Reply-To: <200907141349.00740.hselasky@c2i.net>
References: <200907141030.57784.hselasky@c2i.net>
	<4A5C6DAC.6030103@bulinfo.net>
	<200907141349.00740.hselasky@c2i.net>
Message-ID: <20090714.090023.709401729.imp@bsdimp.com>

In message: <200907141349.00740.hselasky@c2i.net>
            Hans Petter Selasky <hselasky@c2i.net> writes:
: On Tuesday 14 July 2009 13:36:12 Krassimir Slavchev wrote:
: > Hans Petter Selasky wrote:
: > > On Tuesday 14 July 2009 10:04:05 Krassimir Slavchev wrote:
: > >> Hi,
: > >>
: > >> The ohci controller is not detected by FreeBSD 8.0-BETA1 r195653.
: > >> Last working kernel I have is r187522 (~ Jan 2009) with old USB stack.
: > >> I have only removed the 'ugen' device from my kernel config file.
: > >>
: > >> Can someone confirm this? Or may be I miss something?
: > >
: > > Does your kernel contain: "device ohci". Also diff the example config
: > > files for AT91RM9200 with yours under /sys/arm/conf .
: > >
: > > --HPS
: >
: > This fixes the problem:
: >
: > --- files.at91  (revision 195653)
: > +++ files.at91  (working copy)
: > @@ -18,6 +18,7 @@
: >  arm/at91/uart_bus_at91usart.c  optional        uart
: >  arm/at91/uart_cpu_at91rm9200usart.c    optional        uart
: >  arm/at91/uart_dev_at91usart.c  optional        uart
: > +dev/usb/controller/ohci_atmelarm.c     optional        ohci
: >  #
: >  # All the boards we support
: >  #

This patch looks good.

: %grep -d recurse ohci_atmelarm.c .
: ./files:dev/usb/controller/ohci_atmelarm.c      optional ohci at91rm9200
:
: The option is already there, but you need to add:
: 
: device at91rm9200

This is lame.  Totally lame.  For a variety of reasons.

I'll go fix this now.

: To your kernel config file.
: 
: Maybe you also want to test the device port:
: 
: ./files:dev/usb/controller/at91dci_atmelarm.c   optional at91dci at91rm9200
: 
: Requires usb_template module and ustorage module.

dev/usb/controller/at91dci_atmelarm.c   optional at91dci at91rm9200
dev/usb/controller/musb_otg_atmelarm.c  optional musb at91rm9200
dev/usb/controller/uss820dci_atmelarm.c optional uss820dci at91rm9200

How many of these devices are actually present in at91rm9200 systems?

: BTW: Does the OHCI part work like expected?
: 
: --HPS
: _______________________________________________
: freebsd-arm@freebsd.org mailing list
: http://lists.freebsd.org/mailman/listinfo/freebsd-arm
: To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org"
: 
: 
From imp at bsdimp.com  Tue Jul 14 15:05:35 2009
From: imp at bsdimp.com (M. Warner Losh)
Date: Tue Jul 14 15:05:42 2009
Subject: ohci not detected on at91rm9200?
In-Reply-To: <4A5C92DB.7030701@bulinfo.net>
References: <4A5C6DAC.6030103@bulinfo.net>
	<200907141349.00740.hselasky@c2i.net>
	<4A5C92DB.7030701@bulinfo.net>
Message-ID: <20090714.090451.-1264101575.imp@bsdimp.com>

In message: <4A5C92DB.7030701@bulinfo.net>
            Krassimir Slavchev <krassi@bulinfo.net> writes:
: -----BEGIN PGP SIGNED MESSAGE-----
: Hash: SHA1
: 
: Hans Petter Selasky wrote:
: > On Tuesday 14 July 2009 13:36:12 Krassimir Slavchev wrote:
: >> Hans Petter Selasky wrote:
: >>> On Tuesday 14 July 2009 10:04:05 Krassimir Slavchev wrote:
: >>>> Hi,
: >>>>
: >>>> The ohci controller is not detected by FreeBSD 8.0-BETA1 r195653.
: >>>> Last working kernel I have is r187522 (~ Jan 2009) with old USB stack.
: >>>> I have only removed the 'ugen' device from my kernel config file.
: >>>>
: >>>> Can someone confirm this? Or may be I miss something?
: >>> Does your kernel contain: "device ohci". Also diff the example config
: >>> files for AT91RM9200 with yours under /sys/arm/conf .
: >>>
: >>> --HPS
: >> This fixes the problem:
: >>
: >> --- files.at91  (revision 195653)
: >> +++ files.at91  (working copy)
: >> @@ -18,6 +18,7 @@
: >>  arm/at91/uart_bus_at91usart.c  optional        uart
: >>  arm/at91/uart_cpu_at91rm9200usart.c    optional        uart
: >>  arm/at91/uart_dev_at91usart.c  optional        uart
: >> +dev/usb/controller/ohci_atmelarm.c     optional        ohci
: >>  #
: >>  # All the boards we support
: >>  #
: > 
: > Hi,
: > 
: > %grep -d recurse ohci_atmelarm.c .
: > ./files:dev/usb/controller/ohci_atmelarm.c      optional ohci at91rm9200
: > 
: > The option is already there, but you need to add:
: > 
: > device at91rm9200
: > 
: > To your kernel config file.
: 
: Ahh, where this is documented? This device is missing in other
: at91rm9200 kernel config files in sys/arm/conf .

It is bogus.

: > 
: > Maybe you also want to test the device port:
: > 
: > ./files:dev/usb/controller/at91dci_atmelarm.c   optional at91dci at91rm9200
: > 
: > Requires usb_template module and ustorage module.
: 
: You mean:
: device	at91dci
: device	usb_template
: device	usfs
: 
: ugen0.2: <FreeBSD foundation> at usbus0
: ugen0.2: <FreeBSD foundation> at usbus0 (disconnected)
: ugen0.2: <FreeBSD foundation> at usbus0
: ustorage_fs0: <USB Mass Storage Interface> on usbus0
: 
: umass0: FreeBSD foundation USB Memory Stick, rev 1.10/1.00, addr 5
: da0 at umass-sim0 bus 0 target 0 lun 0
: da0: <FreeBSD File-Stor Gadget 0101> Removable Direct Access SCSI-2 device
: da0: 1.000MB/s transfers
: da0: 4MB (8192 512 byte sectors: 64H 32S/T 4C)
: 
: Reading produces only zeros but I am not sure whether device port is
: initialized or not, most probably not.
: 
: > 
: > BTW: Does the OHCI part work like expected?
: 
: At least the most memory sticks I have work. Only one does not work:
: ugen1.2: <Corsair> at usbus1
: umass0: <Corsair Flash Voyager, class 0/0, rev 2.00/11.00, addr 2> on usbus1
: umass0:  SCSI over Bulk-Only; quirks = 0x0000
: umass0:0:0:-1: Attached to scbus0
: 
: This memory stick works on i386 6.x and 7.x
: umass0: Corsair Flash Voyager, rev 2.00/11.00, addr 2
: da0 at umass-sim0 bus 0 target 0 lun 0
: da0: <Corsair Flash Voyager 1100> Removable Direct Access SCSI-0 device
: da0: 40.000MB/s transfers
: da0: 967MB (1981440 512 byte sectors: 64H 32S/T 967C)

Please try the patch enclosed.

Warner
-------------- next part --------------
Index: conf/files
===================================================================
--- conf/files	(revision 195560)
+++ conf/files	(working copy)
@@ -1565,19 +1565,14 @@
 #
 # USB controller drivers
 #
-dev/usb/controller/at91dci.c		optional at91dci
-dev/usb/controller/at91dci_atmelarm.c	optional at91dci at91rm9200
 dev/usb/controller/musb_otg.c		optional musb
-dev/usb/controller/musb_otg_atmelarm.c	optional musb at91rm9200
 dev/usb/controller/ehci.c		optional ehci
 dev/usb/controller/ehci_pci.c		optional ehci pci
 dev/usb/controller/ohci.c		optional ohci
-dev/usb/controller/ohci_atmelarm.c	optional ohci at91rm9200
 dev/usb/controller/ohci_pci.c		optional ohci pci
 dev/usb/controller/uhci.c		optional uhci
 dev/usb/controller/uhci_pci.c		optional uhci pci
 dev/usb/controller/uss820dci.c		optional uss820dci
-dev/usb/controller/uss820dci_atmelarm.c	optional uss820dci at91rm9200
 dev/usb/controller/usb_controller.c	optional usb
 #
 # USB storage drivers
Index: arm/at91/files.at91
===================================================================
--- arm/at91/files.at91	(revision 195560)
+++ arm/at91/files.at91	(working copy)
@@ -25,3 +25,12 @@
 arm/at91/board_hl200.c		optional	at91_board_hl200
 arm/at91/board_kb920x.c		optional	at91_board_kb920x
 arm/at91/board_tsc4370.c	optional	at91_board_tsc4370
+#
+# new usb has the usb support file in dev/usb rather than under
+# at91.
+#
+dev/usb/controller/at91dci.c		optional at91dci
+dev/usb/controller/at91dci_atmelarm.c	optional at91dci
+dev/usb/controller/musb_otg_atmelarm.c	optional musb
+dev/usb/controller/ohci_atmelarm.c	optional ohci
+dev/usb/controller/uss820dci_atmelarm.c	optional uss820dci
From hselasky at c2i.net  Tue Jul 14 15:17:00 2009
From: hselasky at c2i.net (Hans Petter Selasky)
Date: Tue Jul 14 15:17:06 2009
Subject: ohci not detected on at91rm9200?
In-Reply-To: <20090714.090023.709401729.imp@bsdimp.com>
References: <200907141030.57784.hselasky@c2i.net>
	<200907141349.00740.hselasky@c2i.net>
	<20090714.090023.709401729.imp@bsdimp.com>
Message-ID: <200907141716.41209.hselasky@c2i.net>

On Tuesday 14 July 2009 17:00:23 M. Warner Losh wrote:
> How many of these devices are actually present in at91rm9200 systems?

Only the AT91UDP is built into the CPU. The others are only present in 
customisations.

--HPS
From hselasky at c2i.net  Tue Jul 14 15:21:29 2009
From: hselasky at c2i.net (Hans Petter Selasky)
Date: Tue Jul 14 15:21:36 2009
Subject: ohci not detected on at91rm9200?
In-Reply-To: <4A5C92DB.7030701@bulinfo.net>
References: <4A5C3BF5.9030200@bulinfo.net>
	<200907141349.00740.hselasky@c2i.net>
	<4A5C92DB.7030701@bulinfo.net>
Message-ID: <200907141721.10364.hselasky@c2i.net>

On Tuesday 14 July 2009 16:14:51 Krassimir Slavchev wrote:
> Hans Petter Selasky wrote:
> > On Tuesday 14 July 2009 13:36:12 Krassimir Slavchev wrote:
> >> Hans Petter Selasky wrote:
> >>> On Tuesday 14 July 2009 10:04:05 Krassimir Slavchev wrote:
> >>>> Hi,
> >
> > Hi,
> >
> > %grep -d recurse ohci_atmelarm.c .
> > ./files:dev/usb/controller/ohci_atmelarm.c      optional ohci at91rm9200
> >
> > The option is already there, but you need to add:
> >
> > device at91rm9200
> >
> > To your kernel config file.
>
> Ahh, where this is documented? This device is missing in other
> at91rm9200 kernel config files in sys/arm/conf .

See Warner's patch. It's not documented, more of a hack.

>
> > Maybe you also want to test the device port:
> >
> > ./files:dev/usb/controller/at91dci_atmelarm.c   optional at91dci
> > at91rm9200
> >
> > Requires usb_template module and ustorage module.
>
> You mean:
> device	at91dci
> device	usb_template
> device	usfs
>
> ugen0.2: <FreeBSD foundation> at usbus0
> ugen0.2: <FreeBSD foundation> at usbus0 (disconnected)
> ugen0.2: <FreeBSD foundation> at usbus0
> ustorage_fs0: <USB Mass Storage Interface> on usbus0
>
> umass0: FreeBSD foundation USB Memory Stick, rev 1.10/1.00, addr 5
> da0 at umass-sim0 bus 0 target 0 lun 0
> da0: <FreeBSD File-Stor Gadget 0101> Removable Direct Access SCSI-2 device
> da0: 1.000MB/s transfers
> da0: 4MB (8192 512 byte sectors: 64H 32S/T 4C)
>
> Reading produces only zeros but I am not sure whether device port is
> initialized or not, most probably not.
>

That's like expected. You need to format the image, hence it is only a RAM 
disk of 4MBytes.

It there is also an ethernet DCI profile.

See:

sysctl hw.usb.template=1

And you need:

device if_cdce

In the kernel config.

--HPS

From raj at semihalf.com  Tue Jul 14 17:20:25 2009
From: raj at semihalf.com (Rafal Jaworowski)
Date: Tue Jul 14 17:20:36 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <E4BEDDE0-2872-4934-9A53-4121E33390B5@mac.com>
References: <200906231035.43096.kosmo@semihalf.com>
	<200907091834.42462.hselasky@c2i.net>
	<200907141031.11185.kosmo@semihalf.com>
	<200907141036.44652.hselasky@c2i.net>
	<FB60465B-D714-4534-B07F-890C72A9CB33@semihalf.com>
	<E4BEDDE0-2872-4934-9A53-4121E33390B5@mac.com>
Message-ID: <B8FCBB1F-F294-4252-82AE-1466631E548F@semihalf.com>


On 2009-07-14, at 18:58, Marcel Moolenaar wrote:

> On Jul 14, 2009, at 2:21 AM, Rafal Jaworowski wrote:
>
>>
>> On 2009-07-14, at 10:36, Hans Petter Selasky wrote:
>>
>>> On Tuesday 14 July 2009 10:31:10 Piotr Zi?cik wrote:
>>>>> 1) My analysis: Only the data areas are being flushed/ 
>>>>> invalidated. No
>>>>> transfer descriptors are flushed/invalidated. I see no cache  
>>>>> operations
>>>>> happening on any DMA control structures, even though there are  
>>>>> calls from
>>>>> EHCI to xxx_pc_flush() and xxx_pc_invalidate().
>>>>
>>>
>>>> Probaby you see more on your AT91 device as you know USB stack  
>>>> internals.
>>>> Have you tried to bring up OHCI on you ARM board ?
>>>
>>> Not yet. I'm terribly busy with some LibUSB stuff headed for the 8- 
>>> current
>>> release. As soon as I find time I will fire off a build and debug.
>>
>> Please note these problems should be considered as a showstopper  
>> for the release since USB is currently broken on at least three ARM  
>> platforms in the tree (Marvell).
>
> Rafal,
>
> Anything I can do to help?
> (as a reminder: I have an Orion board)

Your input on the desired final solution to these issues would be  
appreciated. Please see the beginning of this thread (originated on  
usb@ ML), where a quick fix was proposed by Piotr. That patch works  
around the problems for us (also for some MIPS and noncoherent PPC),  
but the problem here is that we believe that using bus dma for the  
purpose of cache synchronization is improper in principle, and cpu- 
specific calls should be used instead, although Hans is rather  
reluctant to embracing the latter path.

>>> BTW: Has pmap been fixed for ARM in 8-current?
>>
>> Seems like the most critical problems (panics) are resolved and  
>> will be pushed into SVN shortly. In case you'd like to apply the  
>> fix directly, see: http://people.freebsd.org/~raj/patches/arm/pmap-fixes.diff
>
> Good! I was about to start a discussion about reverting
> rev. 194459 for now. We're about to start BETA-2 and it
> helps (at least Juniper :-) to have 8.0-RELEASE not be
> DOA :-)

No worries, we're also looking forward to 8.0 as a great ARM  
release :-) I will be following up on that pmap-related stuff, but am  
waiting for confirmation from stas@ there are no regressions on AT91  
with this patch.

Rafal

From xcllnt at mac.com  Tue Jul 14 17:59:10 2009
From: xcllnt at mac.com (Marcel Moolenaar)
Date: Tue Jul 14 17:59:20 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <FB60465B-D714-4534-B07F-890C72A9CB33@semihalf.com>
References: <200906231035.43096.kosmo@semihalf.com>
	<200907091834.42462.hselasky@c2i.net>
	<200907141031.11185.kosmo@semihalf.com>
	<200907141036.44652.hselasky@c2i.net>
	<FB60465B-D714-4534-B07F-890C72A9CB33@semihalf.com>
Message-ID: <E4BEDDE0-2872-4934-9A53-4121E33390B5@mac.com>


On Jul 14, 2009, at 2:21 AM, Rafal Jaworowski wrote:

>
> On 2009-07-14, at 10:36, Hans Petter Selasky wrote:
>
>> On Tuesday 14 July 2009 10:31:10 Piotr Zi?cik wrote:
>>>> 1) My analysis: Only the data areas are being flushed/ 
>>>> invalidated. No
>>>> transfer descriptors are flushed/invalidated. I see no cache  
>>>> operations
>>>> happening on any DMA control structures, even though there are  
>>>> calls from
>>>> EHCI to xxx_pc_flush() and xxx_pc_invalidate().
>>>
>>
>>> Probaby you see more on your AT91 device as you know USB stack  
>>> internals.
>>> Have you tried to bring up OHCI on you ARM board ?
>>
>> Not yet. I'm terribly busy with some LibUSB stuff headed for the 8- 
>> current
>> release. As soon as I find time I will fire off a build and debug.
>
> Please note these problems should be considered as a showstopper for  
> the release since USB is currently broken on at least three ARM  
> platforms in the tree (Marvell).

Rafal,

Anything I can do to help?
(as a reminder: I have an Orion board)

>
>> BTW: Has pmap been fixed for ARM in 8-current?
>
> Seems like the most critical problems (panics) are resolved and will  
> be pushed into SVN shortly. In case you'd like to apply the fix  
> directly, see: http://people.freebsd.org/~raj/patches/arm/pmap-fixes.diff

Good! I was about to start a discussion about reverting
rev. 194459 for now. We're about to start BETA-2 and it
helps (at least Juniper :-) to have 8.0-RELEASE not be
DOA :-)

-- 
Marcel Moolenaar
xcllnt@mac.com



From krassi at bulinfo.net  Wed Jul 15 11:48:36 2009
From: krassi at bulinfo.net (Krassimir Slavchev)
Date: Wed Jul 15 11:48:43 2009
Subject: ohci not detected on at91rm9200?
In-Reply-To: <200907141721.10364.hselasky@c2i.net>
References: <4A5C3BF5.9030200@bulinfo.net>
	<200907141349.00740.hselasky@c2i.net>
	<4A5C92DB.7030701@bulinfo.net>
	<200907141721.10364.hselasky@c2i.net>
Message-ID: <4A5DC210.5000806@bulinfo.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Petter Selasky wrote:

>>
>> ugen0.2: <FreeBSD foundation> at usbus0
>> ugen0.2: <FreeBSD foundation> at usbus0 (disconnected)
>> ugen0.2: <FreeBSD foundation> at usbus0
>> ustorage_fs0: <USB Mass Storage Interface> on usbus0
>>
>> umass0: FreeBSD foundation USB Memory Stick, rev 1.10/1.00, addr 5
>> da0 at umass-sim0 bus 0 target 0 lun 0
>> da0: <FreeBSD File-Stor Gadget 0101> Removable Direct Access SCSI-2 device
>> da0: 1.000MB/s transfers
>> da0: 4MB (8192 512 byte sectors: 64H 32S/T 4C)
>>
>> Reading produces only zeros but I am not sure whether device port is
>> initialized or not, most probably not.
>>
> 
> That's like expected. You need to format the image, hence it is only a RAM 
> disk of 4MBytes.

It will be more useful to share the SPI flash for example.

> 
> It there is also an ethernet DCI profile.
> 
> See:
> 
> sysctl hw.usb.template=1
> 
> And you need:
> 
> device if_cdce
> 
> In the kernel config.

It is detected but after configuring the interface I cannot send/receive
packets.

cdce0: <FreeBSD foundation USB Ethernet Adapter, class 2/0, rev
1.10/1.00, addr 2> on uhub4
cdce0: faking MAC address
cdce0: WARNING: using obsoleted IFF_NEEDSGIANT flag
cdce0: Ethernet address: 2a:29:df:24:01:00
cdce0: usb error on tx: TIMEOUT
cdce0: usb error on rx: IOERROR

However, the idea is very good!

> 
> --HPS
> 
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (FreeBSD)

iD8DBQFKXcIQxJBWvpalMpkRAj4XAKCzPTo9tqW8UYj85DNgdfl2owYmNgCgl4ps
gVhlXyyuy2+waFtqJ9M2N0w=
=Yef7
-----END PGP SIGNATURE-----
From hselasky at c2i.net  Wed Jul 15 11:53:44 2009
From: hselasky at c2i.net (Hans Petter Selasky)
Date: Wed Jul 15 11:53:51 2009
Subject: ohci not detected on at91rm9200?
In-Reply-To: <4A5DC210.5000806@bulinfo.net>
References: <4A5C3BF5.9030200@bulinfo.net>
	<200907141721.10364.hselasky@c2i.net>
	<4A5DC210.5000806@bulinfo.net>
Message-ID: <200907151353.21403.hselasky@c2i.net>

Hi,

On Wednesday 15 July 2009 13:48:32 Krassimir Slavchev wrote:
> Hans Petter Selasky wrote:

>
> It will be more useful to share the SPI flash for example.

Yes. Please provide patches if you want additional functionality.

>
> > It there is also an ethernet DCI profile.
> >
> > See:
> >
> > sysctl hw.usb.template=1
> >
> > And you need:
> >
> > device if_cdce
> >
> > In the kernel config.
>
> It is detected but after configuring the interface I cannot send/receive
> packets.
>
> cdce0: <FreeBSD foundation USB Ethernet Adapter, class 2/0, rev
> 1.10/1.00, addr 2> on uhub4
> cdce0: faking MAC address
> cdce0: WARNING: using obsoleted IFF_NEEDSGIANT flag
> cdce0: Ethernet address: 2a:29:df:24:01:00
> cdce0: usb error on tx: TIMEOUT
> cdce0: usb error on rx: IOERROR
>

You need to "ifconfig up" the interface on both sides before any traffic will 
pass.

--HPS
From krassi at bulinfo.net  Wed Jul 15 13:23:18 2009
From: krassi at bulinfo.net (Krassimir Slavchev)
Date: Wed Jul 15 13:23:26 2009
Subject: ohci not detected on at91rm9200?
In-Reply-To: <200907151353.21403.hselasky@c2i.net>
References: <4A5C3BF5.9030200@bulinfo.net>
	<200907141721.10364.hselasky@c2i.net>
	<4A5DC210.5000806@bulinfo.net>
	<200907151353.21403.hselasky@c2i.net>
Message-ID: <4A5DD842.3080007@bulinfo.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Petter Selasky wrote:
>> It is detected but after configuring the interface I cannot send/receive
>> packets.
>>
>> cdce0: <FreeBSD foundation USB Ethernet Adapter, class 2/0, rev
>> 1.10/1.00, addr 2> on uhub4
>> cdce0: faking MAC address
>> cdce0: WARNING: using obsoleted IFF_NEEDSGIANT flag
>> cdce0: Ethernet address: 2a:29:df:24:01:00
>> cdce0: usb error on tx: TIMEOUT
>> cdce0: usb error on rx: IOERROR
>>
> 
> You need to "ifconfig up" the interface on both sides before any traffic will 
> pass.

Of course they are:

cdce0:
flags=108c43<UP,BROADCAST,RUNNING,OACTIVE,SIMPLEX,MULTICAST,NEEDSGIANT>
metric 0 mtu 1500
        ether 2a:00:00:00:00:00
        inet 192.168.2.121 netmask 0xffffff00 broadcast 192.168.2.255
        media: Ethernet 10baseT/UTP
        status: active

ate0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 43:00:00:01:ff:00
        inet 192.168.2.189 netmask 0xffffff00 broadcast 192.168.2.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (FreeBSD)

iD8DBQFKXdhCxJBWvpalMpkRAvX8AJ9qOCXKNwA3pIx/+F7GkmNNbw6MlgCgk+Rx
m7vmhz+1WVYZnpSOLhrO1KU=
=sXH6
-----END PGP SIGNATURE-----
From hselasky at c2i.net  Wed Jul 15 13:32:24 2009
From: hselasky at c2i.net (Hans Petter Selasky)
Date: Wed Jul 15 13:32:30 2009
Subject: ohci not detected on at91rm9200?
In-Reply-To: <4A5DD842.3080007@bulinfo.net>
References: <4A5C3BF5.9030200@bulinfo.net>
	<200907151353.21403.hselasky@c2i.net>
	<4A5DD842.3080007@bulinfo.net>
Message-ID: <200907151532.03875.hselasky@c2i.net>

On Wednesday 15 July 2009 15:23:14 Krassimir Slavchev wrote:
> Hans Petter Selasky wrote:
> >> It is detected but after configuring the interface I cannot send/receive

> >
> > You need to "ifconfig up" the interface on both sides before any traffic
> > will pass.
>
> Of course they are:
>
> cdce0:
> flags=108c43<UP,BROADCAST,RUNNING,OACTIVE,SIMPLEX,MULTICAST,NEEDSGIANT>
> metric 0 mtu 1500
>         ether 2a:00:00:00:00:00
>         inet 192.168.2.121 netmask 0xffffff00 broadcast 192.168.2.255
>         media: Ethernet 10baseT/UTP
>         status: active
>
> ate0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         options=8<VLAN_MTU>
>         ether 43:00:00:01:ff:00
>         inet 192.168.2.189 netmask 0xffffff00 broadcast 192.168.2.255
>         media: Ethernet autoselect (100baseTX <full-duplex>)
>         status: active

You should have cdce devices on both sides, PC and Host. Looks like there will 
be a network conflict if you use 192.168.2.xx for both network devices when  
they are not on the same cable.

?

--HPS

From hpcharles at gmail.com  Wed Jul 15 15:15:14 2009
From: hpcharles at gmail.com (Henri-Pierre Charles)
Date: Wed Jul 15 15:15:20 2009
Subject: qemu / boot parameter
Message-ID: <4734a3ed0907150752g595b680frcacba26da7979479@mail.gmail.com>

Hello list, I try to do my first FreeBSD/arm installation. I plan to
use qemu. I've started with this page
http://people.freebsd.org/~cognet/freebsd_arm.txt for kernel/world
build. For my personnal teaching, I've summarized these instructions
in the attached script.

Now, I'm stuck with qemu. I've tried many invocation, and the best
result I get is with qemu-system-arm hda.img -boot n 		\
    -tftp $ARMFS/boot/kernel/ -bootp /kernel 	\
    -kernel $ARMFS/boot/kernel/kernel 		\
    -m 300 					\
    -M integratorc

And I get :
qemu: fatal: Trying to execute code outside RAM or ROM at 0xc0200100
(which is not far from KERNVIRTADDR)

I miss something somewhere ? Any idea ?

HPC


-- 
HPC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: BuildArmImage
Type: application/octet-stream
Size: 1764 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-arm/attachments/20090715/9528e71c/BuildArmImage.obj
From krassi at bulinfo.net  Wed Jul 15 15:41:18 2009
From: krassi at bulinfo.net (Krassimir Slavchev)
Date: Wed Jul 15 15:41:25 2009
Subject: ohci not detected on at91rm9200?
In-Reply-To: <200907151532.03875.hselasky@c2i.net>
References: <4A5C3BF5.9030200@bulinfo.net>
	<200907151353.21403.hselasky@c2i.net>
	<4A5DD842.3080007@bulinfo.net>
	<200907151532.03875.hselasky@c2i.net>
Message-ID: <4A5DF89A.1070004@bulinfo.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Petter Selasky wrote:
> On Wednesday 15 July 2009 15:23:14 Krassimir Slavchev wrote:
>> Hans Petter Selasky wrote:
>>>> It is detected but after configuring the interface I cannot send/receive
> 
>>> You need to "ifconfig up" the interface on both sides before any traffic
>>> will pass.
>> Of course they are:
>>
>> cdce0:
>> flags=108c43<UP,BROADCAST,RUNNING,OACTIVE,SIMPLEX,MULTICAST,NEEDSGIANT>
>> metric 0 mtu 1500
>>         ether 2a:00:00:00:00:00
>>         inet 192.168.2.121 netmask 0xffffff00 broadcast 192.168.2.255
>>         media: Ethernet 10baseT/UTP
>>         status: active
>>
>> ate0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>         options=8<VLAN_MTU>
>>         ether 43:00:00:01:ff:00
>>         inet 192.168.2.189 netmask 0xffffff00 broadcast 192.168.2.255
>>         media: Ethernet autoselect (100baseTX <full-duplex>)
>>         status: active
> 
> You should have cdce devices on both sides, PC and Host. Looks like there will 
> be a network conflict if you use 192.168.2.xx for both network devices when  
> they are not on the same cable.
> 
> ?

Yes, I realize that.

After adding cdce on arm board:
cdce0: <USB Ethernet Comm interface> on usbus0
ue0: <USB Ethernet> on cdce0
ue0: Ethernet address: 2a:23:45:67:89:54

ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 2a:23:45:67:89:54


But on the other side running 7.2:
cdce0: <FreeBSD foundation USB Ethernet Adapter, class 2/0, rev
1.10/1.00, addr 2> on uhub4
cdce0: setting alternate interface failed
device_attach: cdce0 attach returned 6


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (FreeBSD)

iD8DBQFKXfiaxJBWvpalMpkRAi6fAJwMzhfBcwNouW4Y7iPFFHl9PjiUTQCcCSzB
hlRr1q6cBUpGcGXjgm9/wh8=
=bXm9
-----END PGP SIGNATURE-----
From hselasky at c2i.net  Thu Jul 16 09:33:54 2009
From: hselasky at c2i.net (Hans Petter Selasky)
Date: Thu Jul 16 09:34:07 2009
Subject: ohci not detected on at91rm9200?
In-Reply-To: <4A5DF89A.1070004@bulinfo.net>
References: <4A5C3BF5.9030200@bulinfo.net>
	<200907151532.03875.hselasky@c2i.net>
	<4A5DF89A.1070004@bulinfo.net>
Message-ID: <200907161033.34560.hselasky@c2i.net>

On Wednesday 15 July 2009 17:41:14 Krassimir Slavchev wrote:
> Hans Petter Selasky wrote:
> > On Wednesday 15 July 2009 15:23:14 Krassimir Slavchev wrote:
> >> Hans Petter Selasky wrote:
> >>>> It is detected but after configuring the interface I cannot
> >>>> send/receive
> >>>
> >>> You need to "ifconfig up" the interface on both sides before any
> >>> traffic will pass.
> >>
> >> Of course they are:
> >>
> >> cdce0:
> >> flags=108c43<UP,BROADCAST,RUNNING,OACTIVE,SIMPLEX,MULTICAST,NEEDSGIANT>
> >> metric 0 mtu 1500
> >>         ether 2a:00:00:00:00:00
> >>         inet 192.168.2.121 netmask 0xffffff00 broadcast 192.168.2.255
> >>         media: Ethernet 10baseT/UTP
> >>         status: active
> >>
> >> ate0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> >> 1500 options=8<VLAN_MTU>
> >>         ether 43:00:00:01:ff:00
> >>         inet 192.168.2.189 netmask 0xffffff00 broadcast 192.168.2.255
> >>         media: Ethernet autoselect (100baseTX <full-duplex>)
> >>         status: active
> >
> > You should have cdce devices on both sides, PC and Host. Looks like there
> > will be a network conflict if you use 192.168.2.xx for both network
> > devices when they are not on the same cable.
> >
> > ?
>
> Yes, I realize that.
>
> After adding cdce on arm board:
> cdce0: <USB Ethernet Comm interface> on usbus0
> ue0: <USB Ethernet> on cdce0
> ue0: Ethernet address: 2a:23:45:67:89:54
>
> ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         ether 2a:23:45:67:89:54
>
>
> But on the other side running 7.2:
> cdce0: <FreeBSD foundation USB Ethernet Adapter, class 2/0, rev
> 1.10/1.00, addr 2> on uhub4
> cdce0: setting alternate interface failed
> device_attach: cdce0 attach returned 6

Maybe the following patch helps on the ARM side:

http://perforce.freebsd.org/chv.cgi?CH=166156

I'm going to test this setup, and will probably resolve the issue, if I find 
any, including the cache sync stuff.

--HPS

From krassi at bulinfo.net  Thu Jul 16 10:45:19 2009
From: krassi at bulinfo.net (Krassimir Slavchev)
Date: Thu Jul 16 10:45:27 2009
Subject: ohci not detected on at91rm9200?
In-Reply-To: <200907161033.34560.hselasky@c2i.net>
References: <4A5C3BF5.9030200@bulinfo.net>
	<200907151532.03875.hselasky@c2i.net>
	<4A5DF89A.1070004@bulinfo.net>
	<200907161033.34560.hselasky@c2i.net>
Message-ID: <4A5F04B8.1060109@bulinfo.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Petter Selasky wrote:
>>
>> After adding cdce on arm board:
>> cdce0: <USB Ethernet Comm interface> on usbus0
>> ue0: <USB Ethernet> on cdce0
>> ue0: Ethernet address: 2a:23:45:67:89:54
>>
>> ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>         ether 2a:23:45:67:89:54
>>
>>
>> But on the other side running 7.2:
>> cdce0: <FreeBSD foundation USB Ethernet Adapter, class 2/0, rev
>> 1.10/1.00, addr 2> on uhub4
>> cdce0: setting alternate interface failed
>> device_attach: cdce0 attach returned 6
> 
> Maybe the following patch helps on the ARM side:
> 
> http://perforce.freebsd.org/chv.cgi?CH=166156
> 
> I'm going to test this setup, and will probably resolve the issue, if I find 
> any, including the cache sync stuff.


I have made these changes on the ARM side but the error is same.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (FreeBSD)

iD8DBQFKXwS4xJBWvpalMpkRAssSAJ43mCgijyZKFgnzR/xGquLRqgQnAgCeJsfE
OYEP1MwnCcvGrONUi2DMCJg=
=qCW4
-----END PGP SIGNATURE-----
From stas at FreeBSD.org  Thu Jul 16 13:10:33 2009
From: stas at FreeBSD.org (Stanislav Sedov)
Date: Thu Jul 16 13:10:40 2009
Subject: qemu / boot parameter
In-Reply-To: <4734a3ed0907150752g595b680frcacba26da7979479@mail.gmail.com>
References: <4734a3ed0907150752g595b680frcacba26da7979479@mail.gmail.com>
Message-ID: <20090716171030.a0e7a3f0.stas@FreeBSD.org>

On Wed, 15 Jul 2009 16:52:40 +0200
Henri-Pierre Charles <hpcharles@gmail.com> mentioned:

> Hello list, I try to do my first FreeBSD/arm installation. I plan to
> use qemu. I've started with this page
> http://people.freebsd.org/~cognet/freebsd_arm.txt for kernel/world
> build. For my personnal teaching, I've summarized these instructions
> in the attached script.
> 
> Now, I'm stuck with qemu. I've tried many invocation, and the best
> result I get is with qemu-system-arm hda.img -boot n 		\
>     -tftp $ARMFS/boot/kernel/ -bootp /kernel 	\
>     -kernel $ARMFS/boot/kernel/kernel 		\
>     -m 300 					\
>     -M integratorc
> 
> And I get :
> qemu: fatal: Trying to execute code outside RAM or ROM at 0xc0200100
> (which is not far from KERNVIRTADDR)
> 
> I miss something somewhere ? Any idea ?
> 

As far as I know we don't currently support qemu ARM/intergrator platform.
However, current versions of qemu can emulate XScale devices (gumstix,
for example), so you may want to try this type of emulation with your
XScale kernel.

Please note, however, that qemu usually emulate specific memory layouts
that may not match what FreeBSD kernel which was configured to run
on the different hardware would expect.  So a bit of hand-tweaking might
be required.  You should study where qemu maps memory and onboard devices,
and how it passes the control to FreeBSD kernel and make the appropriate
modifications to the code/kernel config.

Hope that helps.

-- 
Stanislav Sedov
ST4096-RIPE
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 801 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-arm/attachments/20090716/89087a08/attachment.pgp
From jacques.fourie at gmail.com  Thu Jul 16 14:16:31 2009
From: jacques.fourie at gmail.com (Jacques Fourie)
Date: Thu Jul 16 14:16:39 2009
Subject: qemu / boot parameter
In-Reply-To: <20090716171030.a0e7a3f0.stas@FreeBSD.org>
References: <4734a3ed0907150752g595b680frcacba26da7979479@mail.gmail.com>
	<20090716171030.a0e7a3f0.stas@FreeBSD.org>
Message-ID: <be2f52430907160643g6fe3ee97vdcb62939218a1b51@mail.gmail.com>

On Thu, Jul 16, 2009 at 3:10 PM, Stanislav Sedov<stas@freebsd.org> wrote:
> On Wed, 15 Jul 2009 16:52:40 +0200
> Henri-Pierre Charles <hpcharles@gmail.com> mentioned:
>
>> Hello list, I try to do my first FreeBSD/arm installation. I plan to
>> use qemu. I've started with this page
>> http://people.freebsd.org/~cognet/freebsd_arm.txt for kernel/world
>> build. For my personnal teaching, I've summarized these instructions
>> in the attached script.
>>
>> Now, I'm stuck with qemu. I've tried many invocation, and the best
>> result I get is with qemu-system-arm hda.img -boot n ? ? ? ? ?\
>> ? ? -tftp $ARMFS/boot/kernel/ -bootp /kernel ?\
>> ? ? -kernel $ARMFS/boot/kernel/kernel ? ? ? ? ? ? ? ? \
>> ? ? -m 300 ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?\
>> ? ? -M integratorc
>>
>> And I get :
>> qemu: fatal: Trying to execute code outside RAM or ROM at 0xc0200100
>> (which is not far from KERNVIRTADDR)
>>
>> I miss something somewhere ? Any idea ?
>>
>
> As far as I know we don't currently support qemu ARM/intergrator platform.
> However, current versions of qemu can emulate XScale devices (gumstix,
> for example), so you may want to try this type of emulation with your
> XScale kernel.
>
> Please note, however, that qemu usually emulate specific memory layouts
> that may not match what FreeBSD kernel which was configured to run
> on the different hardware would expect. ?So a bit of hand-tweaking might
> be required. ?You should study where qemu maps memory and onboard devices,
> and how it passes the control to FreeBSD kernel and make the appropriate
> modifications to the code/kernel config.
>
> Hope that helps.
>
> --
> Stanislav Sedov
> ST4096-RIPE
>

Here is a recipe that worked for me on qemu 0.9.1. As far as I know it
won't work with the qemu 0.10 but your mileage may vary.

* Apply following patch to sys/arm/xscale/pxa/uart_bus_pxa.c
  (Patch by Mark Tinguely)

static int
uart_pxa_probe(device_t dev)
{
       bus_space_handle_t      base;
       struct                  uart_softc *sc;

+#ifdef QEMU
+       base = (bus_space_handle_t)pxa_get_base(dev);
+       if (0x40100000 != (unsigned int) base)
+               return (ENXIO);
+#else
       /* Check to see if the enable bit's on. */
       if ((bus_space_read_4(obio_tag, base,
           (REG_IER << 2)) & PXA_UART_UUE) == 0)
               return (ENXIO);
+#endif

       sc = device_get_softc(dev);
       sc->sc_class = &uart_ns8250_class;

       return(uart_bus_probe(dev, 2, PXA2X0_COM_FREQ, 0, 0));
}

* Remove ARM_CACHE_LOCK_ENABLE from
  sys/arm/xscale/std.xscale and compile a GUMSTIX kernel

* Build flash image:

dd of=flash bs=1k count=16k if=/dev/zero
dd of=flash bs=1k conv=notrunc if=u-boot-connex-400-r1604.bin
dd of=flash bs=1k conv=notrunc seek=256 if=kernel.gz.tramp

I start qemu as follows:
qemu-system-arm -M connex -m 289 -pflash flash \
-nographic -serial tcp::4000,server \
-net nic,macaddr=52:54:00:12:34:50,vlan=0,model=smc91c111 \
-net tap,vlan=0,ifname=tap1,script=no

I then telnet to port 4000 for the serial console and use 'bootelf
4000' in the uboot prompt to boot the kernel.
From hselasky at c2i.net  Thu Jul 16 18:22:51 2009
From: hselasky at c2i.net (Hans Petter Selasky)
Date: Thu Jul 16 18:22:57 2009
Subject: ohci not detected on at91rm9200?
In-Reply-To: <4A5F04B8.1060109@bulinfo.net>
References: <4A5C3BF5.9030200@bulinfo.net>
	<200907161033.34560.hselasky@c2i.net>
	<4A5F04B8.1060109@bulinfo.net>
Message-ID: <200907162022.25810.hselasky@c2i.net>

On Thursday 16 July 2009 12:45:12 Krassimir Slavchev wrote:
> Hans Petter Selasky wrote:
> >> After adding cdce on arm board:
> >> cdce0: <USB Ethernet Comm interface> on usbus0
> >> ue0: <USB Ethernet> on cdce0
> >> ue0: Ethernet address: 2a:23:45:67:89:54
> >>
> >> ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> >> 1500 ether 2a:23:45:67:89:54
> >>
> >>
> >> But on the other side running 7.2:
> >> cdce0: <FreeBSD foundation USB Ethernet Adapter, class 2/0, rev
> >> 1.10/1.00, addr 2> on uhub4
> >> cdce0: setting alternate interface failed
> >> device_attach: cdce0 attach returned 6
> >
> > Maybe the following patch helps on the ARM side:
> >
> > http://perforce.freebsd.org/chv.cgi?CH=166156
> >
> > I'm going to test this setup, and will probably resolve the issue, if I
> > find any, including the cache sync stuff.
>
> I have made these changes on the ARM side but the error is same.

Hi,

You also need this fix, in addition to the previous one.

http://perforce.freebsd.org/chv.cgi?CH=166168

CDC-ethernet should work now.

--HPS
From hselasky at c2i.net  Thu Jul 16 19:56:29 2009
From: hselasky at c2i.net (Hans Petter Selasky)
Date: Thu Jul 16 19:56:37 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <B8FCBB1F-F294-4252-82AE-1466631E548F@semihalf.com>
References: <200906231035.43096.kosmo@semihalf.com>
	<E4BEDDE0-2872-4934-9A53-4121E33390B5@mac.com>
	<B8FCBB1F-F294-4252-82AE-1466631E548F@semihalf.com>
Message-ID: <200907162156.06598.hselasky@c2i.net>

On Tuesday 14 July 2009 19:20:22 Rafal Jaworowski wrote:

> >> Please note these problems should be considered as a showstopper
> >> for the release since USB is currently broken on at least three ARM
> >> platforms in the tree (Marvell).

Hi,

First of all I'm not able to boot into userland on my ARM board. Maybe I 
missed something. I'm using a custom built userland image.

Trying to mount root from ufs:/dev/md0                                                                                              
warning: no time-of-day clock registered, system time will not be set 
accurately                                                    
Jul 16 18:53:22 init: login_getclass: unknown class 'daemon'                                                                        
Fatal kernel mode data abort: 'Alignment Fault 3'                                                                                   
trapframe: 0xc622cc60                                                                                                               
FSR=00000003, FAR=6e75203a, spsr=600000d3                                                                                           
r0 =60000093, r1 =c10594a0, r2 =c10594a0, r3 =00000002                                                                              
r4 =c10acbd0, r5 =00000000, r6 =6e75203a, r7 =c10594a0                                                                              
r8 =c622cd74, r9 =00000000, r10=c0ab081c, r11=c622cccc                                                                              
r12=c622ccac, ssp=c622ccac, slr=c020a144, pc =c00c02c8                                                                              
                                                                                                                                    
[thread pid 4 tid 100008 ]                                                                                                          
Stopped at      _thread_lock_flags+0x24:        ldr     r4, [r6]                                                                    
db>                            

With the following patch in arm/cpufunc.c my UMASS device gets detected at 
boot time (KB9202B ARM board). Else I get a USB request timeout followed by a 
panic. But data transfers of more than 16KByte are likely to be broken.

struct cpu_functions arm9_cpufuncs = {
...
- /*XXX*/	arm9_dcache_wbinv_range,	/* dcache_inv_range	*/
+ /*XXX*/	arm9_dcache_inv_range,	/* dcache_inv_range	*/

};

Looking at the assembly code behind "arm9_dcache_inv_range" in 
"cpufunc_asm_arm9.S" I see that "arm9_dcache_wbinv_all" gets called if the 
area to invalidate is greater equal to 16KByte. This is wrong. When asked to 
invalidate, then no memory must be flushed in that region. Because USB works 
like this:

invalidate (TD)
read field from (TD)

Next IRQ

invalidate (TD)
read field from (TD)

If the invalidate call flushes any data, you clearly see from the pseudo code 
above that there is a chance that the last read TD field in the cache gets 
written back before the next invalidate at the next IRQ, when the invalidate 
function is implemented like a writeback+invalidate!??

Piotr and Rafal: How does ARM work? Are there two caches? One read cache and 
one write cache. Or are they the same. What happens if you read a value from 
RAM into cache, then writeback+invalidate it. Does it get written back to RAM 
or does it get discarded? Sorry, I have some holes in my ARM knowledge which 
you need to fill in.

Reference:

ENTRY(arm9_dcache_inv_range)
	ldr	ip, .Larm9_line_size
	cmp	r1, #0x4000
	bcs	.Larm9_dcache_wbinv_all

Anything unclear?

Working version of kernel with my patch having stick plugged in at boot:

ugen1.2: <Kingston> at usbus1
umass0: <Kingston DataTraveler 2.0, class 0/0, rev 2.00/1.10, addr 2> on 
usbus1
umass0:  SCSI over Bulk-Only; quirks = 0x0000
umass0:0:0:-1: Attached to scbus0
(probe0:umass-sim0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0 
(probe0:umass-sim0:0:0:0): CAM Status: SCSI Status Error
(probe0:umass-sim0:0:0:0): SCSI Status: Check Condition
(probe0:umass-sim0:0:0:0): UNIT ATTENTION asc:28,0
(probe0:umass-sim0:0:0:0): Not ready to ready change, medium may have changed
(probe0:umass-sim0:0:0:0): Retrying Command (per Sense Data)
da0 at umass-sim0 bus 0 target 0 lun 0
da0: <Kingston DataTraveler 2.0 PMAP> Removable Direct Access SCSI-0 device 
da0: 1.000MB/s transfers
da0: 3842MB (7868416 512 byte sectors: 255H 63S/T 489C)

Stock 8-current code:

ugen1.2: <Kingston> at usbus1
umass0: <Kingston DataTraveler 2.0, class 0/0, rev 2.00/1.10, addr 2> on 
usbus1
umass0:  SCSI over Bulk-Only; quirks = 0x0000
umass0:0:0:-1: Attached to scbus0
(probe0:umass-sim0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0 
(probe0:umass-sim0:0:0:0): CAM Status: SCSI Status Error
(probe0:umass-sim0:0:0:0): SCSI Status: Check Condition
(probe0:umass-sim0:0:0:0): UNIT ATTENTION asc:28,0
(probe0:umass-sim0:0:0:0): Not ready to ready change, medium may have changed
(probe0:umass-sim0:0:0:0): Retrying Command (per Sense Data)
da0 at umass-sim0 bus 0 target 0 lun 0
da0: <Kingston DataTraveler 2.0 PMAP> Removable Direct Access SCSI-0 device 
da0: 1.000MB/s transfers
da0: 3842MB (7868416 512 byte sectors: 255H 63S/T 489C)
ugen1.2: <Kingston> at usbus1 (disconnected)
umass0: at uhub1, port 1, addr 2 (disconnected)
(da0:umass-sim0:0:0:0): lost device
(da0:umass-sim0:0:0:0): removing device entry
ugen1.2: <Kingston> at usbus1
umass0: <Kingston DataTraveler 2.0, class 0/0, rev 2.00/1.10, addr 2> on 
usbus1
umass0:  SCSI over Bulk-Only; quirks = 0x0000
umass0:0:0:-1: Attached to scbus0
Sleeping thread (tid 100031, pid 19) owns a non-sleepable lock
sched_switch() at sched_switch+0x10
scp=0xc00e7830 rlv=0xc00d2228 (mi_switch+0x108)
        rsp=0xc6271ddc rfp=0xc6271e04
        r7=0xc11306f0 r6=0x00000000
        r5=0x000000c0 r4=0x00000000

...
umass0: <Kingston DataTraveler 2.0, class 0/0, rev 2.00/1.10, addr 2> on 
usbus1
umass0:  SCSI over Bulk-Only; quirks = 0x0000
umass0:0:0:-1: Attached to scbus0
(probe0:umass-sim0:0:0:0): AutoSense Failed
usb_alloc_device:1586: set address 3 failed (USB_ERR_IOERROR, ignored)
usb_alloc_device:1624: getting device descriptor at addr 3 failed, 
USB_ERR_IOERROR!
usbd_req_re_enumerate:1539: addr=3, set address failed! (USB_ERR_IOERROR, 
ignored)
usbd_req_re_enumerate:1553: getting device descriptor at addr 3 failed, 
USB_ERR_IOERROR!
usbd_req_re_enumerate:1539: addr=3, set address failed! (USB_ERR_IOERROR, 
ignored)
usbd_req_re_enumerate:1553: getting device descriptor at addr 3 failed, 
USB_ERR_IOERROR!
ugen1.3: <(null)> at usbus1 (disconnected)
uhub_reattach_port:435: could not allocate new device!
(da0:umass-sim0:0:0:0): got CAM status 0x4
(da0:umass-sim0:0:0:0): fatal error, failed to attach to device
(da0:umass-sim0:0:0:0): lost device
(da0:umass-sim0:0:0:0): removing device entry
ugen1.2: <Kingston> at usbus1 (disconnected)
umass0: at uhub1, port 1, addr 2 (disconnected)
ugen1.2: <Kingston> at usbus1
umass0: <Kingston DataTraveler 2.0, class 0/0, rev 2.00/1.10, addr 2> on 
usbus1
umass0:  SCSI over Bulk-Only; quirks = 0x0000

vm_fault(0xc0ac65f4, 0, 1, 0) -> 1
Fatal kernel mode data abort: 'Translation Fault (S)'
trapframe: 0xc6271a38
FSR=00000005, FAR=00000014, spsr=a00000d3
r0 =c0accdcc, r1 =00000000, r2 =c11306f0, r3 =00000004
r4 =00000000, r5 =00000000, r6 =cc05f270, r7 =c1168d78
r8 =c6271b80, r9 =0000000d, r10=00000030, r11=c6271a98
r12=c6271a6c, ssp=c6271a84, slr=c0209ee8, pc =c00fd518

[thread pid 19 tid 100031 ]
Stopped at      turnstile_broadcast+0x30:       ldr     r2, [r1, #0x014]


Are there any pending patches which I should have put into the code before 
building?

--HPS

From krassi at bulinfo.net  Fri Jul 17 05:36:04 2009
From: krassi at bulinfo.net (Krassimir Slavchev)
Date: Fri Jul 17 05:36:11 2009
Subject: ohci not detected on at91rm9200?
In-Reply-To: <200907162022.25810.hselasky@c2i.net>
References: <4A5C3BF5.9030200@bulinfo.net>
	<200907161033.34560.hselasky@c2i.net>
	<4A5F04B8.1060109@bulinfo.net>
	<200907162022.25810.hselasky@c2i.net>
Message-ID: <4A600DBE.9000901@bulinfo.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Petter Selasky wrote:
> On Thursday 16 July 2009 12:45:12 Krassimir Slavchev wrote:
>> Hans Petter Selasky wrote:
>>>> After adding cdce on arm board:
>>>> cdce0: <USB Ethernet Comm interface> on usbus0
>>>> ue0: <USB Ethernet> on cdce0
>>>> ue0: Ethernet address: 2a:23:45:67:89:54
>>>>
>>>> ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
>>>> 1500 ether 2a:23:45:67:89:54
>>>>
>>>>
>>>> But on the other side running 7.2:
>>>> cdce0: <FreeBSD foundation USB Ethernet Adapter, class 2/0, rev
>>>> 1.10/1.00, addr 2> on uhub4
>>>> cdce0: setting alternate interface failed
>>>> device_attach: cdce0 attach returned 6
>>> Maybe the following patch helps on the ARM side:
>>>
>>> http://perforce.freebsd.org/chv.cgi?CH=166156
>>>
>>> I'm going to test this setup, and will probably resolve the issue, if I
>>> find any, including the cache sync stuff.
>> I have made these changes on the ARM side but the error is same.
> 
> Hi,
> 
> You also need this fix, in addition to the previous one.
> 
> http://perforce.freebsd.org/chv.cgi?CH=166168
> 
> CDC-ethernet should work now.
> 
> --HPS
> 


Yes, it works!

ARM:
ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 2a:23:45:67:89:54
        inet 10.0.0.2 netmask 0xff000000 broadcast 10.255.255.255

# ping -c 5 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
64 bytes from 10.0.0.1: icmp_seq=0 ttl=64 time=3.266 ms
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=2.319 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=2.625 ms
64 bytes from 10.0.0.1: icmp_seq=3 ttl=64 time=2.624 ms
64 bytes from 10.0.0.1: icmp_seq=4 ttl=64 time=2.472 ms

- --- 10.0.0.1 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 2.319/2.661/3.266/0.323 ms

On other side running 7.2:
cdce0: flags=108843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,NEEDSGIANT>
metric 0 mtu 1500
        ether 2a:b9:a0:0c:00:00
        inet 10.0.0.1 netmask 0xff000000 broadcast 10.255.255.255
        media: Ethernet 10baseT/UTP
        status: active

#ping -c 5 10.0.0.2
PING 10.0.0.2 (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: icmp_seq=0 ttl=64 time=1.457 ms
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=1.436 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=1.423 ms
64 bytes from 10.0.0.2: icmp_seq=3 ttl=64 time=1.422 ms
64 bytes from 10.0.0.2: icmp_seq=4 ttl=64 time=1.415 ms

- --- 10.0.0.2 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.415/1.431/1.457/0.015 ms
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (FreeBSD)

iD8DBQFKYA2+xJBWvpalMpkRAoMaAJ99I2jfoePH1KsQlWsVKIacUscPtwCfbWyf
qoMtcrbqJaeoa3Z1Wg1egPE=
=T2Fo
-----END PGP SIGNATURE-----
From raj at semihalf.com  Fri Jul 17 10:24:01 2009
From: raj at semihalf.com (Rafal Jaworowski)
Date: Fri Jul 17 10:24:12 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <200907162156.06598.hselasky@c2i.net>
References: <200906231035.43096.kosmo@semihalf.com>
	<E4BEDDE0-2872-4934-9A53-4121E33390B5@mac.com>
	<B8FCBB1F-F294-4252-82AE-1466631E548F@semihalf.com>
	<200907162156.06598.hselasky@c2i.net>
Message-ID: <F04328A2-99F7-4F00-B03B-9EFEC51B28C4@semihalf.com>


On 2009-07-16, at 21:56, Hans Petter Selasky wrote:

> On Tuesday 14 July 2009 19:20:22 Rafal Jaworowski wrote:
>
>>>> Please note these problems should be considered as a showstopper
>>>> for the release since USB is currently broken on at least three ARM
>>>> platforms in the tree (Marvell).
>
> Hi,
>
> First of all I'm not able to boot into userland on my ARM board.  
> Maybe I
> missed something. I'm using a custom built userland image.
>
> Trying to mount root from ufs:/dev/md0
> warning: no time-of-day clock registered, system time will not be set
> accurately
> Jul 16 18:53:22 init: login_getclass: unknown class 'daemon'
> Fatal kernel mode data abort: 'Alignment Fault 3'
> trapframe: 0xc622cc60
> FSR=00000003, FAR=6e75203a, spsr=600000d3
> r0 =60000093, r1 =c10594a0, r2 =c10594a0, r3 =00000002
> r4 =c10acbd0, r5 =00000000, r6 =6e75203a, r7 =c10594a0
> r8 =c622cd74, r9 =00000000, r10=c0ab081c, r11=c622cccc
> r12=c622ccac, ssp=c622ccac, slr=c020a144, pc =c00c02c8
>
> [thread pid 4 tid 100008 ]
> Stopped at      _thread_lock_flags+0x24:        ldr     r4, [r6]
> db>

Just making sure: your kernel and world are in sync, correct?

> With the following patch in arm/cpufunc.c my UMASS device gets  
> detected at
> boot time (KB9202B ARM board). Else I get a USB request timeout  
> followed by a
> panic. But data transfers of more than 16KByte are likely to be  
> broken.
>
> struct cpu_functions arm9_cpufuncs = {
> ...
> - /*XXX*/	arm9_dcache_wbinv_range,	/* dcache_inv_range	*/
> + /*XXX*/	arm9_dcache_inv_range,	/* dcache_inv_range	*/
>
> };

Hm, this looks indeed strange, but I have no idea why both methods  
point to _wbinv_range; it seems like it was there from the beginning.  
I also looked at NetBSD code (from which FreeBSD/arm originates) and  
they still have it this way too...

Olivier, any thoughts why this was done this way?

(FYI: the cpu we work with fall under ARM9E/ARM10, but the situation  
with dcache_inv_range() is similar as for ARM9 case above).

> Piotr and Rafal: How does ARM work? Are there two caches? One read  
> cache and
> one write cache. Or are they the same. What happens if you read a  
> value from
> RAM into cache, then writeback+invalidate it. Does it get written  
> back to RAM
> or does it get discarded? Sorry, I have some holes in my ARM  
> knowledge which
> you need to fill in.

I don't know what separate read/write CPU caches would be; some  
implementations allow for locking [parts of] cache contents, so it can  
mimic read-only, but it's probably not what you're asking for. Usually  
there's cache at various levels, dedicated for data/instructions, or  
unified, depending on implementation with different characteristics  
(associativity etc.). It seems you need to read about cache basics, as  
there are many aspects, and one of the foremost is we have VIVT  
approach (virtual indexed, virtual tagged) in this context, as opposed  
to physical approach.

> Reference:
>
> ENTRY(arm9_dcache_inv_range)
> 	ldr	ip, .Larm9_line_size
> 	cmp	r1, #0x4000
> 	bcs	.Larm9_dcache_wbinv_all
>
> Anything unclear?

Sorry, but what is the question?

> Are there any pending patches which I should have put into the code  
> before
> building?

There are pmap fixes pending, which will be commited shortly http://people.freebsd.org/~raj/patches/arm/pmap-fixes.diff 
  although according to stas@ they had no visible effect on AT91 (but  
they fix real issues in pmap, possibly just not [yet] observed on AT91).

Rafal

From ticso at cicely7.cicely.de  Fri Jul 17 11:08:29 2009
From: ticso at cicely7.cicely.de (Bernd Walter)
Date: Fri Jul 17 11:08:42 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <200907162156.06598.hselasky@c2i.net>
References: <200906231035.43096.kosmo@semihalf.com>
	<E4BEDDE0-2872-4934-9A53-4121E33390B5@mac.com>
	<B8FCBB1F-F294-4252-82AE-1466631E548F@semihalf.com>
	<200907162156.06598.hselasky@c2i.net>
Message-ID: <20090717105233.GB97863@cicely7.cicely.de>

On Thu, Jul 16, 2009 at 09:56:04PM +0200, Hans Petter Selasky wrote:
> On Tuesday 14 July 2009 19:20:22 Rafal Jaworowski wrote:
> 
> > >> Please note these problems should be considered as a showstopper
> > >> for the release since USB is currently broken on at least three ARM
> > >> platforms in the tree (Marvell).
> 
> Hi,
> 
> First of all I'm not able to boot into userland on my ARM board. Maybe I 
> missed something. I'm using a custom built userland image.
> 
> Trying to mount root from ufs:/dev/md0                                                                                              
> warning: no time-of-day clock registered, system time will not be set 
> accurately                                                    
> Jul 16 18:53:22 init: login_getclass: unknown class 'daemon'                                                                        
> Fatal kernel mode data abort: 'Alignment Fault 3'                                                                                   
> trapframe: 0xc622cc60                                                                                                               
> FSR=00000003, FAR=6e75203a, spsr=600000d3                                                                                           
> r0 =60000093, r1 =c10594a0, r2 =c10594a0, r3 =00000002                                                                              
> r4 =c10acbd0, r5 =00000000, r6 =6e75203a, r7 =c10594a0                                                                              
> r8 =c622cd74, r9 =00000000, r10=c0ab081c, r11=c622cccc                                                                              
> r12=c622ccac, ssp=c622ccac, slr=c020a144, pc =c00c02c8                                                                              
>                                                                                                                                     
> [thread pid 4 tid 100008 ]                                                                                                          
> Stopped at      _thread_lock_flags+0x24:        ldr     r4, [r6]                                                                    
> db>                            

The alignment fault is probably our old friend :(
It comes and goes and adding/removing unrelated drivers in the kernel
can help.
I typically have spi and such, which I enable/disable.

My neweset ARM is running 8 from 17th april.
I had problems with login.conf as well.
login.conf.db had to be erased, otherwise bas things happened.
Not sure if it was panicing the whole kernel or just crashing processes.
Very strange problem, but since I never read about it on this list I
really thought it is my local problem somehow.
Now a few months later your init complains about login classes...

-- 
B.Walter <bernd@bwct.de> http://www.bwct.de
Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.
From hselasky at c2i.net  Fri Jul 17 11:10:14 2009
From: hselasky at c2i.net (Hans Petter Selasky)
Date: Fri Jul 17 11:10:22 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <F04328A2-99F7-4F00-B03B-9EFEC51B28C4@semihalf.com>
References: <200906231035.43096.kosmo@semihalf.com>
	<200907162156.06598.hselasky@c2i.net>
	<F04328A2-99F7-4F00-B03B-9EFEC51B28C4@semihalf.com>
Message-ID: <200907171309.52316.hselasky@c2i.net>

On Friday 17 July 2009 12:23:57 Rafal Jaworowski wrote:
> On 2009-07-16, at 21:56, Hans Petter Selasky wrote:
> > On Tuesday 14 July 2009 19:20:22 Rafal Jaworowski wrote:
> >>>> Please note these problems should be considered as a showstopper
> >>>> for the release since USB is currently broken on at least three ARM
> >>>> platforms in the tree (Marvell).
> >
> > Hi,
> >
> > First of all I'm not able to boot into userland on my ARM board.
> > Maybe I
> > missed something. I'm using a custom built userland image.
> >
> > Trying to mount root from ufs:/dev/md0
> > warning: no time-of-day clock registered, system time will not be set
> > accurately
> > Jul 16 18:53:22 init: login_getclass: unknown class 'daemon'
> > Fatal kernel mode data abort: 'Alignment Fault 3'
> > trapframe: 0xc622cc60
> > FSR=00000003, FAR=6e75203a, spsr=600000d3
> > r0 =60000093, r1 =c10594a0, r2 =c10594a0, r3 =00000002
> > r4 =c10acbd0, r5 =00000000, r6 =6e75203a, r7 =c10594a0
> > r8 =c622cd74, r9 =00000000, r10=c0ab081c, r11=c622cccc
> > r12=c622ccac, ssp=c622ccac, slr=c020a144, pc =c00c02c8
> >
> > [thread pid 4 tid 100008 ]
> > Stopped at      _thread_lock_flags+0x24:        ldr     r4, [r6]
> > db>
>
> Just making sure: your kernel and world are in sync, correct?

Yes, I made a clean toolchain, a clean buildworld and a clean buildkernel.

>
> > With the following patch in arm/cpufunc.c my UMASS device gets
> > detected at
> > boot time (KB9202B ARM board). Else I get a USB request timeout
> > followed by a
> > panic. But data transfers of more than 16KByte are likely to be
> > broken.
> >
> > struct cpu_functions arm9_cpufuncs = {
> > ...
> > - /*XXX*/	arm9_dcache_wbinv_range,	/* dcache_inv_range	*/
> > + /*XXX*/	arm9_dcache_inv_range,	/* dcache_inv_range	*/
> >
> > };
>
> Hm, this looks indeed strange, but I have no idea why both methods
> point to _wbinv_range; it seems like it was there from the beginning.
> I also looked at NetBSD code (from which FreeBSD/arm originates) and
> they still have it this way too...
>
> Olivier, any thoughts why this was done this way?
>
> (FYI: the cpu we work with fall under ARM9E/ARM10, but the situation
> with dcache_inv_range() is similar as for ARM9 case above).
>
> > Piotr and Rafal: How does ARM work? Are there two caches? One read
> > cache and
> > one write cache. Or are they the same. What happens if you read a
> > value from
> > RAM into cache, then writeback+invalidate it. Does it get written
> > back to RAM
> > or does it get discarded? Sorry, I have some holes in my ARM
> > knowledge which
> > you need to fill in.
>
> I don't know what separate read/write CPU caches would be; some
> implementations allow for locking [parts of] cache contents, so it can
> mimic read-only, but it's probably not what you're asking for. Usually
> there's cache at various levels, dedicated for data/instructions, or
> unified, depending on implementation with different characteristics
> (associativity etc.). It seems you need to read about cache basics, as
> there are many aspects, and one of the foremost is we have VIVT
> approach (virtual indexed, virtual tagged) in this context, as opposed
> to physical approach.
>
> > Reference:
> >
> > ENTRY(arm9_dcache_inv_range)
> > 	ldr	ip, .Larm9_line_size
> > 	cmp	r1, #0x4000
> > 	bcs	.Larm9_dcache_wbinv_all
> >
> > Anything unclear?
>
> Sorry, but what is the question?

I was just pointing out in the assembly code, the fact that the invalidate 
range call can in some cases do a writeback invalidate.

>
> > Are there any pending patches which I should have put into the code
> > before
> > building?
>
> There are pmap fixes pending, which will be commited shortly
> http://people.freebsd.org/~raj/patches/arm/pmap-fixes.diff although
> according to stas@ they had no visible effect on AT91 (but they fix real
> issues in pmap, possibly just not [yet] observed on AT91).

Ok.

--HPS

From cognet at ci0.org  Fri Jul 17 12:48:32 2009
From: cognet at ci0.org (Olivier Houchard)
Date: Fri Jul 17 12:49:04 2009
Subject: CPU Cache and busdma usage in USB
In-Reply-To: <F04328A2-99F7-4F00-B03B-9EFEC51B28C4@semihalf.com>
References: <200906231035.43096.kosmo@semihalf.com>
	<E4BEDDE0-2872-4934-9A53-4121E33390B5@mac.com>
	<B8FCBB1F-F294-4252-82AE-1466631E548F@semihalf.com>
	<200907162156.06598.hselasky@c2i.net>
	<F04328A2-99F7-4F00-B03B-9EFEC51B28C4@semihalf.com>
Message-ID: <20090717121746.GA38852@ci0.org>

On Fri, Jul 17, 2009 at 12:23:57PM +0200, Rafal Jaworowski wrote:
> 
> >With the following patch in arm/cpufunc.c my UMASS device gets  
> >detected at
> >boot time (KB9202B ARM board). Else I get a USB request timeout  
> >followed by a
> >panic. But data transfers of more than 16KByte are likely to be  
> >broken.
> >
> >struct cpu_functions arm9_cpufuncs = {
> >...
> >- /*XXX*/	arm9_dcache_wbinv_range,	/* dcache_inv_range	*/
> >+ /*XXX*/	arm9_dcache_inv_range,	/* dcache_inv_range	*/
> >
> >};
> 
> Hm, this looks indeed strange, but I have no idea why both methods  
> point to _wbinv_range; it seems like it was there from the beginning.  
> I also looked at NetBSD code (from which FreeBSD/arm originates) and  
> they still have it this way too...
> 
> Olivier, any thoughts why this was done this way?
> 
> (FYI: the cpu we work with fall under ARM9E/ARM10, but the situation  
> with dcache_inv_range() is similar as for ARM9 case above).
> 

I can't get the reason why it's this way, reading the CVS logs from 
NetBSD doesn't enlight me, but I'm pretty sure we should switch to use
the proper function for _inv_range, as done in this patch. That's what we
do for Xscale/Xscale core 3 anyway, and if it breaks something, we should
jut fix it.

Olivier
From xiechao.mail at gmail.com  Mon Jul 20 04:46:19 2009
From: xiechao.mail at gmail.com (Chao Xie)
Date: Mon Jul 20 04:46:26 2009
Subject: start_init data abort
Message-ID: <c11a0ecf0907192038t6ba7ba11g2b41591fe4285e3@mail.gmail.com>

hi
I am trying to port Freebsd to pxa310. After I have compiled, and get
kernel.bin, i will directly download it to SDRAM, and begin to run it. The
SYSINITs are all right, and i get the final init - start_init. At this time,
a exception of data abort happens.
After i debug it, i find it is caused by subyte at the following code.
/*
         * Move out the boot flag argument.
         */
        options = 0;
        ucp = (char *)p->p_sysent->sv_usrstack;
        (void)subyte(--ucp, 0);        /* trailing zero */
        if (boothowto & RB_SINGLE) {
            (void)subyte(--ucp, 's');
            options = 1;
        }

It seems that subyte will store a byte to user stack of initproc. The user
stack of initproc is started at 0xC0000000. i have checked the page mapping,
and find the veirtual address 0xC0000000 - PAGESIZE to 0xC0000000 is not
mapped. So i get the data abort.
I am confuesed. Where should i map the user stack of initproc? I appreciate
that someone can help me.Thanks.
From bugmaster at FreeBSD.org  Mon Jul 20 11:06:51 2009
From: bugmaster at FreeBSD.org (FreeBSD bugmaster)
Date: Mon Jul 20 11:07:37 2009
Subject: Current problem reports assigned to freebsd-arm@FreeBSD.org
Message-ID: <200907201106.n6KB6oY0002199@freefall.freebsd.org>

Note: to view an individual PR, use:
  http://www.freebsd.org/cgi/query-pr.cgi?pr=(number).

The following is a listing of current problems submitted by FreeBSD users.
These represent problem reports covering all versions including
experimental development code and obsolete releases.


S Tracker      Resp.      Description
--------------------------------------------------------------------------------
o arm/134368   arm        [patch] nslu2_led driver for the LEDs on the NSLU2
o arm/134338   arm        [patch] Lock GPIO accesses on ixp425
o arm/134092   arm        [patch] NSLU.hints contains wrong hints for on board n

3 problems total.

From tinguely at casselton.net  Tue Jul 21 16:48:03 2009
From: tinguely at casselton.net (Mark Tinguely)
Date: Tue Jul 21 16:48:10 2009
Subject: start_init data abort
In-Reply-To: <c11a0ecf0907192038t6ba7ba11g2b41591fe4285e3@mail.gmail.com>
Message-ID: <200907211648.n6LGm2KW076061@casselton.net>

>
>  hi
>  I am trying to port Freebsd to pxa310. After I have compiled, and get
>  kernel.bin, i will directly download it to SDRAM, and begin to run it. The
>  SYSINITs are all right, and i get the final init - start_init. At this time,
>  a exception of data abort happens.
>  After i debug it, i find it is caused by subyte at the following code.
>  /*
>           * Move out the boot flag argument.
>           */
>          options = 0;
>          ucp = (char *)p->p_sysent->sv_usrstack;
>          (void)subyte(--ucp, 0);        /* trailing zero */
>          if (boothowto & RB_SINGLE) {
>              (void)subyte(--ucp, 's');
>              options = 1;
>          }
>
>  It seems that subyte will store a byte to user stack of initproc. The user
>  stack of initproc is started at 0xC0000000. i have checked the page mapping,
>  and find the veirtual address 0xC0000000 - PAGESIZE to 0xC0000000 is not
>  mapped. So i get the data abort.
>  I am confuesed. Where should i map the user stack of initproc? I appreciate
>  that someone can help me.Thanks.

In the begining of that routine, the user stack is put into the map
backed by a default (NULL) object. A page for the stack should fault
a page into that location when accessed. For testing, you can speed it
along a bit by setting the MAP_PREFAULT bit:

	addr = p->p_sysent->sv_usrstack - PAGE_SIZE;
	if (vm_map_find(&p->p_vmspace->vm_map, NULL, 0, &addr, PAGE_SIZE,
-			FALSE, VM_PROT_ALL, VM_PROT_ALL, 0) != 0)
+			FALSE, VM_PROT_ALL, VM_PROT_ALL, MAP_PREFAULT) != 0)
                panic("init: couldn't allocate argument space");

If there is still no page at that location and you have a console, you
can print out the vm_map.
From xcllnt at mac.com  Thu Jul 23 19:53:32 2009
From: xcllnt at mac.com (Marcel Moolenaar)
Date: Thu Jul 23 19:53:38 2009
Subject: About the "USB Cache and busdma usage in USB" thread
Message-ID: <3E1658AF-67C6-4E61-B6E7-BEF528C3FF4D@mac.com>

All,

I went over the thread and this is what I have to say about it:

Using busdma to manage/control CPU caches is wrong for the
following simple reason: bus_dmamap_sync() has the side-effect
of copying to and from the bounce buffer (if applicable).

CPU caches should be kept coherent by using an appropriate API.
We already have cpu_flush_dcache(). All we have to do is add
cpu_inval_dcache() and let the MD code determine how best to
do this -- even if they decide to use busdma.

In general: D-cache and I-cache control/handling should not be
hidden from MI code. It should not be treated as an artifact of
some platform. It should not be implemented by banking on some
side-effect of other function(s). We only achieve efficient
cache control if MI code calls appropriate APIs so that we can
precisely express what we need to achieve at that point.

For example: when we write a breakpoint into the text segment
of some process by using ptrace(2), the ptrace(2) code must
call an appropriate API to make sure that the I-cache is made
coherent with memory. This may require a previous D-cache
flush! We should not kluge uiomove(9) like we did on PowerPC
to deal with this. Note ARM and ia64 are still broken in this
respect.

My $0.02...

-- 
Marcel Moolenaar
xcllnt@mac.com



From hselasky at c2i.net  Thu Jul 23 21:10:02 2009
From: hselasky at c2i.net (Hans Petter Selasky)
Date: Thu Jul 23 21:10:09 2009
Subject: About the "USB Cache and busdma usage in USB" thread
In-Reply-To: <3E1658AF-67C6-4E61-B6E7-BEF528C3FF4D@mac.com>
References: <3E1658AF-67C6-4E61-B6E7-BEF528C3FF4D@mac.com>
Message-ID: <200907232209.47729.hselasky@c2i.net>

On Thursday 23 July 2009 20:53:06 Marcel Moolenaar wrote:
> All,
>
> I went over the thread and this is what I have to say about it:
>
> Using busdma to manage/control CPU caches is wrong for the
> following simple reason: bus_dmamap_sync() has the side-effect
> of copying to and from the bounce buffer (if applicable).
>
> CPU caches should be kept coherent by using an appropriate API.
> We already have cpu_flush_dcache(). All we have to do is add
> cpu_inval_dcache() and let the MD code determine how best to
> do this -- even if they decide to use busdma.
>
> In general: D-cache and I-cache control/handling should not be
> hidden from MI code. It should not be treated as an artifact of
> some platform. It should not be implemented by banking on some
> side-effect of other function(s). We only achieve efficient
> cache control if MI code calls appropriate APIs so that we can
> precisely express what we need to achieve at that point.
>
> For example: when we write a breakpoint into the text segment
> of some process by using ptrace(2), the ptrace(2) code must
> call an appropriate API to make sure that the I-cache is made
> coherent with memory. This may require a previous D-cache
> flush! We should not kluge uiomove(9) like we did on PowerPC
> to deal with this. Note ARM and ia64 are still broken in this
> respect.

Hi,

I would be fine with a solution where cpufunctions are used directly in USB. 
The only problem is that if bounce pages are used, which happens in the case 
of loading kernel virtual data into DMA, then busdma sync calls would still be 
required.

--HPS

From imp at bsdimp.com  Sat Jul 25 03:36:13 2009
From: imp at bsdimp.com (M. Warner Losh)
Date: Sat Jul 25 03:36:20 2009
Subject: About the "USB Cache and busdma usage in USB" thread
In-Reply-To: <200907232209.47729.hselasky@c2i.net>
References: <3E1658AF-67C6-4E61-B6E7-BEF528C3FF4D@mac.com>
	<200907232209.47729.hselasky@c2i.net>
Message-ID: <20090724.233404.-399282844.imp@bsdimp.com>

In message: <200907232209.47729.hselasky@c2i.net>
            Hans Petter Selasky <hselasky@c2i.net> writes:
: On Thursday 23 July 2009 20:53:06 Marcel Moolenaar wrote:
: > All,
: >
: > I went over the thread and this is what I have to say about it:
: >
: > Using busdma to manage/control CPU caches is wrong for the
: > following simple reason: bus_dmamap_sync() has the side-effect
: > of copying to and from the bounce buffer (if applicable).
: >
: > CPU caches should be kept coherent by using an appropriate API.
: > We already have cpu_flush_dcache(). All we have to do is add
: > cpu_inval_dcache() and let the MD code determine how best to
: > do this -- even if they decide to use busdma.
: >
: > In general: D-cache and I-cache control/handling should not be
: > hidden from MI code. It should not be treated as an artifact of
: > some platform. It should not be implemented by banking on some
: > side-effect of other function(s). We only achieve efficient
: > cache control if MI code calls appropriate APIs so that we can
: > precisely express what we need to achieve at that point.
: >
: > For example: when we write a breakpoint into the text segment
: > of some process by using ptrace(2), the ptrace(2) code must
: > call an appropriate API to make sure that the I-cache is made
: > coherent with memory. This may require a previous D-cache
: > flush! We should not kluge uiomove(9) like we did on PowerPC
: > to deal with this. Note ARM and ia64 are still broken in this
: > respect.
: 
: Hi,
: 
: I would be fine with a solution where cpufunctions are used directly in USB. 
: The only problem is that if bounce pages are used, which happens in the case 
: of loading kernel virtual data into DMA, then busdma sync calls would still be 
: required.

They are needed on i386 kernels with more than 4GB of ram...  Or ram
located above 4GB...

Warner
From bugmaster at FreeBSD.org  Mon Jul 27 11:06:50 2009
From: bugmaster at FreeBSD.org (FreeBSD bugmaster)
Date: Mon Jul 27 11:07:44 2009
Subject: Current problem reports assigned to freebsd-arm@FreeBSD.org
Message-ID: <200907271106.n6RB6nft018859@freefall.freebsd.org>

Note: to view an individual PR, use:
  http://www.freebsd.org/cgi/query-pr.cgi?pr=(number).

The following is a listing of current problems submitted by FreeBSD users.
These represent problem reports covering all versions including
experimental development code and obsolete releases.


S Tracker      Resp.      Description
--------------------------------------------------------------------------------
o arm/134368   arm        [patch] nslu2_led driver for the LEDs on the NSLU2
o arm/134338   arm        [patch] Lock GPIO accesses on ixp425
o arm/134092   arm        [patch] NSLU.hints contains wrong hints for on board n

3 problems total.

From merketing.e at email.it  Wed Jul 29 19:46:42 2009
From: merketing.e at email.it (M. Lorona)
Date: Wed Jul 29 19:46:47 2009
Subject: Richiesta
Message-ID: <20090729192106.236B1109F89@davidovitch.com>

<html>
<TABLE>
<TR>
<TD>
<b>SOFTWARE PER L' EMAIL-MARKETING</b>
</TD>
</TR>
<tr><td><hr></td></tr>
<TR><TD>
<LI>Software che estrapola dati dalle Pagine Gialle e Bianche.<br>
<LI>Software per creare le email info@ da ricerche on line su motori
<li>Software Gestionale per inviare newsletter
<li>800 server smtp per inviare le tue newsletter (10.000 per ora)<br>
ps: il servizio smtp non ? soggetto a sospensioni del servizio per segnalazione di terzi.<br>
Creare appuntamenti inbound in poco tempo si pu?, cosa aspetti, noi ti diamo gli strumenti.
</td>
</tr>
<tr><td><hr></td></tr>
<tr>
<td>
se sei interessato all'acquisto clicca il link sottostante.
</TD></TR>
<tr>
<td bgcolor=green height=1></td></tr>
<tr>
<td align=center><a href="mailto:email.marketing.spa@gmail.com?subject=Richiesta info&Body=Indichi:%0ARagione%20Sociale%0ANome%0ACogome%0ATelefono%0Aquanti%20venditori%20lavorano%20per%20voi"><b>Clicca per ulteriori chiarimenti</b></a></td></tr>
<td bgcolor=green height=1></td></tr>
<tr><td>Se non riesci ad aprire il link scrivi una e-mail email.marketing.spa@gmail.com</td></tr>
From info at omegaworldclass.org  Wed Jul 29 21:52:53 2009
From: info at omegaworldclass.org (Customer Insights)
Date: Wed Jul 29 21:53:22 2009
Subject: 'Mastering Customer Insights & Superior Marketing Strategies"
	Workshop 2009, 2 September @ Conrad Hotel, Bangkok
Message-ID: <20090730045242.416768609@omegaworldclass.org>