DWC OTG TX path optimisation for 11-current

[Svatopluk Kraus]

On Fri, Aug 14, 2015 at 11:49 AM, Hans Petter Selasky <hps at selasky.org> wrote:
> On 08/12/15 20:16, Svatopluk Kraus wrote:
>>
>> On Wed, Aug 12, 2015 at 3:35 PM, Hans Petter Selasky <hps at selasky.org>
>> wrote:
>>>
>>> On 07/31/15 17:11, Svatopluk Kraus wrote:
>>>>
>>>>
>>>> Note that I got about 24000 interrupts per a second during buildworld
>>>> too. If I remember it correctly, it was about 4000 before r285935.
>>>> When the trigger is pulled, the count is changing (in range of 4000 to
>>>> 21000). It looks like something is taking more time (interrupt
>>>> servicing probably). According to "systat -v", disk is going to be
>>>> 100% (and more) busy very often for example.
>>>
>>>
>>>
>>> Hi,
>>>
>>> Finally my RPI-2 arrived :-)
>>
>>
>>
>> It's great! ;) I can do some debugging but it's hard to go deeper
>> without usb controller specification.
>>
>>
>>>
>>> After r285935 we are polling for data more frequently, so the IRQ rate
>>> will
>>> go up. 3x 8000 IRQs/second would be normal to query the device every
>>> 125us,
>>> which the EHCI does.
>>>
>>> Can you give me some more information about your setup, so that I can
>>> reproduce it?
>>
>>
>> I use in-tree RPI2 configuration but with INVARIANTS and
>> INVARIANT_SUPPORT options disabled. This is crucial as it's hard to
>> trigger it with these options enabled.
>>
>> I use usb disk with three partitions:
>> First one is mounted as a root.
>> Second one is used for swapping.
>> The third one is not mounted.
>>
>> The disk is connected thru usb hub.
>>
>> My fstab is empty.
>> My kernel is loaded from SD card.
>> I use UART as console.
>>
>> root at raspberry-pi:~ # usbconfig
>> ugen0.1: <OTG Root HUB DWCOTG> at usbus0, cfg=0 md=HOST spd=HIGH
>> (480Mbps) pwr=SAVE (0mA)
>> ugen0.2: <product 0x9514 vendor 0x0424> at usbus0, cfg=0 md=HOST
>> spd=HIGH (480Mbps) pwr=SAVE (2mA)
>> ugen0.3: <product 0xec00 vendor 0x0424> at usbus0, cfg=0 md=HOST
>> spd=HIGH (480Mbps) pwr=ON (2mA)
>> ugen0.4: <USB2.0 Hub VIA Labs, Inc.> at usbus0, cfg=0 md=HOST spd=HIGH
>> (480Mbps) pwr=SAVE (0mA)
>> ugen0.5: <USB2.0 Hub VIA Labs, Inc.> at usbus0, cfg=0 md=HOST spd=HIGH
>> (480Mbps) pwr=SAVE (0mA)
>> ugen0.6: <HD710 ADATA> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON
>> (0mA)
>>
>>
>> FreeBSD source tree is on /usr/src and I run native buildworld : "make
>> -j6 buildworld"
>> I run the buildworld on ssh to keep my console "clean and free" for
>> debugging. The trigger is pulled in about 25 minutes, however it's
>> more expessive with more time elapsed.
>>
>> So far it looks that it happens during "stage 2.1: cleaning up the
>> object tree" when a lot of small and different disk request are done.
>>
>> I have tried to get big picture of the problem with KTR option
>> enabled, but it's not possible as it's not triggered when all KTR
>> classes are compiled in. What works is the following configuration:
>>
>> options  KTR
>> options  KTR_ENTRIES=65536
>> options  KTR_MASK=(KTR_SPARE3)
>> options  KTR_COMPILE=(KTR_SPARE3)
>>
>> However, even using of KTR_SPARE3 class for debugging is limited
>> considering the trigger.
>>
>> I will send you more info about my investigation tomorow.
>>
>> Svata
>>
>
> Hi,
>
> Here are some more clues. I've found a situation where if one USB class
> driver gets too busy, the others gets starved. I have a patch in the works
> to implement a round robin mechanism for this.
>
> In the meanwhile, maybe you can try to:
>
> cpuset -l 1-3 -p XXX
>
> Where XXX is pid for the usbus0 processes, typically 13 with my RPI-2.
>
> The DWC OTG interrupts on CPU #0 and if the usbus0 processes are running
> there too, it is suboptimal.


Well, I have tried it and it ran fine but it almost blocked buildworld
in time when the trigger is pulled usually. According to systat, the
disk is almost all time 100% and more busy.

Svata