Ubiquiti EdgeRouter Lite works multi-user with -CURRENT.

[Joe Holden]

Joe Holden wrote:
> Juli Mallett wrote:
>> On Sat, May 25, 2013 at 7:35 AM, Milan Obuch <freebsd-mips at dino.sk> 
>> wrote:
>>
>>> On Thu, 23 May 2013 10:56:04 -0700, Juli Mallett <jmallett at FreeBSD.org>
>>> wrote:
>>>
>>> [ snip ]
>>>
>>>> In this case, I'd say that the most useful thing to instrument is
>>>> likely cvm_do_timer in sys/mips/cavium/octe/ethernet.c.  I'd suggest
>>>> first changing:
>>>>
>>>> 138 if (priv->need_link_update) {
>>>> 139 updated++;
>>>> 140 taskqueue_enqueue(cvm_oct_link_taskq, &priv->link_task);
>>>> 141 }
>>>>
>>>> To be just
>>>>
>>>> taskqueue_enqueue(cvm_oct_link_taskq, &priv->link_task);
>>>>
>>> It took me some time to respond for some reasons, anyway I built a
>>> kernel with this change and there is something clearly wrong. There is
>>> basically no change in behavior, if booted with gigabit connection, it
>>> works with gigabit speed but not with 100 megabit and vice versa. Just
>>> some cometic change - once per two seconds or so I keep getting on
>>> console following
>>>
>>> octe0: Link down
>>> octe1: Link down
>>> octe2: Link down
>>>
>>> It looks like some phy register is not read correctly or not correct
>>> phy register is read and acted upon. Just a guess, of course.
>>> Interesting thing also not yet mentioned here - ifconfig octeN output
>>> show correct speed in its media: line.
>>>
>>> Also, when changing cables, I got following printed on console:
>>>
>>> Port 0 receive error code 13, packet dropped
>>>
>>> No idea it if shows something, for now.
>>>
>>>> If that helps, then I think there's two things that need to be done:
>>>>
>>>> First, we shouldn't use slow-polling to do that anyway, we should just
>>>> call taskqueue_enqueue whenever we set need_link_update to 1 if it was
>>>> not 1 already.  This is mostly a structural-cosmetic kind of thing.
>>>>
>>>> Second, cvm_oct_rgmii_poll in ethernet-rgmii.c is probably wrong in
>>>> some way.  Instrumenting it to print out the various register states
>>>> and see if a human can see link status changes even if it can't may be
>>>> helpful.  Alternately, you may want/need to look at UBNT's patches for
>>>> the ERLite.  I may have missed some change it requires (if so, please
>>>> let me know rather than just committing the changes; I've tried to
>>>> isolate and keep minimal vendor-specific changes since we support so
>>>> many vendors; not every vendor shares my concern, and a lot of them
>>>> make the code too complex, or other things we don't want.)  It could
>>>> be just a matter of not using/programming/undermining RGMII in the way
>>>> the hardware wants, or it could be that we need a PHY driver or
>>>> something else.
>>>>
>>>> Actually, looking at how the hardware is set up, it looks like we
>>>> should be using a PHY driver already, so it may be that none of that
>>>> is helpful (since the internal link status management is only used if
>>>> we don't have a PHY) and the real fix needs to happen in atphy.c.  Or
>>>> that for this hardware we should just use the internal link status
>>>> management rather than using the PHY.
>>>>
>>> In dmesg, following is relevant to ethernet, I think:
>>>
>>> octebus0: <Cavium Octeon Ethernet pseudo-bus> on ciu0
>>> Interface 0 has 3 ports (RGMII)
>>> Warning: Enabling IPD when IPD already enabled.
>>> Warning: Enabling PKO when PKO already enabled.
>>> octe0: <Cavium Octeon RGMII Ethernet> on octebus0
>>> miibus0: <MII bus> on octe0
>>> atphy0: <Atheros F1 10/100/1000 PHY> PHY 7 on miibus0
>>> atphy0: OUI 0x00c82e, model 0x0007, rev. 2
>>> atphy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX,
>>> 1000baseSX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto octe0: bpf
>>> attached octe0: Ethernet address: dc:9f:db:29:a3:92
>>> octe1: <Cavium Octeon RGMII Ethernet> on octebus0
>>> miibus1: <MII bus> on octe1
>>> atphy1: <Atheros F1 10/100/1000 PHY> PHY 6 on miibus1
>>> atphy1: OUI 0x00c82e, model 0x0007, rev. 2
>>> atphy1:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX,
>>> 1000baseSX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto octe1: bpf
>>> attached octe1: Ethernet address: dc:9f:db:29:a3:93
>>> octe2: <Cavium Octeon RGMII Ethernet> on octebus0
>>> miibus2: <MII bus> on octe2
>>> atphy2: <Atheros F1 10/100/1000 PHY> PHY 5 on miibus2
>>> atphy2: OUI 0x00c82e, model 0x0007, rev. 2
>>> atphy2:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX,
>>> 1000baseSX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto octe2: bpf
>>> attached octe2: Ethernet address: dc:9f:db:29:a3:94
>>>
>>> so maybe something should be done in atphy driver as someone mentioned
>>> later in this thread or between phy driver and some cavium code... I am
>>> going to try to undestand the code in cavium/octe directory, but any
>>> hint would be welcome for sure.
>>>
>>
>> Ok, well, I have a slightly-radical hint :)
>>
>> In cvm_oct_mdio_setup_device, set phy_id to -1 instead of to
>> cvmx_helper_board_get_mii_address(...).  It's a silly thought, but I 
>> think
>> there's a chance it might work.  This would be using the internal link
>> state management rather than using a PHY.  (This isn't the right 
>> change to
>> make to effect that, but it's worth a shot.)
>>
>> Thanks,
>> Juli.
> Just gave that a shot - results in unknown link type and no connectivity
> _______________________________________________

Hm, one thing I have noticed is that when using carp, packets destined 
for the virtual mac address never reach the kernel - is this due to the 
hardware acceleration, and if so, is it possible to disable checking or 
some such without losing that ability (although at this point it may be 
futile anyway, since vlans and bridges seem to impair it's usefulness 
anyway)

Thanks,
Joe