cvs commit: src/sys/powerpc/booke machdep.c

[Rafal Jaworowski]

Bill Paul wrote:
>>>> BTW: currently U-Boot is broken for most of PPC variations (including
>>>> 85xx) in
>>>> the following way: d-cache is not being flushed before disabling and
>>>> strange
>>>> things happen as you imagine. For a quick workaround apply something
>>>> like this
>>>> when using ELF images:
>>>> http://people.freebsd.org/~raj/patches/misc/uboot-dcache-workaround.diff
>>> Yup. That's what we do at Juniper. I don't see any bad effects other
>>> than having to invalidate the cache before (re-)enabling it so I don't
>>> have that patch applied myself...
>>>
>> I have seen strange things, including hangs, and also TSEC non-functional
>> effects due to this issue... :)
>>
>>> The problem I'm having with U-Boot is with the TSEC driver as you've
>>> seen on the U-Boot mailing list. I have one MPC8555CDS sitting idle
>>> after the U-Boot upgrade because I can't netboot :-/
>>>
>> Yes, I saw. Let me re-flash my 8555 with the latest image and try, but when
>> I'm not remote i.e. Monday. Don't know if you know this tip, but there used to
>> be a second flash bank on the MPC8555CDS that you can access while running
>> from the boot one, so you can recover in such scenario without a h/w debugger
>> by copying over that backup flash bank, provided you have a working image there...
>>
>> Rafal
>>
> 
> I am not on the U-Boot mailing list, so I'm not familiar with the TSEC
> problems you're having, but I'm curious: are you saying that when you
> try to boot a FreeBSD kernel image with the cache disabled, U-Boot can
> download and start the kernel, but FreeBSD fails to mount its rootfs
> correctly because of problems with the TSEC?
> 

I guess I didn't say anything like this :) There are two issues, both local to
U-Boot and not affecting kernel TSEC driver operation:

1. What Marcel is suffering from is some basic problem with TSEC not working
at all (link failure) after U-Boot update on one of his systems.

2. I referred to a U-Boot bug present on most PPC variations, which might
surface when running standalone apps enveloped in ELF: d-cache is disabled
before passing control, without flushing it first, so depending on the data
which didn't get flushed there might be corruptions or undesired effects
within the standalone app. One of the problems I was observing was that TSEC
stopped working after return from a standalone app to U-Boot, also other
corrupt data happend, hangs etc.

FreeBSD loader(8) for U-Boot environment is such an external app and when this
issue hit the loader wouldn't usually be able to load the kernel via network
at all. The pure binary produced from this same ELF image works without ill
effects ('go' command doesn't do cache trickery before passing control).

Please note the workaround patch for handling ELF images does not disable
cache, just the opposite: it prevents from doing so.

> If that is what you're saying, then something about this is very confusing:
> the FreeBSD TSEC driver appears to be enabling the TSEC's RX and TX snooping
> featurss, so in theory (if I understand things correctly, which I might not),
> the cache behavior really shouldn't affect it. With VxWorks, the bootrom
> typically runs with the data cache disabled and then the VxWorks image
> enables it once it's loaded, but the same TSEC driver works properly in
> both scenarios.
> 

Our TSEC driver works fine with caches enabled.

> Some questions that I have:
> 
> - Is bus_dma aware of the fact that the TSEC is snooped? Because if it is,
>   it seems to me that all the sync operations should reduce to no-ops.
> 

_bus_dmamap_sync() for PowerPC is a no-op. TSEC driver uses DMA sync ops per
convention (it's also possible it could be integrated with something else than
PowerPC core one day, so we are prepared :)

> - In the encap routine, the driver always sets the R bit immediately on the
>   first descriptor of every outbound frame. (The Ready bit is basically
>   the ownership bit that indicates to the TSEC that the descriptor is
>   ready for processing.) The VxWorks driver always leaves the R bit in
>   first descriptor cleared until all subsequent descriptors have been
>   set up. This is to prevent the TSEC from potentially racing with the
>   encap routine and processing the first few descriptors of an outbound
>   frame before all descriptors are ready. It might be a good idea to have
>   the encap routine in the FreeBSD driver work the same way. I know the
>   idea is to set up all the descriptors first and then initiate transmission
>   by setting the THLT bit in the TSTAT register, but this doesn't account
>   for the possibility that the transmitter is already busy with a previous
>   transmit request when you go to set up new descriptors.
> 

Thanks for pointing this out. I'll have a look; there are more changes and
improvements to TSEC in my queue, hope you don't mind reviewing these too?

> - Has anyone actually tried to debug the TSEC issue, such as by setting up
>   tcpdump to capture traffic from the 8555 or instrumenting the driver?
>   It might be worthwhile to actually find the problem rather than just
>   saying "it works if we hack U-Boot" since this might be a symptom of
>   a larger problem that needs to be fixed.
> 

As above, the hack is preventing from some other problems, that need to be
addressed in U-Boot code, and not quite related to the Ethernet controller and
its driver.

Rafal