HEADS UP: Cavium ThunderX support in the tree

[Zbigniew Bodek]

2015-11-06 12:36 GMT+01:00 Andrew Turner <andrew at fubar.geek.nz>:
> On Fri, 6 Nov 2015 12:23:11 +0100
> Zbigniew Bodek <zbb at semihalf.com> wrote:
>
>> 2015-11-06 10:47 GMT+01:00 Andrew Turner <andrew at fubar.geek.nz>:
>> > On Thu, 5 Nov 2015 17:14:03 +0100
>> > Zbigniew Bodek <zbb at semihalf.com> wrote:
>> >
>> >> Hello all,
>> >>
>> >> Semihalf is happy to officially announce that starting from SVN
>> >> revision r289550 FreeBSD is ready to run on Cavium ThunderX
>> >> system-on-chip!
>> >> ThunderX is the first ARM64 (ARMv8) hardware platform to be
>> >> supported by FreeBSD and the only one that introduces 96-CPU cores
>> >> SMP. The integrated code includes support for:
>> >>
>> > ...
>> >> - Network Interface Controller (VNIC) 1/10/40G
>> >
>> > We do? Then why don't I see it in ifconfig on the ThunderX in the
>> > cluster?
>> >
>> > root at cavium:~ # ifconfig -a
>> > em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
>> > 1500
>> > options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
>> >         ether 00:15:17:0b:6b:08
>> >         inet 192.168.5.167 netmask 0xffffff00 broadcast
>> > 192.168.5.255 media: Ethernet autoselect (1000baseT <full-duplex>)
>> >         status: active
>> >         nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>> > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>> >         options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
>> >         inet6 ::1 prefixlen 128
>> >         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
>> >         inet 127.0.0.1 netmask 0xff000000
>> >         groups: lo
>> >         nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
>> >
>> > Andrew
>>
>> Strange. Works for me ;P
>>
>> First of all, networking cards on ThunderX are PCI Virtual Functions
>> so you need to bring them up using iovctl.
>> If you watched the youtube demo you should have seen that at first
>> there are no interfaces in ifconfig and then I do iovctl and they
>> appear.
>
> Ok, so it's similar to wlan where we need some extra configuration
> before we can use it. What is the option I need to enable in rc.conf to
> enable this?

Yes. You need to invoke iovctl with proper parameters and configuration file.
https://www.freebsd.org/cgi/man.cgi?query=iovctl.conf&sektion=5&apropos=0&manpath=FreeBSD+11-current

>
>> Secondly, EFI that runs on the board needs to pass the correct DTB to
>> the kernel (it has to have PCIB, BGX, MDIO, PHYs nodes and they need
>> to be in the correct order [MDIO needs to attach before BGX]).
> If this is the case the code is broken. It should just work with
> the vendor dtb and GENERIC.

Thanks for quick evaluation but please see comment below.

>
> We already have to handle this in other parts of the code by having the
> dependent drivers on an earlier pass than later drivers.

MDIO is a platform device whereas BGX, NICPF (physical function) and
NICVF (virtual function) are devices on PCI.
Therefore if the PCIB is placed prior to MDIO in DTB all devices on
this PCI bridge will be attached before MDIO is attached.
As far as I know there is no straightforward way to create a logical
relation between device on PCI and platform devices enumerated using
DTB.
That is why we asked Cavium to change the order in their DTBs.
Of course if you know how to workaround that I would be happy to apply
this change to "unbreak" the existing code.

>
> I also don't see any documentation. I can see the bgx driver, bit there
> doesn't seem to be a manual page.
> root at cavium:~ # man bgx
> No manual entry for bgx
>
> Andrew

Correct. There is no manual entry for: BGX, Cavium MDIO, NICPF and NICVF.

Best regards
zbb