Recommendations for 10gbps NIC

[Alexander V. Chernikov]

On 27.07.2013 12:15, Luigi Rizzo wrote:
> On Sat, Jul 27, 2013 at 10:02 AM, Alexander V. Chernikov
> <melifaro at freebsd.org>  wrote:
>> On 27.07.2013 02:14, Barney Cordoba wrote:
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>> *From:* Daniel Feenberg<feenberg at nber.org>
>>> *To:* Alexander V. Chernikov<melifaro at FreeBSD.org>
>>> *Cc:* Barney Cordoba<barney_cordoba at yahoo.com>;
>>> "freebsd-net at freebsd.org"<freebsd-net at freebsd.org>
>>> *Sent:* Friday, July 26, 2013 4:59 PM
>>>
>>> *Subject:* Re: Recommendations for 10gbps NIC
>>>
>>>
>>> On Fri, 26 Jul 2013, Alexander V. Chernikov wrote:
>>>
>>>   >  On 26.07.2013 19:30, Barney Cordoba wrote:
>>>   >>
>>>   >>
>>>   >>
>>> ------------------------------------------------------------------------
>>>   >>  *From:* Alexander V. Chernikov<melifaro at FreeBSD.org
>>> <mailto:melifaro at FreeBSD.org>>
>>>   >>  *To:* Boris Kochergin<spawk at acm.poly.edu<mailto:spawk at acm.poly.edu>>
>>>   >>  *Cc:* freebsd-net at freebsd.org<mailto:freebsd-net at freebsd.org>
>>>
>>>   >>  *Sent:* Thursday, July 25, 2013 2:10 PM
>>>   >>  *Subject:* Re: Recommendations for 10gbps NIC
>>>   >>
>>>   >>  On 25.07.2013 00:26, Boris Kochergin wrote:
>>>   >>  >  Hi.
>>>   >>  Hello.
>>>   >>  >
>>>   >>  >  I am looking for recommendations for a 10gbps NIC from someone who
>>> has
>>>   >>  >  successfully used it on FreeBSD. It will be used on FreeBSD
>>> 9.1-R/amd64
>>>   >>  >  to capture packets. Some desired features are:
>>>   >>  >
>>>
>>> We have experience with HP NC523SFP and Chelsio N320E. The key difference
>>> among 10GBE cards for us is how they treat foreign DACs. The HP would PXE
>>> boot with several brands and generic DACs, but the Chelsio required a
>>> Chelsio brand DAC to PXE boot. There was firmware on the NIC to check the
>>> brand of cable. Both worked fine once booted. The Chelsio cables were hard
>>> to find, which became a problem. Also, when used with diskless Unix
>>> clients the Chelsio cards seemed to hang from time to time. Otherwise
>>> packet loss was one in a million for both cards, even with 7 meter cables.
>>>
>>> We liked the fact that the Chelsio cards were single-port and cheaper. I
>>> don't really understand why nearly all 10GBE cards are dual-port. Surely
>>> there is a market for NICs between 1 gigabit and 20 gigabit.
>>>
>>> The NIC heatsinks are too hot to touch during use unless specially cooled.
>>>
>>> Daniel Feenberg
>>> NBER
>>>
>>>
>>> ---------------------
>>> The same reason that they don't make single core cpus anymore. It costs
>>> about the
>>> same to make a 1 port chip as a 2 port chip.
>>>
>>> I find it interesting how so many talk about "the cards", when most
>>> often the
>>> differences are with "the drivers". Luigi made the most useful comment;
>>> if you ever
>>> want to use netmap, you need to buy a card compatible with netmap.
>>> Although
>>> you don't need netmap just to capture 10Gb/s. Forwarding, Maybe.
>>>
>>> I also find it interesting that nobody seems to have a handle on the
>>> performance
>>> differences. Obviously they're all different. Maybe substantially
>>> different.
>>
>> It depends on what kind of performance you are talking about.
>> All NICs are capable of doing linerate RX/TX for both small/big packets.
>
> this is actually not true. I have direct experience with Intel,
> Mellanox and Broadcom,
> and small packets are a problem across the board even with 1 port.

>
>  From my experience only intel can do line rate (14.88Mpps) with 64-byte frames,
> but suffers a bit with sizes that are not multiple of 64.
> Mellanox peaks at around 7Mpps.
> Broadcom is limited to some 2.5Mpps.
Wow. So I'm wrong.
However, Chelsio T4 can do at least ~8/port. I'll check for full 
linerate and report.
> This is all with netmap, using the regular stack you are going to see
> much much less.

>
> Large frames (1400+) are probably not a problem for anyone, but since the
> original post asked for packet capture, i thought the small-frame case
> is a relevant one.
>
>> The only notable exception I;m aware of are Intel 82598-based NICs which
>> advertise PCI-E X8 gen2 with _2.5GT_ link speed, giving you maximum
>> ~14Gbit/s bw for 2 ports instead of 20.
>
> This makes me curious because i believe people have used netmap with
> the 82598 and achieved close to line rate even with 64-byte frames/one port,
> and i thought (maybe I am wrong ?) the various 2-port NICs use 4 lanes per port.
> So the number i remember does not match with your quote of 2.5Gt/s.
> Are all 82598 using 2.5GT/s (which is a gen.1 speed) instead of 5 ?
Quoting 82598EB datasheet:
The PCIe v2.0 (2.5 GT/s) interface is used by the 82598EB as a host 
interface. It supports x8, x4, x2 and x1 configurations at a speed of 
2.5 GHz. The maximum aggregated raw ban..

Actually I discovered this exactly with netmap and 82598*-DA2 NIC :)

>
> cheers
> luigi
>