Test on 10GBE Intel based network card

Fri Aug 14 22:15:30 UTC 2009

I've talked over the issues with the guy on our team who has been most
involved in 10G performance, he asserts that 3Gbs will saturate a single
cpu with a small packet size, this is why you need multiqueue across
multiple cores.  He was dubious about the FIFO assertion, its a relative
thing, if you can keep the thing drained it won't be a problem, doing that
is a complex combination of factors, the cpu, the bus, the memory....

If you don't deal with the systemic issues just cuz you go from an 82598
to a 82599 is not going to solve things.

What about LRO, are/can you use that? I never saw an answer about the
forwarding question, you can't use LRO in that case of course.

Regards,

Jack

On Fri, Aug 14, 2009 at 2:33 PM, Carlos Pardo <cpardo at fastsoft.com> wrote:

> Hi Jack,
>
> I have a quick question. We are getting too many missed packets per minute
> running about 3Gbs traffic. We can not use frame control in our application.
> We are assuming that there is no way to improve upon the problem since it
> seems to be a hardware limitation with the receive FIFO. We are using the
> Intel® 82598EB 10 Gigabit Ethernet Controller. When can we expect the next
> generation card from Intel? Thanks for any information you may provide.
>
> Typical error count "ix0: Missed Packets = 81174" after a few minutes.
>
> Best Regards,
>
> Cpardo
>
>
> -----Original Message-----
> From: owner-freebsd-performance at freebsd.org [mailto:
> owner-freebsd-performance at freebsd.org] On Behalf Of Invernizzi Fabrizio
> Sent: Wednesday, August 05, 2009 3:13 AM
> To: Jack Vogel; Julian Elischer
> Cc: freebsd-performance at freebsd.org; Stefan Lambrev
> Subject: RE: Test on 10GBE Intel based network card
>
> No improvement with kern.ipc.nmbclusters=262144 and 1.8.6 driver :<(((((
>
> ++fabrizio
>
> ------------------------------------------------------------------
> Telecom Italia
> Fabrizio INVERNIZZI
> Technology - TILAB
> Accesso Fisso e Trasporto
> Via Reiss Romoli, 274 10148 Torino
> Tel.  +39 011 2285497
> Mob. +39 3316001344
> Fax +39 06 41867287
>
>
> ________________________________
> From: Jack Vogel [mailto:jfvogel at gmail.com]
> Sent: martedì 4 agosto 2009 18.42
> To: Julian Elischer
> Cc: Invernizzi Fabrizio; freebsd-performance at freebsd.org; Stefan Lambrev
> Subject: Re: Test on 10GBE Intel based network card
>
> Your nmbclusters is very low, you list it twice so I'm assuming the second
> value is
> what it ends up being, 32K :(
>
> I would set it to:
>
> kern.ipc.nmbclusters=262144
>
> Also, I thought you were using the current driver, but now it looks like
> you are
> using something fairly old,  use my latest code which is 1.8.8
>
> Jack
>
>
> On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer <julian at elischer.org
> <mailto:julian at elischer.org>> wrote:
> Invernizzi Fabrizio wrote:
> The limitation that you see is about the max number of packets that
> FreeBSD can handle - it looks like your best performance is reached at
> 64 byte packets?
> If you are meaning in term of Packet per second, you are right. These
> are the packet per second measured during tests:
> 64 byte:        610119 Pps
> 512 byte:       516917 Pps
> 1492 byte:      464962 Pps
>
>
> Am I correct that the maximum you can reach is around 639,000 packets
> per second?
> Yes, as you can see the maximum is 610119 Pps.
> Where does this limit come from?
> ah that's the whole point of tuning :-)
> there are severalpossibities:
> 1/ the card's interrupts are probably attache dto aonly 1 cpu,
> so that cpu can do no more work
>
> This seems not to be the problem. See below a top snapshot during a
> 64byte-long packet storm
>
> last pid:  8552;  load averages:  0.40,  0.09,  0.03
>                                                                   up
> 0+20:36:58  09:40:29
> 124 processes: 13 running, 73 sleeping, 38 waiting
> CPU:  0.0% user,  0.0% nice, 86.3% system, 12.3% interrupt,  1.5% idle
> Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free
> Swap: 2048M Total, 2048M Free
>
>  PID USERNAME    THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
>  11 root          1 171 ki31     0K    16K RUN    3  20.2H 51.17% idle:
> cpu3
>  14 root          1 171 ki31     0K    16K RUN    0  20.2H 50.88% idle:
> cpu0
>  12 root          1 171 ki31     0K    16K RUN    2  20.2H 50.49% idle:
> cpu2
>  13 root          1 171 ki31     0K    16K RUN    1  20.2H 50.10% idle:
> cpu1
>  42 root          1 -68    -     0K    16K RUN    1  14:20 36.47% ix0 rxq
>  38 root          1 -68    -     0K    16K CPU0   0  14:15 36.08% ix0 rxq
>  44 root          1 -68    -     0K    16K CPU2   2  14:08 34.47% ix0 rxq
>  40 root          1 -68    -     0K    16K CPU3   3  13:42 32.37% ix0 rxq
> ....
>
> It looks like the 4 rxq processes are bound to the 4 available cores with
> equal distribution.
>
>
>
> 2/ if more than 1 cpu is working, it may be that there is a lock in
> heavy contention somewhere.
>
> This I think is the problem. I am trying to understand how to
> 1- see where the heavy contention is (context switching? Some limiting
> setting?)
> 2- mitigate it
>
>
>
> there ia a lock profiling tool that right now I can't remember the name
> of..
>
> look it up with google :-)  FreeBSD lock profiling tool
>
> ah, first hit...
>
> http://blogs.epfl.ch/article/23832
>
>
>
> is the machine still responsive to other networks while
> running at maximum capacity on this network? (make sure that
> the other networks are on a differnet CPU (hmm I can't remember how to
> do that :-).
>
>
>
> Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle
> persone indicate. La diffusione, copia o qualsiasi altra azione derivante
> dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora
> abbiate ricevuto questo documento per errore siete cortesemente pregati di
> darne immediata comunicazione al mittente e di provvedere alla sua
> distruzione, Grazie.
>
> This e-mail and any attachments is confidential and may contain privileged
> information intended for the addressee(s) only. Dissemination, copying,
> printing or use by anybody else is unauthorised. If you are not the intended
> recipient, please delete this message and any attachments and advise the
> sender by return e-mail, Thanks.
>
> _______________________________________________
> freebsd-performance at freebsd.org<mailto:freebsd-performance at freebsd.org>
> mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "
> freebsd-performance-unsubscribe at freebsd.org<mailto:
> freebsd-performance-unsubscribe at freebsd.org>"
>
> Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle
> persone indicate. La diffusione, copia o qualsiasi altra azione derivante
> dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora
> abbiate ricevuto questo documento per errore siete cortesemente pregati di
> darne immediata comunicazione al mittente e di provvedere alla sua
> distruzione, Grazie.
>
> This e-mail and any attachments is confidential and may contain privileged
> information intended for the addressee(s) only. Dissemination, copying,
> printing or use by anybody else is unauthorised. If you are not the intended
> recipient, please delete this message and any attachments and advise the
> sender by return e-mail, Thanks.
>
> [cid:00000000000000000000000000000001 at TI.Disclaimer]Rispetta l'ambiente.
> Non stampare questa mail se non è necessario.
>
>