Test on 10GBE Intel based network card
ap00 at mail.ru
Mon Aug 10 08:47:18 UTC 2009
First of all I didn't catch for what task you are tuning.
For forwarding or for transfer?
In any case you should start with minimal configuration (no netgraph
modules, no any firewall).
Choose a test similiar to your typical load.
Provide us with test result, cpu consumption and output of
debug values from driver (sysctl (?).debug)).
Wednesday, August 05, 2009, 2:13:23 PM, you wrote:
IF> No improvement with kern.ipc.nmbclusters=262144 and 1.8.6 driver :<(((((
IF> Telecom Italia
IF> Fabrizio INVERNIZZI
IF> Technology - TILAB
IF> Accesso Fisso e Trasporto
IF> Via Reiss Romoli, 274 10148 Torino
IF> Tel. +39 011 2285497
IF> Mob. +39 3316001344
IF> Fax +39 06 41867287
IF> From: Jack Vogel [mailto:jfvogel at gmail.com]
IF> Sent: martedì 4 agosto 2009 18.42
IF> To: Julian Elischer
IF> Cc: Invernizzi Fabrizio; freebsd-performance at freebsd.org; Stefan Lambrev
IF> Subject: Re: Test on 10GBE Intel based network card
IF> Your nmbclusters is very low, you list it twice so I'm assuming the second value is
IF> what it ends up being, 32K :(
IF> I would set it to:
IF> Also, I thought you were using the current driver, but now it looks like you are
IF> using something fairly old, use my latest code which is 1.8.8
IF> On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer <julian at elischer.org<mailto:julian at elischer.org>> wrote:
IF> Invernizzi Fabrizio wrote:
IF> The limitation that you see is about the max number of packets that
IF> FreeBSD can handle - it looks like your best performance is reached at
IF> 64 byte packets?
IF> If you are meaning in term of Packet per second, you are right. These
IF> are the packet per second measured during tests:
IF> 64 byte: 610119 Pps
IF> 512 byte: 516917 Pps
IF> 1492 byte: 464962 Pps
IF> Am I correct that the maximum you can reach is around 639,000 packets
IF> per second?
IF> Yes, as you can see the maximum is 610119 Pps.
IF> Where does this limit come from?
IF> ah that's the whole point of tuning :-)
IF> there are severalpossibities:
IF> 1/ the card's interrupts are probably attache dto aonly 1 cpu,
IF> so that cpu can do no more work
IF> This seems not to be the problem. See below a top snapshot during a 64byte-long packet storm
IF> last pid: 8552; load averages: 0.40, 0.09, 0.03 up 0+20:36:58 09:40:29
IF> 124 processes: 13 running, 73 sleeping, 38 waiting
IF> CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle
IF> Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free
IF> Swap: 2048M Total, 2048M Free
IF> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
IF> 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: cpu3
IF> 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: cpu0
IF> 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: cpu2
IF> 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: cpu1
IF> 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq
IF> 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq
IF> 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq
IF> 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq
IF> It looks like the 4 rxq processes are bound to the 4 available cores with equal distribution.
IF> 2/ if more than 1 cpu is working, it may be that there is a lock in
IF> heavy contention somewhere.
IF> This I think is the problem. I am trying to understand how to
IF> 1- see where the heavy contention is (context switching? Some limiting setting?)
IF> 2- mitigate it
IF> there ia a lock profiling tool that right now I can't remember the name of..
IF> look it up with google :-) FreeBSD lock profiling tool
IF> ah, first hit...
IF> is the machine still responsive to other networks while
IF> running at maximum capacity on this network? (make sure that
IF> the other networks are on a differnet CPU (hmm I can't remember how to
IF> do that :-).
Anthony mailto:ap00 at mail.ru
More information about the freebsd-performance