Broadcom bge forwarding performance (was em forwarding
performance )
Mike Tancsa
mike at sentex.net
Thu Nov 23 22:14:13 UTC 2006
>>test setup description at
>>http://www.tancsa.com/blast.html
More testing :) This time with a pair of PCIe 1x bge nics, as well
as using the patch at
http://lists.freebsd.org/pipermail/freebsd-net/2006-November/012389.html
As I switched to bge, I could test against Dragonfly as well since
its em driver is broken with the dual port PT1000
Couple of interesting notes. Unlike the em nics, I dont see the box
being overwhelmed with packets to the point of netstat -ni showing
input errors. I dont know if this is just a measurement issue, or
the driver is smarter about throttling back the sender rate with the
switch ? I also noticed that the initial blast, as seen by the
routing box, sees pps rates in around 500k, but then slowly goes down
to about 460k and settles in around that rate but then eventually go
back up towards 500kpps.
26479 0 1588740 2 0 332 0
511592 0 30695562 3 0 386 0
508043 0 30482580 2 0 332 0
508026 0 30481560 2 0 332 0
input (Total) output
packets errs bytes packets errs bytes colls
506675 0 30400628 2 0 332 0
511805 0 30708402 3 0 498 0
477792 0 28667520 2 0 332 0
454527 0 27271620 2 0 332 0
454815 0 27288900 2 0 332 0
453920 0 27235200 2 0 444 0
449027 0 26941620 2 0 332 0
449383 0 26962980 2 0 332 0
461838 0 27710280 2 0 332 0
455068 0 27304080 2 0 332 0
455014 0 27300840 2 0 332 0
411416 0 24684960 4 0 664 0
3 0 180 2 0 332 0
2 0 120 2 0 332 0
4 0 293 2 0 332 0
2 0 120 2 0 332 0
443378 0 26602680 2 0 332 0
455782 0 27346920 2 0 332 0
453432 0 27205920 2 0 332 0
453816 0 27228960 2 0 332 0
450561 0 27033660 2 0 332 0
input (Total) output
packets errs bytes packets errs bytes colls
450317 0 27019020 2 0 332 0
454149 0 27248940 2 0 444 0
With the above patch and from current as of today (including
http://lists.freebsd.org/pipermail/cvs-all/2006-November/197096.html)
the box is *very* responsive even when two blasts of packets are
going in opposite directions!
Here initially is one blast coming from 192.168.88.176 to
192.168.44.1, then I start up another blast from 192.168.44.244 to
192.168.88.206. The box is totally responsive from the management interface
[r2-current]# vmstat -i
interrupt total rate
irq4: sio0 929 0
irq14: ata0 1489 0
irq15: ata1 67 0
irq16: bge1 3158402 1980
irq17: bge0 4918 3
irq19: bge2 2358601 1478
cpu0: timer 3187013 1998
irq256: em0 4 0
irq257: em1 4 0
cpu1: timer 3186657 1997
Total 11898084 7459
[r2-current]#
[r2-current]# ifstat -b
bge0 bge1 bge2
Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out
0.47 1.67 222466.8 0.00 0.00 174265.6
0.94 1.17 233668.8 0.00 0.00 183040.6
0.47 1.17 212013.7 0.00 0.00 166077.4
0.94 1.17 220571.9 0.00 0.00 172781.3
0.47 1.17 240600.0 0.00 0.00 188470.0
0.94 1.17 239261.9 0.00 0.00 187421.8
0.47 1.17 210286.3 0.00 0.00 164724.2
32.20 46.60 236021.0 0.00 0.00 184883.1
9.18 8.46 240039.5 0.00 0.00 188030.9
7.31 7.54 237144.6 0.00 0.00 185763.3
7.59 7.62 231464.9 0.00 0.00 181314.2
0.47 2.47 210767.9 0.00 0.00 165101.5
1.40 2.47 216543.6 0.00 0.00 169625.8
0.94 2.47 213539.4 0.00 0.00 167272.6
1.40 2.47 214488.8 0.00 0.00 168016.2
0.94 2.47 197169.2 21056.33 24296.24 154449.2
1.40 2.47 117003.6 101840.6 117507.9 91652.81
2.20 2.89 119036.2 100521.9 115987.3 93245.06
1.87 2.47 117262.4 101812.5 117475.5 91855.57
0.94 2.47 118018.5 101145.1 116706.3 92447.80
1.40 2.47 116557.3 102412.8 118168.2 91303.23
0.94 2.47 117745.3 101736.3 117388.0 92233.84
1.40 2.47 117146.3 102174.3 117893.4 91764.60
0.94 2.47 116071.5 102631.6 118421.1 90922.66
0.94 2.47 116262.2 102233.0 117961.1 91072.04
0.94 2.47 116043.0 102018.6 117713.2 90899.94
1.87 2.47 115890.5 101801.2 117463.4 90780.86
0.94 2.47 117120.8 101347.2 116939.1 91744.60
1.40 2.47 118051.4 100909.8 116434.4 92473.56
0.94 2.47 118767.3 102499.5 118268.6 93034.42
1.40 2.47 116915.7 101836.6 117503.8 91583.94
1.40 3.34 116517.4 102566.9 118346.0 91271.97
1.73 2.89 116316.7 102750.2 118558.4 91114.77
0.94 2.47 117356.1 101465.1 117074.6 91928.93
1.40 2.47 117906.8 101133.9 116693.0 92360.30
1.82 2.47 116012.6 102530.9 118305.3 90876.51
1.40 2.47 115859.8 103404.1 119311.9 90756.86
0.94 2.47 117364.7 102034.7 117732.3 91935.72
1.40 2.47 117721.1 100830.2 116342.5 92214.86
1.40 2.47 116521.8 102289.6 118026.9 91275.45
0.94 2.47 117061.8 101941.4 117624.2 91698.38
bge0 bge1 bge2
Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out
1.87 2.47 91718.83 119571.1 137967.1 71846.42
0.94 3.47 0.00 182065.9 210075.6 0.00
1.40 2.47 0.00 150718.8 173906.3 0.00
0.47 2.47 0.00 0.00 0.00 0.00
1.40 2.47 0.00 0.00 0.00 0.00
0.94 2.47 0.00 0.00 0.00 0.00
1.40 2.47 0.00 0.00 0.00 0.00
[r2-current]# netstat -ni 1
input (Total) output
packets errs bytes packets errs bytes colls
449655 0 26979300 449622 0 21134055 0
462036 0 27722160 462018 0 21715820 0
457761 0 27465660 457814 0 21514989 0
454539 0 27272340 454446 0 21363555 0
469598 0 28175880 469277 0 22175141 0
501175 0 30070500 501315 0 24809563 0
500932 0 30055920 500938 0 24777688 0
502038 0 30122322 501827 0 24855812 0
501276 0 30076560 501521 0 24804540 0
501378 0 30082680 501418 0 24821941 0
501099 0 30065940 501067 0 24806099 0
502164 0 30129840 501890 0 24862704 0
501911 0 30114660 502006 0 24856474 0
499672 0 29980320 499854 0 24742438 0
499575 0 29974502 499586 0 24738495 0
498042 0 29882520 498027 0 24659622 0
500104 0 30006240 500065 0 24751670 0
500569 0 30034140 500653 0 24768087 0
506240 0 30374400 506113 0 25066903 0
500648 0 30038880 500563 0 24777197 0
input (Total) output
packets errs bytes packets errs bytes colls
501650 0 30099000 501554 0 24845586 0
501332 0 30079962 501550 0 24825352 0
500664 0 30039840 500713 0 24778844 0
501305 0 30078300 501235 0 24812561 0
500147 0 30008873 499945 0 24769644 0
501150 0 30069000 501284 0 24827402 0
503208 0 30192420 503357 0 24909495 0
499578 0 29974740 499528 0 24717105 0
500846 0 30050760 500855 0 24799047 0
501295 0 30077700 501261 0 24825246 0
496749 0 29804940 496919 0 24691116 0
449463 0 26967780 449513 0 23372416 0
427765 0 25665900 427857 0 22244044 0
2 0 120 2 0 316 0
Results along with ipfw rule tests at http://www.tancsa.com/blast.html
I also compared Dragonfly using the bge nics and their behaviour is
quite different as compared to FreeBSD. Out of the box its "slower"
but it keeps its forwarding speed with or without ipfw
loaded. Development kernel results are up there as well and it
behaves much like the other BSDs but is notably faster on the
firewall rule processing.
However, FreeBSD HEAD when in UP mode will live lock with 2 blasts of
packets going in opposite directions. The management interface as
well as the serial console is not responsive at all.
[r2-dragonfly]# netstat -ni
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
bge0 1500 <Link#1> 00:10:18:14:15:12 600 0 232 0 0
bge0 1500 fe80:1::210 fe80:1::210:18ff: 0 - 0 - -
bge0 1500 192.168.43 192.168.43.224 280 - 223 - -
bge1 1500 <Link#2> 00:10:18:14:27:d5 76475440 1271 8606533 0 0
bge1 1500 192.168.88 192.168.88.223 4 - 4 - -
bge1 1500 fe80:2::210 fe80:2::210:18ff: 0 - 0 - -
bge2 1500 <Link#3> 00:10:18:14:38:d2 8631578 2465 76061495 0 0
bge2 1500 192.168.44 192.168.44.223 1 - 1 - -
bge2 1500 fe80:3::210 fe80:3::210:18ff: 0 - 0 - -
More information about the freebsd-performance
mailing list