Freebsd IP Forwarding performance (question, and some info)
[7-stable, current, em, smp]
Paul
paul at gtcomm.net
Sun Jun 29 18:29:30 UTC 2008
Other interesting behavior:
netstat -rs hitting up arrow and enter to get repeated views of it:
[root at irc-router1 /usr/src/sys/net]# netstat -rs
routing:
0 bad routing redirects
0 dynamically created routes
0 new gateways due to redirects
25691 destinations found unreachable
0 uses of a wildcard route
0 routes not in table but not freed
[root at router1 /usr/src/sys/net]# netstat -rs
routing:
0 bad routing redirects
0 dynamically created routes
0 new gateways due to redirects
27879 destinations found unreachable
0 uses of a wildcard route
0 routes not in table but not freed
[root at router1 /usr/src/sys/net]# netstat -rs
routing:
0 bad routing redirects
0 dynamically created routes
0 new gateways due to redirects
30256 destinations found unreachable
0 uses of a wildcard route
0 routes not in table but not freed
[root at router1 /usr/src/sys/net]# netstat -rs
routing:
0 bad routing redirects
0 dynamically created routes
0 new gateways due to redirects
32699 destinations found unreachable
0 uses of a wildcard route
0 routes not in table but not freed
[root at router1 /usr/src/sys/net]# netstat -rs
routing:
0 bad routing redirects
0 dynamically created routes
0 new gateways due to redirects
4294936832 destinations found unreachable
0 uses of a wildcard route
0 routes not in table but not freed
[root at router1 /usr/src/sys/net]# netstat -rs
routing:
0 bad routing redirects
0 dynamically created routes
0 new gateways due to redirects
4294941139 destinations found unreachable
0 uses of a wildcard route
0 routes not in table but not freed
[root at router1 /usr/src/sys/net]#
after 32768 it jumps to close to a 32 bit number limit and then after
adding a few more it goes back to zero..
This doesn't happen on 7.0-RELEASE but it happens on 7.0-STABLE and -CURRENT
Whatever is generating the RTM_MISS is incrementing this also.
There's a lot of differences in the /net dir from release to stable but
most are in bridging/gif/lagg etc.
Route.c the only difference is
-RELEASE
+STABLE
- rtfree(rt);
+ RTFREE_LOCKED(rt);
Must be generating it from somewhere else?
Paul wrote:
>
> [RTM_MISS information added]
>
> I have noticed something weird.. It doesn't generate the RTM_MISS with
> all traffic...
> Check this out..
> flooding packets through the router
> 11:29:56.147177 IP (tos 0x0, ttl 255, id 51487, offset 0, flags
> [none], proto TCP (6), length 40) 87.42.160.8.8195 > 10.3.9.50.623: .,
> cksum 0x999e (incorrect (-> 0xb7e6), 2714784062:2714784062(0) ack
> 1638282847 win 16384
> 11:29:56.147182 IP (tos 0x0, ttl 255, id 22197, offset 0, flags
> [none], proto TCP (6), length 40) 43.253.49.61.13857 > 10.3.9.50.6232:
> ., cksum 0x51b4 (incorrect (-> 0xeb52), 2685378913:2685378913(0) ack
> 3576016642 win 16384
> 11:29:56.147186 IP (tos 0x0, ttl 255, id 19160, offset 0, flags
> [none], proto TCP (6), length 40) 148.198.237.1.31784 > 10.3.9.50.63:
> ., cksum 0x7d2b (incorrect (-> 0xcedf), 2790811715:2790811715(0) ack
> 3733955172 win 16384
> 11:29:56.147190 IP (tos 0x0, ttl 255, id 45483, offset 0, flags
> [none], proto TCP (6), length 40) 50.140.153.13.23035 > 10.3.9.50.324:
> ., cksum 0x5e56 (incorrect (-> 0xdb81), 2403102209:2403102209(0) ack
> 923149825 win 16384
> 11:29:56.147194 IP (tos 0x0, ttl 255, id 53653, offset 0, flags
> [none], proto TCP (6), length 40) 215.20.25.100.18066 >
> 10.3.9.50.6232: ., cksum 0x592b (incorrect (-> 0x9f26),
> 3678810149:3678810149(0) ack 916597768 win 16384
> 11:29:56.147198 IP (tos 0x0, ttl 255, id 25330, offset 0, flags
> [none], proto TCP (6), length 40) 30.216.125.43.23715 >
> 10.3.9.50.5325: ., cksum 0x7d62 (incorrect (-> 0xdbb8),
> 4058239037:4058239037(0) ack 1225812033 win 16384
> 11:29:56.147210 IP (tos 0x0, ttl 255, id 56031, offset 0, flags
> [none], proto TCP (6), length 40) 92.202.56.54.1180 > 10.3.9.50.623:
> ., cksum 0x45fb (incorrect (-> 0xc35d), 1051146817:1051146817(0) ack
> 1914442290 win 16384
> 11:29:56.147214 IP (tos 0x0, ttl 255, id 17414, offset 0, flags
> [none], proto TCP (6), length 40) 243.139.107.120.22763 >
> 10.3.9.50.63: ., cksum 0x3f12 (incorrect (-> 0x983d),
> 343363109:343363109(0) ack 2790458180 win 16384
> 11:29:56.147219 IP (tos 0x0, ttl 255, id 15785, offset 0, flags
> [none], proto TCP (6), length 40) 193.30.160.14.19071 > 10.3.9.50.324:
> ., cksum 0x0021 (incorrect (-> 0x3f33), 3909194617:3909194617(0) ack
> 1335272554 win 16384
> 11:29:56.147223 IP (tos 0x0, ttl 255, id 36379, offset 0, flags
> [none], proto TCP (6), length 40) 93.46.193.34.22779 > 10.3.9.50.5325:
> ., cksum 0x2f95 (incorrect (-> 0x2fb6), 1506935083:1506935083(0) ack
> 2882181640 win 16384
> 11:29:56.147228 IP (tos 0x0, ttl 255, id 43639, offset 0, flags
> [none], proto TCP (6), length 40) 50.78.186.1.1437 > 10.3.9.50.623: .,
> cksum 0xba44 (incorrect (-> 0xe9d9), 3585077271:3585077271(0) ack
> 1754157076 win 16384
> 11:29:56.147232 IP (tos 0x0, ttl 255, id 31282, offset 0, flags
> [none], proto TCP (6), length 40) 230.61.232.98.9799 > 10.3.9.50.6232:
> ., cksum 0xe26a (incorrect (-> 0x9caf), 2184338509:2184338509(0) ack
> 1881236495 win 16384
> 11:29:56.147242 IP (tos 0x0, ttl 255, id 18266, offset 0, flags
> [none], proto TCP (6), length 40) 185.133.224.35.18694 > 10.3.9.50.63:
> ., cksum 0x5bb9 (incorrect (-> 0x3e24), 1265308989:1265308989(0) ack
> 898540886 win 16384
> 11:29:56.147247 IP (tos 0x0, ttl 255, id 63295, offset 0, flags
> [none], proto TCP (6), length 40) 60.149.107.97.14907 > 10.3.9.50.324:
> ., cksum 0x75ac (incorrect (-> 0xd165), 680418413:680418413(0) ack
> 1399770970 win 16384
> 11:29:56.147264 IP (tos 0x0, ttl 255, id 44265, offset 0, flags
> [none], proto TCP (6), length 40) 226.37.11.125.16380 >
> 10.3.9.50.5325: ., cksum 0xb763 (incorrect (-> 0x2d10),
> 1640245563:1640245563(0) ack 3534655349 win 16384
> 11:29:56.147276 IP (tos 0x0, ttl 255, id 62142, offset 0, flags
> [none], proto TCP (6), length 40) 76.66.176.108.8173 > 10.3.9.50.623:
> ., cksum 0xf66b (incorrect (-> 0xadcf), 1200243821:1200243821(0) ack
> 398846984 win 16384
> 11:29:56.147281 IP (tos 0x0, ttl 255, id 60663, offset 0, flags
> [none], proto TCP (6), length 40) 241.35.125.100.4600 > 10.3.9.50.324:
> ., cksum 0x7299 (incorrect (-> 0x63e8), 4145462333:4145462333(0) ack
> 1113096518 win 16384
> 11:29:56.147286 IP (tos 0x0, ttl 255, id 55417, offset 0, flags
> [none], proto TCP (6), length 40) 152.7.87.5.14990 > 10.3.9.50.6232:
> ., cksum 0xd955 (incorrect (-> 0xcfc1), 892720934:892720934(0) ack
> 1334374150 win 16384
> ^C11:29:56.147290 IP (tos 0x0, ttl 255, id 29468, offset 0, flags
> [none], proto TCP (6), length 40) 246.253.188.115.23425 >
> 10.3.9.50.63: ., cksum 0xf14e (incorrect (-> 0xcaa4),
> 1030196069:1030196069(0) ack 2661620567 win 16384
>
> these generate RTM_MISS for what looks like every packet :
>
> got message of size 160 on Sun Jun 29 11:31:04 2008
> RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno
> 0, flags:<DONE>
> locks: inits:
> sockaddrs: <DST>
> default
>
> got message of size 160 on Sun Jun 29 11:31:04 2008
> RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno
> 0, flags:<DONE>
> locks: inits:
> sockaddrs: <DST>
> default
>
> got message of size 160 on Sun Jun 29 11:31:04 2008
> RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno
> 0, flags:<DONE>
> locks: inits:
> sockaddrs: <DST>
> default
>
> got message of size 160 on Sun Jun 29 11:31:04 2008
> RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno
> 0, flags:<DONE>
> locks: inits:
> sockaddrs: <DST>
> default
>
> got message of size 160 on Sun Jun 29 11:31:04 2008
> RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno
> 0, flags:<DONE>
> locks: inits:
> sockaddrs: <DST>
> default
>
> got message of size 160 on Sun Jun 29 11:31:04 2008
> RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno
> 0, flags:<DONE>
> locks: inits:
> sockaddrs: <DST>
> default
>
> got message of size 160 on Sun Jun 29 11:31:04 2008
> RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno
> 0, flags:<DONE>
> locks: inits:
> sockaddrs: <DST>
> default
>
>
> And then these packets:
>
> 11:32:50.540714 IP (tos 0x0, ttl 128, id 60717, offset 0, flags [DF],
> proto TCP (6), length 48) 199.253.132.217.1145 > 10.3.9.50.1230: S,
> cksum 0xe22a (correct), 2254729531:2254729531(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540719 IP (tos 0x0, ttl 128, id 29798, offset 0, flags [DF],
> proto TCP (6), length 48) 128.201.66.139.1074 > 10.3.9.50.1076: S,
> cksum 0xbc56 (correct), 699104812:699104812(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540737 IP (tos 0x0, ttl 128, id 24548, offset 0, flags [DF],
> proto TCP (6), length 48) 64.6.56.150.1137 > 10.3.9.50.1065: S, cksum
> 0x3d9f (correct), 3231494262:3231494262(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540742 IP (tos 0x0, ttl 128, id 24872, offset 0, flags [DF],
> proto TCP (6), length 48) 209.236.188.202.1119 > 10.3.9.50.1137: S,
> cksum 0xb4bb (correct), 3403159757:3403159757(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540746 IP (tos 0x0, ttl 128, id 7860, offset 0, flags [DF],
> proto TCP (6), length 48) 193.42.23.202.1147 > 10.3.9.50.1047: S,
> cksum 0x7900 (correct), 2067225130:2067225130(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540751 IP (tos 0x0, ttl 128, id 6369, offset 0, flags [DF],
> proto TCP (6), length 48) 198.222.98.165.1070 > 10.3.9.50.1197: S,
> cksum 0x8245 (correct), 1566580196:1566580196(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540759 IP (tos 0x0, ttl 128, id 29494, offset 0, flags [DF],
> proto TCP (6), length 48) 194.32.233.216.1217 > 10.3.9.50.1035: S,
> cksum 0x0036 (correct), 608655014:608655014(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540772 IP (tos 0x0, ttl 128, id 57975, offset 0, flags [DF],
> proto TCP (6), length 48) 205.206.20.208.1220 > 10.3.9.50.1232: S,
> cksum 0x52fc (correct), 1339728095:1339728095(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540777 IP (tos 0x0, ttl 128, id 5551, offset 0, flags [DF],
> proto TCP (6), length 48) 4.42.66.89.1142 > 10.3.9.50.1178: S, cksum
> 0xaa57 (correct), 1309337587:1309337587(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540781 IP (tos 0x0, ttl 128, id 64726, offset 0, flags [DF],
> proto TCP (6), length 48) 208.166.197.140.1052 > 10.3.9.50.1084: S,
> cksum 0x4dfd (correct), 3514659298:3514659298(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540786 IP (tos 0x0, ttl 128, id 51126, offset 0, flags [DF],
> proto TCP (6), length 48) 206.67.34.143.1071 > 10.3.9.50.1100: S,
> cksum 0xbf84 (correct), 3369643581:3369643581(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540790 IP (tos 0x0, ttl 128, id 59088, offset 0, flags [DF],
> proto TCP (6), length 48) 210.246.25.177.1277 > 10.3.9.50.1205: S,
> cksum 0x7080 (correct), 1667720615:1667720615(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540795 IP (tos 0x0, ttl 128, id 2326, offset 0, flags [DF],
> proto TCP (6), length 48) 129.251.33.231.1154 > 10.3.9.50.1195: S,
> cksum 0x28e5 (correct), 312297303:312297303(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540799 IP (tos 0x0, ttl 128, id 8268, offset 0, flags [DF],
> proto TCP (6), length 48) 216.84.213.152.1115 > 10.3.9.50.1027: S,
> cksum 0x7a81 (correct), 2232908292:2232908292(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540803 IP (tos 0x0, ttl 128, id 15857, offset 0, flags [DF],
> proto TCP (6), length 48) 199.228.246.182.1053 > 10.3.9.50.1053: S,
> cksum 0xe187 (correct), 376795414:376795414(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540808 IP (tos 0x0, ttl 128, id 33924, offset 0, flags [DF],
> proto TCP (6), length 48) 128.92.244.80.1060 > 10.3.9.50.1148: S,
> cksum 0xc4e1 (correct), 2038527032:2038527032(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540812 IP (tos 0x0, ttl 128, id 12114, offset 0, flags [DF],
> proto TCP (6), length 48) 64.118.145.169.1252 > 10.3.9.50.1148: S,
> cksum 0xb270 (correct), 3489780214:3489780214(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540816 IP (tos 0x0, ttl 128, id 23385, offset 0, flags [DF],
> proto TCP (6), length 48) 211.212.238.186.1217 > 10.3.9.50.1083: S,
> cksum 0xc082 (correct), 2887120836:2887120836(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540821 IP (tos 0x0, ttl 128, id 10979, offset 0, flags [DF],
> proto TCP (6), length 48) 24.119.38.75.1088 > 10.3.9.50.1027: S, cksum
> 0xee10 (correct), 3079816000:3079816000(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540830 IP (tos 0x0, ttl 128, id 44562, offset 0, flags [DF],
> proto TCP (6), length 48) 193.99.52.25.1249 > 10.3.9.50.1047: S, cksum
> 0x9ef4 (correct), 1263552303:1263552303(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540835 IP (tos 0x0, ttl 128, id 45332, offset 0, flags [DF],
> proto TCP (6), length 48) 209.168.140.136.1275 > 10.3.9.50.1053: S,
> cksum 0x03b5 (correct), 1521904180:1521904180(0) win 16384 <mss
> 1460,nop,[bad opt]>
> 11:32:50.540840 IP (tos 0x0, ttl 128, id 10244, offset 0, flags [DF],
> proto TCP (6), length 48) 198.114.145.88.1214 > 10.3.9.50.1083: S,
> cksum 0x51a9 (correct), 2907689003:2907689003(0) win 16384 <mss
> 1460,nop,[bad opt]>
>
> I get no RTM MISS. So this absoutely makes no sense?? It should not
> depend on the TYPE of packet at ALL. It's just forwarding IP
> regardless of what's in the TCP header so why on earth would the tcp
> header many any difference to how it routes.. It should not even be
> looking at the tcp header unless I have a firewall turned on or any
> type of rules that would filter based on tcp options. Forwarding a
> packet doesn't require looking at TCP option bits or even TCP port
> numbers. I suppose it could be checking the mss or trying to
> calculate checksums but I would prefer it didn't even do that :/ Raw
> performance is all I care about at this point, I don't even need to
> use any firewall at all.
> It's also funny to notice, that tcpdump does not see those packets as
> TCP..
> tcpdump -n -i em0 tcp
> does not show either of my 'floods'
> tcpdump -n -i em1 tcp
> shows my floods
> tcpdump -n -i em0
> shows them all.. and they say TCP ..lol :) Weird I tell ya!
>
> So the bottom line is, why do one type of packet generate RTM_MISS and
> another doesn't?
> I haven't confirmed this, but I don't think it does it on 7.0-RELEASE,
> only 7.0-STABLE and CURRENT.
> I will try and confirm though.
>
> Paul
>
>
>
> Mike Tancsa wrote:
>> At 04:04 AM 6/29/2008, Paul wrote:
>>> This is just a question but who can get more than 400k pps
>>> forwarding performance ?
>>> I have tested fbsd 6/7/8 so far with many different configs. (all
>>> using intel pci-ex nic and SMP)
>>> fbsd 7-stable/8(current) seem to be the fastest and always hit this
>>> ceiling of 400k pps. Soon as it hits that I get errors galore.
>>
>> In your testing of current, or even RELENG_7, did you ever solve the
>> problem of
>>
>> http://www.freebsd.org/cgi/query-pr.cgi?pr=123621&cat=kern
>> that you also ran into in a previous thread ? I wonder if the
>> generation of all those seemingly bogus RTM_MISS messages is
>> hampering performance.
>>
>> ---Mike
>> _______________________________________________
>> freebsd-net at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>
>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>
More information about the freebsd-net
mailing list