paket loss on freebsd router if (b)snmpd is running##SPAM

Mon Oct 30 09:34:08 UTC 2006

Hi Thomas,

it seems that 5.4 has the old routing table code. This code used a TAILQ 
to hold all the routes. This turned out to be a problem for large routing 
tables so I replaced it with a red-black tree. This happened between 6.0 
and 6.1 - 6.0 has still the old code, 6.1 the new one. The old code 
consumes a huge amount of memory (I think around 600 byte per route) and a 
huge amount of time, because it needs to loop through the ordered tailq 
for each routing update. It is espacially bad for the initial full read of 
the routing table and the full read that happens periodically (when you do 
an SNMP operation on the routing table), because it builds and ordered 
list from an ordered input set.

Because the daemon opens the routing socket to get routing updates, it 
will do some work even when no SNMP requests come in. I suppose this is 
what is happening in your case. As a first step I would recommend to 
update the bsnmp daemon to at least RELENG_6_1. Updating contrib/bsnmp, 
lib/libbsnmp and usr.sbin/bsnmp and rebuilding/installing first the 
library and then the daemon should do the trick. 6.1 should also have the 
64-bit counters in the interface statistics if I interpret the CVS log 
correctly.

In any case it is still a question why a user space program doing work 
causes the interfaces to loose packets. I don't know enough about the 
POLLING stuff though to make a qualified guess.

harti

On Fri, 27 Oct 2006, Thomas wrote:

T>Hello Harti
T>
T>Harti Brandt schrieb:
T>> On Fri, 27 Oct 2006, Thomas wrote:
T>> 
T>> T>Hello Harti
T>> T>
T>> T>Harti Brandt schrieb:
T>> T>> On Fri, 27 Oct 2006, Thomas wrote:
T>> T>> 
T>> T>> T>Hello
T>> T>> T>
T>> T>> T>I use several 5.4 and 6.1 FBSD machines as router (with quagga). The
T>> T>> T>average traffic is 300mbit/s (em interfaces with polling enabled). It
T>> T>> T>works more or less.
T>> T>> T>
T>> T>> T>Problem:
T>> T>> T>If bsnmpd is running and I'm doing a snmpwalk from a remote machine the
T>> T>> T>router has some significant packet loss. We are talking about 3-15% of
T>> T>> T>packet loss. It's a simple snmpwalk (ipRouteTable and the interfaces stats).
T>> T>> 
T>> T>> Does this happen also if you walk only the interface tables? What size is 
T>> T>> your routing table?
T>> T>
T>> T>The routing table has approx. 200k entries. I've to correct my
T>> T>statement. I don't walk over the RoutingTable.
T>> 
T>> Oh. That would be a nice test how good the RB-tree implementation of the 
T>> routing table is :-)
T>> 
T>> T>It's only an interface table walk (IF-MIB). bsnmpd consumes approx.
T>> T>6-10% cpu. I already have significant packet loss if bsnmpd is started.
T>> T>
T>> T>bsnmpd not running:
T>> T>
T>> T>ping www.switch.ch
T>> T>
T>> T>--- oreius.switch.ch ping statistics ---
T>> T>44 packets transmitted, 44 packets received, 0% packet loss
T>> T>round-trip min/avg/max/stddev = 6.825/7.425/8.007/0.252 ms
T>> T>
T>> T>zero packet loss.
T>> T>
T>> T>
T>> T>netstat -w 1 shows no error.
T>> T>
T>> T>            input        (Total)           output
T>> T>   packets  errs      bytes    packets  errs      bytes colls
T>> T>     39877     0   34330807      38614     0   34087518     0
T>> T>     45522     0   38730142      44124     0   38424881     0
T>> T>     44671     0   38783909      43455     0   38604698     0
T>> T>     43140     0   36939542      42059     0   36691583     0
T>> T>     43428     0   36858817      42213     0   36700954     0
T>> T>     43748     0   37994949      42687     0   37780190     0
T>> T>     42698     0   36451927      41409     0   36270936     0
T>> T>     38695     0   32184588      37119     0   31791417     0
T>> T>
T>> T>
T>> T>
T>> T>
T>> T>bsnmpd started:
T>> T>
T>> T>ping www.switch.ch
T>> T>--- oreius.switch.ch ping statistics ---
T>> T>57 packets transmitted, 53 packets received, 7% packet loss
T>> T>round-trip min/avg/max/stddev = 6.715/12.638/95.183/19.041 ms
T>> T>
T>> T>7% packet loss.
T>> T>
T>> T>
T>> T>netstat shows some errors too.
T>> T>netstat -w 1
T>> T>            input        (Total)           output
T>> T>   packets  errs      bytes    packets  errs      bytes colls
T>> T>     33597     0   27508009      32433     0   27344720     0
T>> T>     38204     0   32117848      36950     0   31852251     0
T>> T>     39171     0   33870984      37999     0   33696346     0
T>> T>     32160     0   26825650      31065     0   26591079     0
T>> T>     26915   726   20902878      25771     0   20791170     0
T>> T>     35167     0   29173014      33945     0   28904867     0
T>> T>     33282   370   27478393      32140     0   27297079     0
T>> T>     29738     0   23859586      28464     0   23524984     0
T>> T>     38456   598   31559055      36881     0   31313396     0
T>> T>     34738     0   28088584      33247     0   27743827     0
T>> T>     38193   648   30854513      36657     0   30556948     0
T>> T>
T>> T>
T>> T>A snmpwalk makes it a bit worse but not much. I already have packet loss
T>> T>if bsnmpd is just started without any walks.
T>> 
T>> You mean, you have loss if it is just ideling around? The only thing 
T>> bsnmpd does in this situation is polling the interface statistics, which 
T>> is needed because the kernel has only 32-bit counters. Could you please 
T>> check what the polling interval is? You find this in the BEGEMOT-MIB2-MIB. 
T>> Something like
T>>
T>
T>Yes, bsnmpd is just ideling.
T>
T>> snmpwalk ... <host> begemotMib2
T>
T>On this freebsd 5.4 machine I have bsnmpd-1.12:
T>share/snmp/mibs/BEGEMOT-MIB.txt
T>share/snmp/mibs/BEGEMOT-NTP-MIB.txt
T>share/snmp/mibs/BEGEMOT-SNMPD.txt
T>
T>but no BEGEMOT-MIB2-MIB.
T>
T>
T>snmpwalk -v 2c -m /usr/share/snmp/mibs/BEGEMOT-MIB.txt -c public
T>localhost BEGEMOT
T>BEGEMOT-MIB::begemot.1.1.1.1.0 = INTEGER: 2048
T>BEGEMOT-MIB::begemot.1.1.1.2.0 = INTEGER: 2048
T>BEGEMOT-MIB::begemot.1.1.1.3.0 = INTEGER: 1
T>BEGEMOT-MIB::begemot.1.1.1.4.0 = IpAddress: 0.0.0.0
T>BEGEMOT-MIB::begemot.1.1.1.5.0 = Gauge32: 3
T>BEGEMOT-MIB::begemot.1.1.2.1.3.192.168.1.10.162 = INTEGER: 1
T>BEGEMOT-MIB::begemot.1.1.2.1.4.192.168.1.10.162 = STRING: "SNTrap"
T>BEGEMOT-MIB::begemot.1.1.2.1.5.192.168.1.10.162 = INTEGER: 2
T>BEGEMOT-MIB::begemot.1.1.4.1.3.127.0.0.1.161 = INTEGER: 1
T>BEGEMOT-MIB::begemot.1.1.4.1.3.192.168.1.100.161 = INTEGER: 1
T>BEGEMOT-MIB::begemot.1.1.6.1.2.5.109.105.98.73.73 = STRING:
T>"/usr/local/lib/snmp_mibII.so"
T>BEGEMOT-MIB::begemot.1.1.6.1.3.5.109.105.98.73.73 = STRING: "This module
T>implements the interface and ip groups."
T>BEGEMOT-MIB::begemot.1.1.7.1.0 = Counter32: 0
T>BEGEMOT-MIB::begemot.1.1.7.2.0 = Counter32: 0
T>BEGEMOT-MIB::begemot.1.1.7.3.0 = Counter32: 0
T>BEGEMOT-MIB::begemot.1.1.7.4.0 = Counter32: 0
T>BEGEMOT-MIB::begemot.1.1.8.1.0 = INTEGER: 2
T>BEGEMOT-MIB::begemot.1.1.8.2.0 = Gauge32: 0
T>BEGEMOT-MIB::begemot.1.1.8.3.0 = INTEGER: 7
T>BEGEMOT-MIB::begemot.1.1.9.1.2.19.47.118.97.114.47.114.117.110.47.115.110.109.112.100.46.115.111.99.107
T>= INTEGER: 1
T>BEGEMOT-MIB::begemot.1.1.9.1.3.19.47.118.97.114.47.114.117.110.47.115.110.109.112.100.46.115.111.99.107
T>= INTEGER: 4
T>BEGEMOT-MIB::begemot.1.1.10.1.1.2.3.117.100.112 = INTEGER: 1
T>BEGEMOT-MIB::begemot.1.1.10.1.1.2.5.108.115.111.99.107 = INTEGER: 1
T>BEGEMOT-MIB::begemot.1.1.10.1.1.3.3.117.100.112 = OID:
T>BEGEMOT-MIB::begemot.1.1.10.2
T>BEGEMOT-MIB::begemot.1.1.10.1.1.3.5.108.115.111.99.107 = OID:
T>BEGEMOT-MIB::begemot.1.1.10.3
T>BEGEMOT-MIB::begemot.3.1.1.1.0 = Counter64: 1000000000
T>BEGEMOT-MIB::begemot.3.1.1.2.0 = Timeticks: (2000) 0:00:20.00
T>BEGEMOT-MIB::begemot.3.1.1.3.0 = Timeticks: (0) 0:00:00.00
T>
T>> given that /usr/share/snmp/mibs is in your MIB path.
T>
T>snmpd.conf:
T>
T># Configuration
T>%snmpd
T>begemotSnmpdDebugDumpPdus       = 2
T>begemotSnmpdDebugSyslogPri      = 7
T>
T>begemotSnmpdCommunityString.0.1 = $(read)
T>begemotSnmpdCommunityString.0.2 = $(write)
T>begemotSnmpdCommunityDisable    = 1
T>
T># open standard SNMP ports
T>begemotSnmpdPortStatus.[$(host)].161 = 1
T>begemotSnmpdPortStatus.127.0.0.1.161 = 1
T>
T># open a unix domain socket
T>begemotSnmpdLocalPortStatus."/var/run/snmpd.sock" = 1
T>begemotSnmpdLocalPortType."/var/run/snmpd.sock" = 4
T>
T># send traps to the traphost
T>begemotTrapSinkStatus.[$(traphost)].$(trapport) = 4
T>begemotTrapSinkVersion.[$(traphost)].$(trapport) = 2
T>begemotTrapSinkComm.[$(traphost)].$(trapport) = $(trap)
T>
T>sysContact      = $(contact)
T>sysLocation     = $(location)
T>sysObjectId     = 1.3.6.1.4.1.12325.1.1.2.1.$(system)
T>
T>snmpEnableAuthenTraps = 2
T>
T>begemotSnmpdModulePath."mibII"  = "/usr/local/lib/snmp_mibII.so"
T>
T>Regards,
T>Thomas
T>
T>
T>