problem analysys (Re: [Panic] Dummynet/IPFW related recurring crash.)

[Pawel Tyll]

> The way a problem is presented has a big impact on how it gets handled:
> in this specific case the poster is pointing out a possible culprit
> (which may be helpful or misleading), and gives no hint on other
> things that may be relevant: number of interfaces, vlans, tunnels, taps,
> bpf etc ? any significant other activity on the machine such as interfaces
> going up or down, routing deamons etc ? amount of traffic ?
> Without furter details, I can only trust the statements in the report,
> and this determines how i classify the bug and decide whether i have time
> or ideas to pursue it.

> The bug in this case seems to fall in the 'hard, irreproducible' category:
> panics *always* need many many days to happen on machines under heavy load,
> no panics on similar machines under lighter load.
> With a description like this, i am afraid, i can't even start looking
> at the problem becaue i have no chance to reproduce it.
This machine is only doing dummynet traffic shaping from significant
things (otherwise it runs a dhcpd, ntpd and named). It's pretty
straight-forward routing, packets come in, packets come out via static
routes - there are currently no routing daemons involved. As to the
interfaces, there are two physical ifaces, em0 and em1, only em1 is
currently used. There are 49 vlan interfaces connected to em1, and
they are pretty much static, no IP address changes, no interfaces
going up or down, sometimes new one is being added, but there is no
automation here, and panics do not coincide with anything significant
in logs, or being done manually. Traffic oscillates between 20k pps at
night and close to 35-40k pps daytime, slightly more on weekends.
There are currently 2556 pipes defined and traffic shaping is done
with two rules:

30000 pipe tablearg ip from table(100) to any in
30001 pipe tablearg ip from any to table(101) out

Currently running FreeBSD 8.2-PRERELEASE #1: Fri Jan  7 17:19:28 CET
2011, but problem persists since 8.0-RELEASE

net.inet.flowtable.enable=0
net.inet.ip.fw.one_pass=0
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.ip.random_id=1
net.inet.ip.forwarding=1
kern.ipc.maxsockbuf=8388608
net.inet.tcp.sendspace=4194304
net.inet.tcp.recvspace=4194304
net.inet.udp.recvspace=1048576
net.inet.udp.maxdgram=65536
net.inet.ip.fw.dyn_buckets=16384
net.inet.ip.fw.dyn_max=65536

hw.machine: amd64
hw.model: Intel(R) Xeon(R) CPU           X3440  @ 2.53GHz
hw.ncpu: 8
hw.byteorder: 1234
hw.physmem: 4240433152
hw.usermem: 2734768128

Base Board Information
        Manufacturer: Intel Corporation
        Product Name: S3420GP

em1: <Intel(R) PRO/1000 Network Connection 7.1.8> port 0x1000-0x101f mem 0xb1900000-0xb191ffff,0xb1920000-0xb1923fff irq 16 at device 0.0 on pci2
em1: Using MSIX interrupts with 3 vectors

If I missed anything here, then just tell me what more I can do, my
intentions were never to make this harder to debug or hide anything
relevant.