problem analysys (Re: [Panic] Dummynet/IPFW related recurring crash.)

[Luigi Rizzo]

On Mon, Feb 21, 2011 at 12:13:12AM +0100, Pawel Tyll wrote:
> > understood. I am just saying that for instance the vlan presence and
> > changes is  quite significant in this context.
> > You say vlans are "pretty much static" but can you tell us who adds/remove
> > them, assign addresses ?
> It's not that much work and changes are simple and far between. I do
> that personally. IP addresses don't change, however I sometimes
> (rarely) destroy and recreate vlans. Panics don't happen immediately
> after this operation, or while it happens, and there were times from
> panic to panic that I didn't touch a thing.
> 
> > Also the ruleset must have something more than those two rules.
> > From the stack trace, the panic seems to occur in a call to the
> > "antispoof" option which presumably is somewhere in your ruleset.
> > If not, then the stack is corrupt.
> Full ruleset with IP addresses removed:
> 00010       1691        128516 deny ip from any to any not antispoof in
> 00020   87440010    6826835332 fwd [removed] ip from table(60) to table(61)
> 00050       3246        156244 allow tcp from any to [removed] dst-port 53 // DNS Rules 50-59
> 00051    2463493     260607132 allow udp from any to [removed] // DNS Rules 50-59
> 00059      23891       1091822 deny ip from any to [removed] // DNS Rules 50-59
> 00100         32          2176 allow ip from any to any via lo0
> 00100     929493      48342523 deny ip from any to table(10) dst-port 131-139,445
> 00102      56574       2779124 fwd [removed] tcp from table(1) to not table(5) dst-port 80
> 00103          0             0 fwd [removed] tcp from table(2) to not table(5) dst-port 80
> 00104        427         17244 fwd [removed] tcp from table(3) to not table(5)
> 00105          6           808 deny ip from table(3) to not table(5)
> 00200          0             0 deny ip from any to 127.0.0.0/8
> 00300          0             0 deny ip from 127.0.0.0/8 to any
> 00400          0             0 deny ip from any to ::1
> 00500          0             0 deny ip from ::1 to any
> 00600          0             0 allow ipv6-icmp from :: to ff02::/16
> 00700          0             0 allow ipv6-icmp from fe80::/10 to fe80::/10
> 00800          0             0 allow ipv6-icmp from fe80::/10 to ff02::/16
> 00900          0             0 allow ipv6-icmp from any to any ip6 icmp6types 1
> 01000          0             0 allow ipv6-icmp from any to any ip6 icmp6types 2,135,136
> 30000  462392089  204487140826 pipe tablearg ip from table(100) to any in
> 30001  535282183  461888428313 pipe tablearg ip from any to table(101) out
> 34900   11650783    1216622001 skipto 35001 ip from table(10) to table(10)
> 35000  597825867  244960831012 fwd [removed] ip from 192.168.0.0/16 to not 192.168.0.0/16
> 65534 1595697378 1254723485778 allow ip from any to any
> 65535          0             0 allow ip from any to any
> 
> 12:07AM  up 1 day, 21 mins, 1 user, load averages: 0.08, 0.06, 0.01
> 
> Should IP addresses be required, I'll gladly send "uncensored" ruleset
> to you privately.

addresses not needed, thanks. From what i saw in the backtrace, the panic
occurred on an incoming packet on the 'antispoof' option.
The ruleset confirms the backtrace, but since
'antispoof' happens
to be run on every packet given it is on the first rule,
it apparently has nothing to do with dummynet because even if
you reinjected the packets, they go to rule 34900.

So, i'd still focus the attention on a corrupt interface list.

Sure, that memory can be corrupt by anything including dummynet,
but there is no reason to believe that dummynet is more likely
than other subsystems to cause the breakage.

Unfortunately i don't think I can be of more help.

cheers
luigi