ipfw / routing issue on 9.2-RELEASE

[Andreas Nilsson]

On Wed, Mar 26, 2014 at 5:42 PM, Andreas Nilsson <andrnils at gmail.com> wrote:

>  ... snip ...
>
>
>>> I'm wondering what's happening on the outbound path, most of your rules
>>> handle inbound (to kernel) and it seems that rule 65535 deals with most
>>> outbound, except those specifically acting on both paths.
>>>
>> So do I :)
>>
>>>
>>> Maybe try adding to the above:
>>> ipfw add 63510 count log ip from table(1) to any out recv table(8)
>>> and
>>> ipfw add 64510 count log ip from table(1) to any out recv table(8)
>>>
>>> which should catch them on the outbound path before the default rule;
>>> outbound packets have both receive and transmit interfaces available.
>>>
>>> They never get that far :( Those rules log nothing. So I see the packet
>> coming in, triggering the skipto, triggering the count log ... in recv but
>> not the count log ... out.
>>
>>
>>> I guess you're sure that these packets are actually going out to some
>>> external address, not a localhost or alias address (which of course are
>>> received and not sent out anywhere)?
>>>
>> Yes, when the go through they go to external address, which is in the
>> routing table.
>>
>>>
>>> I guess the above new log lines should show the interface (if any)
>>> these packets are leaving on, after routing (maybe a routing issue?)
>>>
>>> Good luck hunting.  If in doubt, throw ever more logging at it! :)  And
>>> please let the list know if you solve it - or not!
>>>
>>> cheers, Ian
>>>
>>
>> To make it even more interesting, it tested this on another machine,
>> where I'm unable to reproduce this "problem". I'm thinking that a reboot
>> might help, but this is kind of fascinating so I would like to actually
>> find the cause. I will investigate further.
>>
>> Best regards
>> Andreas
>>
>
> And now it happened on another machine, but different type of traffic.
> Traffic triggering the divert rule work fine, some traffic not triggering
> the divert rule does not. After everything works as expected.
>
> Before reboot I flushed the firewall so that only default rule was in play
> ( allow all from any to any ), and then *no* traffic for that destination
> went out of the box.
>
> There are something like ~1000 routes in the routing table. Routes are
> added and removed according to some triggers, so during uptime of the
> machine there is a bit of messing with the routing table. At this stage I'm
> at a loss on how to continue investigating this, so I'll just implement the
> forwarding via fwd rules in ipfw and altogether ignore the routing table.
>
> Best regards
> Andreas
>

Ok, found the culprit:

-# $FreeBSD: releng/9.1/etc/rc.d/netif 212579 2010-09-13 19:55:40Z hrs $

+# $FreeBSD: releng/9.2/etc/rc.d/netif 253238 2013-07-12 01:34:24Z hrs $

+ if [ -f /etc/rc.d/routing -a -n "$cmdifn" ] ; then

+ for _if in $cmdifn; do

+  /etc/rc.d/routing start any $_if

+ done
+ fi

etc/rc.d/routing enforces gateway_enable setting in rc.conf which does not
play well with setting net.inet.ip.forwarding=1 in sysctl.conf directly.
Very annoying.

I can find nothing i UPDATING on the subject. Even more annoying.

Best regards
Andreas