PF suddenly malfunctioned

[Jason Mattax]

On 07/22/2012 07:30 PM, Damien Fleuriot wrote:
>
> On 23 Jul 2012, at 01:49, jmattax at clanspum.net wrote:
>
>> A few weeks ago (I've been trying to debug it myself since then) my pf
>> firewall stopped working fully correctly. The symptom is that I can no longer
>> access a variety of websites when I'm behind the firewall. I have verified
>> that I can access all of the affected websites from outside my firewall. I
>> have since stripped down my firewall (and general home server) so that it is
>> no longer running named, sshguard or any useful firewalling rules in an
>> attempt to figure out was broken but have been unable to do so.
>>
>> Attached are my current /etc/pf.conf and /etc/rc.conf, to ensure that these
>> are the configurations being used as of my last test I restarted the system
>> and am still getting the same behavior. This behavior started sometime around
>> a storm at my house, but since the firewall can see the websites that the
>> computers behind it can't I don't believe the hardware is an issue.
>>
>> Also, some websites (like anything google hosts) are just fine.
>>
>> The also, so people can see what my kernel thinks I've attach the output of a
>> couple of commands below
>>
>> [root@ ~]# pfctl -s rules
>> No ALTQ support in kernel
>> ALTQ related functions disabled
>> pass in quick all flags S/SA keep state
>> pass out quick all flags S/SA keep state
>> [root@ ~]# pfctl -s nat
>> No ALTQ support in kernel
>> ALTQ related functions disabled
>> nat on xl0 inet from 10.11.10.0/24 to any -> 192.168.0.200
>> [root at stilgar ~]# ifconfig
>> re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>         options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
>>         ether 90:e6:ba:60:9a:33
>>         inet 10.11.10.1 netmask 0xffffff00 broadcast 10.11.10.255
>>         media: Ethernet autoselect (100baseTX <full-duplex>)
>>         status: active
>> xl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>         options=82009<RXCSUM,VLAN_MTU,WOL_MAGIC,LINKSTATE>
>>         ether 00:01:03:d1:fa:90
>>         inet 192.168.0.200 netmask 0xffffff00 broadcast 192.168.0.255
>>         media: Ethernet autoselect (100baseTX
>> <full-duplex,flowcontrol,rxpause,txpause>)
>>         status: active
>> plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
>> ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
>> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>>         options=3<RXCSUM,TXCSUM>
>>         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
>>         inet6 ::1 prefixlen 128
>>         inet 127.0.0.1 netmask 0xff000000
>>         nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
>> pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33152
>>
>> I would be very appreciative of any suggestions anyone can offer.
>>
>>      Jason Mattax
>>
>
> 1/ OS version ? We can't tell from the current info
>
[jmattax@ ~]$ uname -a
FreeBSD hostname 8.2-RELEASE-p9 FreeBSD 8.2-RELEASE-p9 #0: Mon Jun 11 
23:00:11 UTC 2012 
root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

based on that I could easily upgrade to 8.3, or possibly 9.0 tomorrow if 
I have the inclination.

> 2/ When the problem appears. Have you tried disabling PF ? (pfctl -d)
> Does it help ?
>
Since I can consistently reproduce the problem with en.wikipedia.org I 
have a good way to test. When I run pfctl -d on the firewall it looks 
like no traffic is being forwarded, including DNS so I eventually get a 
notice that the web page timed out because I typed the address wrong. 
That is as opposed to the web browser saying waiting for 
en.wikipedia.org (and if I recall correctly occasionally getting the 
redirect to en.wikipedia.org/wiki/Main_Page.) I just tested and got 
stuck at the waiting for en.wikipedia.org for a couple of minutes before 
I called it good enough to report here.

> 3/ The websites wouldn't be using connection recycling per chance ? (linux)
> We've had a lot of problems with Linux enabled hosts using recycling, having them turn it off solved the problems.
> There was not a thing we found on our side to fix it.
> Disabling scrubbing wouldn't help either.

To be clear wikipedia was working with a full firewall configuration, so 
although I believe they are hosted on linux I would hope someone else 
would also see this problem. I know there are other websites that also 
became broken around the same time, but because they are largely 
advertisement hosting websites I don't know their names off hand and 
have been bypassing the firewall for the moment.

Thanks
	Jason