pf rules not being loaded during boot on 7.1-PRERELEASE

[Jeremy Chadwick]

On Fri, Oct 03, 2008 at 04:17:03AM -0700, Jeremy Chadwick wrote:
> On Thu, Oct 02, 2008 at 09:57:55PM +0100, Bruce Cran wrote:
> > I recently upgraded my i386 router from 7.0 to 7.1-PRERELEASE.  I  
> > rebooted it today but despite pf_enable="YES" being in /etc/rc.conf no  
> > rules got loaded during boot, despite pf itself having been enabled:
> >
> > router# pfctl -s rules
> > router# pfctl -e -f /etc/pf.conf
> > pfctl: pf already enabled
> > [connection is closed due to new rules being loaded]
> > router# pfctl -s rules
> > scrub in all fragment reassemble
> > [... lots of rules listed]
> >
> > Has anyone else seen this problem, or have I just missed something  
> > that's changed between 7.0 and 7.1 in the way pf works?
> 
> I was seeing something similar on my own box which I just upgraded from
> a 150-day-old RELENG_6 to present RELENG_6.  pfctl -s rules output no
> rules.  pfctl -s info showed packet counters, but no interface stats
> (due to the rules not being loaded, e.g. no loginterface).
> 
> kldstat showed pflog.ko and pf.ko loaded.
> 
> If I did /etc/rc.d/pf start, the rules would loaded, and everything
> starts working as expected.
> 
> I rebooted the box and saw the following on serial console, which I'm
> pretty sure is what's responsible for the breakage:
> 
> Enabling pf.
> Oct  3 04:14:51 pflogd[374]: [priv]: msg PRIV_OPEN_LOG received
> cannot determine interface bandwidth for bge0, specify an absolute
> bandwidth
> altq not defined on bge0
> altq not defined on bge0
> /conf/ME/pf.conf:52: errors in queue definition
> altq not defined on bge0
> /conf/ME/pf.conf:53: errors in queue definition
> altq not defined on bge0
> /conf/ME/pf.conf:54: errors in queue definition
> pfctl: Syntax error in config file: pf rules not loaded
> pf enabled

Cross-posting to freebsd-pf (I'm sorry for doing this, but it needs
attention from both -pf and -stable).

I've figured out what the problem is.  This is not good, and is
guaranteed to bite other people.  I'd like to believe this is an
rc-related problem, but I'm not sure how to fix it.

The problem in my case:

The physical interfaces were brought online, but were still technically
offline (the switch and NIC PHY were taking some time to negotiate speed
and duplex).  Boot messages:

bge0: link state changed to DOWN
bge1: link state changed to DOWN
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
        inet 127.0.0.1 netmask 0xff000000
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=1b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING>
        inet XXXXXXXXXXX netmask 0xffffff80 broadcast XXXXXXXXXXXXX
        ether 00:30:48:81:fc:8a
        media: Ethernet autoselect (none)
        status: no carrier
bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=1b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING>
        inet XXXXXXXXX netmask 0xffffff00 broadcast XXXXXXXXXXX
        ether 00:30:48:81:fc:8b
        media: Ethernet autoselect (none)
        status: no carrier

Note that the interfaces are UP, not DOWN.  Then the very next thing
seen on the console:

Starting pflog.
pflog0: promiscuous mode enabled
Enabling pf.
Oct  3 04:14:51 pflogd[374]: [priv]: msg PRIV_OPEN_LOG received
cannot determine interface bandwidth for bge0, specify an absolute
bandwidth
altq not defined on bge0
altq not defined on bge0
/conf/ME/pf.conf:52: errors in queue definition
altq not defined on bge0
/conf/ME/pf.conf:53: errors in queue definition
altq not defined on bge0
/conf/ME/pf.conf:54: errors in queue definition
pfctl: Syntax error in config file: pf rules not loaded
pf enabled

The error message about "interface bandwidth" is the key here.  My ALTQ
rules use "bandwidth <percentage>", not a static amount in bits:

altq on $ext_if cbq bandwidth 100% queue { std, blah, blah2 }
queue std      bandwidth 95%   cbq(default borrow)
queue blah     bandwidth 384Kb
queue blah2    bandwidth 384Kb

Since the PHY hadn't negotiated speed, pf was unable to determine what
the percentage really mapped to bandwidth/bit-wise.

If at all possible, pf should wait for the interfaces to come up fully
(that includes autonegotiation being completed; do we have framework for
this?) before starting.

I changed my rules to use a static speeds (100% --> 100Mb, and 95% -->
95Mb), which appear to work, but after the 2nd reboot the speed/duplex
had been negotiated by the time pf had started, so I don't know if it
truly fixed anything.  I don't know what pf will do if you say "100Mb"
for an interface which has no link/speed defined yet.  It may behave the
same way as shown above; I don't know.

This needs some thought and definitely a solution.  Again, note that I'm
using RELENG_6, but I've a feeling this might bite RELENG_7 too.

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |