Is there anything weird I should know about using ipfw on alias
smithi at nimnet.asn.au
Thu Dec 4 06:36:15 PST 2008
On Thu, 4 Dec 2008, Brett Davidson wrote:
> Ian Smith wrote:
> > On Tue, 2 Dec 2008, Brett Davidson wrote:
> > > Ian Smith wrote:
> > > > On Mon, 01 Dec 2008 16:52:12 +1300 Brett Davidson <brett at net24.co.nz>
> > > > wrote:
> > > > > > > ifconfig shows the alias addresses correctly bound.
> > > > > Creating an ipfw rule and testing it from the command line works >
> > > > (connects out from master address, not alias)
> > > > > > From website on alias address, the firewall blocks the packets.
> > > > >
> > > > > The weird thing is that it tags them (in the security log) as
> > coming >
> > > > from the master address (not the alias) out the correct interface. In
> > a >
> > > > normal world that would mean the packet would match!!!!!
> > > > > > What's goin' on here Willis?
> > > > > > Difficult to tell without seeing a) ifconfig b) netstat -rn c) at
> > least the
> > > > relevant firewall rule/s and d) log entries that illustrate your
> > problem.
> > > > Obscure sensitive information by all means, but otherwise pretend we
> > > > haven't the slightest clue how your system is configured :)
> > >
> > > Fair enough.
> > > > ifconfig below:
> > > > bce1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> > > options=3b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU>
> > > inet 22.214.171.124 netmask 0xffffffe0 broadcast 126.96.36.199
> > NB ..
> > > inet 188.8.131.52 netmask 0xffffffff broadcast 184.108.40.206
> > > inet 220.127.116.11 netmask 0xffffffff broadcast 18.104.22.168
> > > inet 22.214.171.124 netmask 0xffffffff broadcast 126.96.36.199
> > > inet 188.8.131.52 netmask 0xffffffff broadcast 184.108.40.206
> > > inet 220.127.116.11 netmask 0xffffffff broadcast 18.104.22.168
> > > inet 22.214.171.124 netmask 0xffffffff broadcast 126.96.36.199
> > > ether 00:1c:c4:c0:56:94
> > > media: Ethernet autoselect (1000baseSX <full-duplex>)
> > > status: active
> > > > Relevant /etc/rc.conf entries :
> > > ifconfig_bce1="inet 188.8.131.52 netmask 255.255.255.224"
> > > ifconfig_bce1_alias0="inet 184.108.40.206 netmask 255.255.255.224"
> > Your first alias here is a repeat of the 'primary' address. ifonfig seems
> > to have resolved/merged that above, but it's not an alias.
> True. Blame that on the piece of software (Plesk) that manages the IP
> addresses for the websites we host.
Ok in this instance. Please copy the list on replies, for archives.
> > > ifconfig_bce1_alias1="inet 220.127.116.11 netmask 255.255.255.255"
> > > ifconfig_bce1_alias2="inet 18.104.22.168 netmask 255.255.255.255"
> > > ifconfig_bce1_alias3="inet 22.214.171.124 netmask 255.255.255.255"
> > > ifconfig_bce1_alias4="inet 126.96.36.199 netmask 255.255.255.255"
> > > ifconfig_bce1_alias5="inet 188.8.131.52 netmask 255.255.255.255"
> > > ifconfig_bce1_alias6="inet 184.108.40.206 netmask 255.255.255.255"
> > I didn't spot on first reading this that the first address is in a
> > different subnet than all the others. I'm not entirely sure whether that's
> > relevant, or how, just pointing it out as being non-obvious, and suspecting
> > one of the 210.5.51 subnet should show a broader netmask.
> I've wondered that as well but it all works EXCEPT for when ipfw is involved.
Looks like we may need to see more, if not all, of your ipfw ruleset.
'ipfw -ted show' is pretty good for seeing everything. try adding 'log'
to some more rules, until you can SEE where packets are getting blocked.
Doesn't 'tcpdump -pn -i bce1 host 220.127.116.11 and host 18.104.22.168'
provide any good clues to these flows? Or in this case maybe better:
tcpdump -pn -i bce1 host \(22.214.171.124 or 126.96.36.199\) and host 188.8.131.52
> > > Relevant ipfw rules :
> > > ipfw -q add 02012 allow tcp from any to 184.108.40.206 80 out via bce1
> > setup
> > > keep-state
> > > ipfw -q add 02012 allow tcp from any to 220.127.116.11 443 out via bce1
> > setup
> > > keep-state
Do you have a check-state rule? Where? Are there any skiptos that
might miss anything? Do you have rules affecting established traffic?
Sorry, but I find this too like a guessing game, or pulling teeth :)
> > netstat -finet -rn (or -rna) please? unclear where your default route
> > goes, or how the 210.5.51 subnet is routed or its netmask, but assume that
> > 18.104.22.168 is probably accessed via the default route ..
> Routing tables
> Destination Gateway Flags Refs Use Netif Expire
> default 22.214.171.124 UGS 0 296628406 bce1
> 127.0.0.1 127.0.0.1 UH 0 4339898 lo0
> 172.16.1/24 link#1 UC 0 0 bce0
> 172.16.1.1 00:04:28:ad:10:00 UHLW 1 0 bce0 1035
> 172.16.1.4 00:04:23:08:28:30 UHLW 1 167202525 bce0 1189
> 172.16.1.8 00:04:23:b2:f7:17 UHLW 1 0 bce0 1021
> 172.16.1.9 00:04:23:c7:79:0d UHLW 1 1 bce0 1190
> 172.16.1.12 00:07:e9:f4:cc:51 UHLW 1 0 bce0 1021
> 172.16.1.23 00:04:23:c7:78:cd UHLW 1 0 bce0 1199
> 172.16.1.30 00:04:23:d1:b9:c7 UHLW 1 0 bce0 1104
> 172.16.1.255 ff:ff:ff:ff:ff:ff UHLWb 1 156242 bce0
> 210.5.50/27 link#2 UC 0 0 bce1
> 126.96.36.199 00:04:28:ad:10:00 UHLW 2 0 bce1 1060
> 188.8.131.52 00:1c:c4:c0:56:94 UHLW 1 5010367 lo0
> 184.108.40.206 ff:ff:ff:ff:ff:ff UHLWb 1 350707 bce1
> 220.127.116.11/32 link#2 UC 0 0 bce1
> 18.104.22.168/32 link#2 UC 0 0 bce1
> 22.214.171.124/32 link#2 UC 0 0 bce1
> 126.96.36.199/32 link#2 UC 0 0 bce1
> 188.8.131.52/32 link#2 UC 0 0 bce1
> 184.108.40.206/32 link#2 UC 0 0 bce1
> As you can see,. it's pretty straightforward and works except where ipfw is
> involved with the website on 220.127.116.11.
Ok, so if it works without ipfw running, then some ipfw rule is blocking
traffic, it's safe enough to say. See above about showing timestamps
to see what happened when (ie what just happened on the latest test).
> > > Interesting entries in /var/log/security :
> > > Dec 1 16:42:25 <servername> kernel: ipfw: 9999 Deny TCP
> > 18.104.22.168:49708
> > > 22.214.171.124:80 out via bce1
> > Did that occur =after= the above rules were installed? Just the one?
> > Seems odd on face value, but without knowing what your other rules do.
> Yes - it's our last default catch-all rule that blocks and logs all traffic
> not handled by any previous rules.
Mmm. And these keep on occurring all the while? Or just on startup?
> > > What makes this interesting is that I can connect to that port via the
> > > command line.
> > You mean like with 'telnet 126.96.36.199 80' ? With 188.8.131.52 as source
> > address? tcpdump output may help understand or explain this.
> Indeed! That's exactly what I mean which is why the security log output is a
> little weird.
tcpdump output may help .. and some timestamped logging. Dig deeper.
> > > It's the website that lives on 184.108.40.206 that is having problems. Why,
> > if
> > > the rule is valid enough for the command line is it having problems from
> > an
> > > aliased address?
> > Hang on; do you mean you're having a webserver on 220.127.116.11 trying to
> > connect out to another webserver on 18.104.22.168 ? If not, what?
Ah well. So what are you using / how are you getting this webserver's
alias to establish outbound sockets using that address as the source
address, rather than the 22.214.171.124 the system will normally assume?
Some things - named's one example - allow specifying the source query
address, so show how are you doing that from a webserver, where you can
specify listening address/es inbound, but not (usually) outbound?
If these packets were sourced from 126.96.36.199 and not 188.8.131.52 as
you're expecting, then replies will never go back to 184.108.40.206, eh?
> > I guess you have rules allowing inbound port 80 access to 220.127.116.11 ?
> Yes, as well as 443 (the reason for having a seperate address in the first
> > And that your upstream is routing 18.104.22.168/something to 22.214.171.124 ?
> > > This MUST have something to do with the way ipfw is working with aliased
> > > addresses but I'm blowed if I know what is wrong.
> > ipfw doesn't do anything different with any address in particular except
> > when using the forward action. ipfw certainly has no concept of primary or
> > alias addresses, it just applies the addresses/masks you specify.
> > Nor does ipfw know or care (even when forwarding) whence the stack is next
> > going to route outbound packets .. but netstat -rn will tell us.
Sorry, still way too little evidence to convict this masked culprit ..
for this non-psychic bear of little brain anyway.
Did I mention tcpdump?
More information about the freebsd-questions