Local IPv6 traffic not send over loopback?

Wed Feb 15 21:44:28 UTC 2012

Hi Julian and Terrence,

Thanks for your tests!

I'm now convinced there is a bug in ipfw.

As Terrence and I tested, ipfw is matching rules, and reporting in the
log, that IPv6 traffic between local IPv6 addresses (from me6 to me6)
using an interface other than lo0.

e.g.
ipfw: 1 Count ICMPv6:128.0 [2001:610:767:1981::1] [2001:610:767:1981::1]
in via em3

This was in contrast with "route get", which reported that the traffic
would use lo0.

So either "route" or ipfw is wrong. By varying the MTU size, I verified
that ping6 traffic from two local IPv6 addresses (not ::1) indeed uses
the lo0 (loopback) interface. Thus not the em3 interface as ipfw is
reporting.

I can only conclude that this is a bug in ipfw.
Is there a place where I can submit a bug report?

While writing down the bug report, I realised that in my previous mail
there were actually two issues:

1. ipfw reports that IPv6 traffic from me to me is using a non-loopback
interface. This is unexpected to me, since the traffic really seems to
use the loopback interface lo0, as reported by "route get" and tested by
varying the MTU.

2. ICMPv6 replies originating at the pinged host match a rule with the
"recv" pattern. This is unexpected to me, given the line man 8 ipfw:
"packets originating from the local host have no receive interface"

To separate the two issues, I've included two bug reports below.

Regards,
Freek

_ ______________________________________________

Report #1: ipfw reports wrong interface for IPv6 loopback traffic

Summary:
ipfw matches and reports incorrect interface for IPv6 traffic between
through the lookback interface in case the source/destination IP is not ::1.

Steps to reproduce:
1. Determine a global IPv6 address of your host. In this report,
2001:610:767:4ec1::1 is used.
2. Verify that the routing table has an entry associating this IP
address with the loopback interface. E.g.:
% route get -inet6 2001:610:767:4ec1::1
   route to: 2001:610:767:4ec1::1
destination: 2001:610:767:4ec1::1
  interface: lo0
3. Add the following rules to ipfw:
# ipfw add 1 count log ipv6 from me6 to me6 recv lo0
# ipfw add 2 count log ipv6 from me6 to me6 not recv lo0
4. Run sudo sysctl net.inet.ip.fw.one_pass=0
5. Ping the local IP:
% ping6 -c 1 2001:610:767:4ec1::1
6. examine the IPFW log file.

Expected result:
I expected rule 1 to match:
> ipfw: 1 Count ICMPv6:128.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::1] out via lo0
> ipfw: 1 Count ICMPv6:128.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::1] in via lo0
> ipfw: 1 Count ICMPv6:129.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::1] out via lo0
> ipfw: 1 Count ICMPv6:129.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::1] in via lo0

Actual result:
Rule 2 matched:
> ipfw: 2 Count ICMPv6:128.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::1] out via lo0
> ipfw: 2 Count ICMPv6:128.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::1] in via em0
> ipfw: 2 Count ICMPv6:129.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::1] out via lo0
> ipfw: 2 Count ICMPv6:129.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::1] in via em0

Further analysis:
Adding rules for "xmit lo0" and "not xmit lo0" clearly show that
according to ipfw, the packet is "xmit lo0 recv em0".

Beside using "route get" in step 2 above, there is another method to
determine the interface that is used by the packet, the MTU size.
By default, the MTU of lo0 is 16384 bytes, and other interface have a
MTU of 1500 bytes:
% ifconfig:
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

% ping6 -c 1 -D -s 16336 -b 32000 "2001:610:767:4ec1::1"
works fine
% ping6 -c 1 -D -s 16337 -b 32000 "2001:610:767:4ec1::1"
fails

Hence, the MTU is 16336 bytes payload + 48 byte overhead = 16384 bytes,
so that indicates the actual interface used is lo0, not em0.

Since a packet of 16384 bytes comes through, and em0 has a MTU of 1500
bytes, this also means that em0 is really not used, and the receiving
interface as reported and matched by ipfw is wrong.

_ ______________________________________________

Report #2: ipfw matches recv data for outgoing ICMPv6 replies.

Summary:
When a host creates an IPv6 reply packets and sends it out, the packet
is matched by a ipfw rule with a "recv" pattern.

Steps to reproduce:
1. Determine the IPv6 addresses of two hosts, and make sure they can
ping each other over IPv6.
2. On one host, add the following rules to ipfw:
# ipfw add 10 count log icmp6 from any to me
# ipfw add 11 count log icmp6 from any to me recv \*
# ipfw add 12 count log icmp6 from any to me xmit \*
# ipfw add 13 count log icmp6 from any to me in recv \*
# ipfw add 14 count log icmp6 from any to me out recv \*
# ipfw add 15 count log icmp6 from any to me out xmit \*
# ipfw add 20 count log icmp6 from me to any
# ipfw add 21 count log icmp6 from me to any recv \*
# ipfw add 22 count log icmp6 from me to any xmit \*
# ipfw add 23 count log icmp6 from me to any in recv \*
# ipfw add 24 count log icmp6 from me to any out recv \*
# ipfw add 25 count log icmp6 from me to any out xmit \*
3. From the other host, ping the first host (with the ipfw rules), e.g.:
% ping6 -c 1 2001:610:767:4ec1::1
4. examine the IPFW log file.

Expected result:
> ipfw: 10 Count ICMPv6:128.0 [2001:610:767:4ec1::6] [2001:610:767:4ec1::1] in via em3
> ipfw: 11 Count ICMPv6:128.0 [2001:610:767:4ec1::6] [2001:610:767:4ec1::1] in via em3
> ipfw: 13 Count ICMPv6:128.0 [2001:610:767:4ec1::6] [2001:610:767:4ec1::1] in via em3
> ipfw: 20 Count ICMPv6:129.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::6] out via em3
> ipfw: 22 Count ICMPv6:129.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::6] out via em3
> ipfw: 25 Count ICMPv6:129.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::6] out via em3

Actual result:
> ipfw: 10 Count ICMPv6:128.0 [2001:610:767:4ec1::6] [2001:610:767:4ec1::1] in via em3
> ipfw: 11 Count ICMPv6:128.0 [2001:610:767:4ec1::6] [2001:610:767:4ec1::1] in via em3
> ipfw: 13 Count ICMPv6:128.0 [2001:610:767:4ec1::6] [2001:610:767:4ec1::1] in via em3
> ipfw: 20 Count ICMPv6:129.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::6] out via em3
> ipfw: 21 Count ICMPv6:129.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::6] out via em3
> ipfw: 22 Count ICMPv6:129.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::6] out via em3
> ipfw: 24 Count ICMPv6:129.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::6] out via em3
> ipfw: 25 Count ICMPv6:129.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::6] out via em3

Further analysis:
It seem that the outgoing ICMP reply (ICMPv6:129) matches the rules with
"recv" match as well. This is unexpected, as the ICMPv6 reply originates
at the first host, and the man page of ipfw clearly states: "packets
originating from the local host have no receive interface"

This behaviour only seem to occur with ICMPv6 replies. ICMPv6 requests
work fine, as can be seem from this result to ping host two from host one:
% ping6 -c 1 2001:610:767:4ec1::6
> ipfw: 20 Count ICMPv6:135.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::6] out via em3
> ipfw: 22 Count ICMPv6:135.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::6] out via em3
> ipfw: 25 Count ICMPv6:135.0 [2001:610:767:4ec1::1] [2001:610:767:4ec1::6] out via em3
> ipfw: 10 Count ICMPv6:136.0 [2001:610:767:4ec1::6] [2001:610:767:4ec1::1] in via em3
> ipfw: 11 Count ICMPv6:136.0 [2001:610:767:4ec1::6] [2001:610:767:4ec1::1] in via em3
> ipfw: 13 Count ICMPv6:136.0 [2001:610:767:4ec1::6] [2001:610:767:4ec1::1] in via em3

Not all incoming traffic is affected. The following excerpt is from a
SSH session that the second host initiates, connecting to the first host:
% ssh -6 2001:610:767:4ec1::1
> ipfw: 10 Count TCP [2001:610:767:4ec1::6]:55370 [2001:610:767:4ec1::1]:22 in via em3
> ipfw: 11 Count TCP [2001:610:767:4ec1::6]:55370 [2001:610:767:4ec1::1]:22 in via em3
> ipfw: 13 Count TCP [2001:610:767:4ec1::6]:55370 [2001:610:767:4ec1::1]:22 in via em3
> ipfw: 20 Count TCP [2001:610:767:4ec1::1]:22 [2001:610:767:4ec1::6]:55370 out via em3
> ipfw: 22 Count TCP [2001:610:767:4ec1::1]:22 [2001:610:767:4ec1::6]:55370 out via em3
> ipfw: 25 Count TCP [2001:610:767:4ec1::1]:22 [2001:610:767:4ec1::6]:55370 out via em3

I would expect ICMPv6 replies to behave (with regard to matching "recv"
patterns) the same as other packets originating from a host, and never
matches.

_ ______________________________________________