kern/170070: 9.1-PRERELEASE host cannot talk to itself over IPv6 (except on loopback)

Sun Jul 22 18:50:03 UTC 2012

>Number:         170070
>Category:       kern
>Synopsis:       9.1-PRERELEASE host cannot talk to itself over IPv6 (except on loopback)
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Jul 22 18:50:03 UTC 2012
>Closed-Date:
>Last-Modified:
>Originator:     Mike Andrews
>Release:        FreeBSD 9.1-PRERELEASE amd64
>Organization:
Fark, Inc
>Environment:

	Multiple amd64 systems w/ multiple NICs

>Description:

With a 9.1-PRERELEASE (r238602) kernel built July 17, my systems cannot talk to
themselves over IPv6 on anything except the loopback address.  Other systems can
talk to them fine, IPv4 works fine, and an older 9.0-STABLE (r236963) kernel built
June 12, works fine.  This is across a mix of Intel systems with a mix of NICs,
mostly em and rl, but all amd64.  Obviously this causes some bizarre problems,
like hosts not being able to talk to their own jails, DNS failures if resolv.conf
is pointed to a locally running named....

>How-To-Repeat:

IPs and domains redacted somewhat.

root at whitedog:/etc # head -7 rc.conf
ipv6_activate_all_interfaces="YES"
cloned_interfaces="lagg0"
ifconfig_re1="up -tso -rxcsum -txcsum mtu 5000"
ifconfig_re0="up -tso -rxcsum -txcsum mtu 5000"
ifconfig_lagg0="laggproto failover laggport re0 laggport re1 10.0.27.15/25 mtu 5000"
ifconfig_lagg0_ipv6="inet6 YYYY:YYYY:Y:11b::fafa:330/64"
ifconfig_lagg0_alias0="inet6 fc00::fafa:330/64"
root at whitedog:/etc # ifconfig
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 5000
        options=82098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether XX:XX:XX:da:58:92
        inet6 fe80::XXX:XXff:feda:5892%re0 prefixlen 64 scopeid 0x1 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
re1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 5000
        options=82098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether XX:XX:XX:da:58:92
        inet6 fe80::XXX:XXff:feda:5892%re1 prefixlen 64 scopeid 0x2 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8 
        inet 127.0.0.1 netmask 0xff000000 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 5000
        options=82098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether XX:XX:XX:da:58:92
        inet 10.0.27.15 netmask 0xffffff80 broadcast 10.0.27.127
        inet6 fe80::XXX:XXff:feda:5892%lagg0 prefixlen 64 scopeid 0x9 
        inet6 YYYY:YYYY:Y:11b::fafa:330 prefixlen 64 
        inet6 fc00::fafa:330 prefixlen 64 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        laggproto failover lagghash l2,l3,l4
        laggport: re1 flags=0<>
        laggport: re0 flags=5<MASTER,ACTIVE>
root at whitedog:/etc # telnet 10.0.27.15 22
Trying 10.0.27.15...
Connected to whitedog.int.OURDOMAIN.com.
Escape character is '^]'.
SSH-1.99-OpenSSH_5.8p2_hpn13v11 FreeBSD-20110503
quit
Protocol mismatch.
Connection closed by foreign host.
root at whitedog:/etc # telnet 127.0.0.1 22
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
SSH-1.99-OpenSSH_5.8p2_hpn13v11 FreeBSD-20110503
quit
Protocol mismatch.
Connection closed by foreign host.
root at whitedog:/etc # telnet ::1 22
Trying ::1...
Connected to localhost.
Escape character is '^]'.
SSH-1.99-OpenSSH_5.8p2_hpn13v11 FreeBSD-20110503
quit
Protocol mismatch.
Connection closed by foreign host.
root at whitedog:/etc # telnet fc00::fafa:330 22
Trying fc00::fafa:330...
telnet: connect to address fc00::fafa:330: Operation timed out
telnet: Unable to connect to remote host
root at whitedog:/etc # telnet YYYY:YYYY:Y:11b::fafa:330 22
Trying YYYY:YYYY:Y:11b::fafa:330...
telnet: connect to address YYYY:YYYY:Y:11b::fafa:330: Operation timed out
telnet: Unable to connect to remote host
root at whitedog:/etc # telnet fe80::XXX:XXff:feda:5892%lagg0 22
Trying fe80::XXX:XXff:feda:5892...
telnet: connect to address fe80::XXX:XXff:feda:5892: Operation timed out
telnet: Unable to connect to remote host
root at whitedog:/etc # uname -a
FreeBSD whitedog.int.OURDOMAIN.com 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #44 r238602: Wed Jul 18 17:18:20 EDT 2012     root at beer.int.OURDOMAIN.com:/usr/obj/usr/src/sys/BLAH64  amd64
root at whitedog:/etc # ipfw -a list
00100  463  68374 allow ip from any to any via lo0
00200    0      0 deny ip from any to 127.0.0.0/8
00300    0      0 deny ip from 127.0.0.0/8 to any
00400    0      0 deny ip from any to ::1
00500    0      0 deny ip from ::1 to any
00600    0      0 allow ipv6-icmp from :: to ff02::/16
00700    5    336 allow ipv6-icmp from fe80::/10 to fe80::/10
00800    2    208 allow ipv6-icmp from fe80::/10 to ff02::/16
00900    0      0 allow ipv6-icmp from any to any ip6 icmp6types 1
01000   41   2792 allow ipv6-icmp from any to any ip6 icmp6types 2,135,136
65000 1716 167475 allow ip from any to any
65535    0      0 deny ip from any to any

>From another system on the same subnet at the same time, though:

root at beer:/etc # telnet 10.0.27.15 22
Trying 10.0.27.15...
Connected to whitedog.int.OURDOMAIN.com.
Escape character is '^]'.
SSH-1.99-OpenSSH_5.8p2_hpn13v11 FreeBSD-20110503
quit
Protocol mismatch.
Connection closed by foreign host.
root at beer:/etc # telnet fc00::fafa:330 22
Trying fc00::fafa:330...
Connected to fc00::fafa:330.
Escape character is '^]'.
SSH-1.99-OpenSSH_5.8p2_hpn13v11 FreeBSD-20110503
quit
Protocol mismatch.
Connection closed by foreign host.
root at beer:/etc # telnet YYYY:YYYY:Y:11b::fafa:330 22
Trying YYYY:YYYY:Y:11b::fafa:330...
Connected to whitedog.int6.OURDOMAIN.com.
Escape character is '^]'.
SSH-1.99-OpenSSH_5.8p2_hpn13v11 FreeBSD-20110503
quit
Protocol mismatch.
Connection closed by foreign host.
root at beer:/etc # telnet fe80::XXX:XXff:feda:5892%lagg0 22
Trying fe80::XXX:XXff:feda:5892...
Connected to fe80::XXX:XXff:feda:5892%lagg0.
Escape character is '^]'.
SSH-1.99-OpenSSH_5.8p2_hpn13v11 FreeBSD-20110503
quit
Protocol mismatch.
Connection closed by foreign host.

>Fix:

Revert kernel to r236963 or to IPv4.
I have not yet narrowed down further the specific revision that breaks this.

>Release-Note:
>Audit-Trail:
>Unformatted: