IPv6 Tunnel Shared With Jails via epair Devices

Tue Jan 15 05:29:46 UTC 2013

Quoth Shawn Webb <lattera at gmail.com>:
> 
> I've been working on sharing a 6in4 IPv6 tunnel (via a gif device) I have
> with Hurricane Electric (tunnelbroker.net) to my jails via epair devices.
> My setup is a bit unique in that the IPv6 tunnel is behind an OpenVPN
> connection. I've had varying degrees of success. I might have a bug to
> report, but I thought I'd post here to get input from people who know
> better than I do about these kinds of things.
> 
> I have a bridge device (we'll call it bridge0) with a /64 IPv6 address
> (2001:470:8142:1::1). Each jail's epair[n]b device will get an IPv6 address
> in that same prefix. For example, one of my jails is 2001:470:8142:1::3.
> The default IPv6 gateway is the IPv6 address of bridge0.
> 
> Giving one jail an IP address works fine. For each jail after that, the
> IPv6 address stays in tentative mode. FreeBSD gets stuck trying to use DAD
> to figure out if there's an address conflict. It never leaves tentative
> mode. This is the bug I'm working out.
> 
> Here's bridge0's config:
> 
> # ifconfig bridge0
> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 1500
> ether 02:fe:21:34:d3:00
> inet6 2001:470:8142:1::1 prefixlen 64
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
> maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
> root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
> member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>        ifmaxaddr 0 port 19 priority 128 path cost 2000
> member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>        ifmaxaddr 0 port 21 priority 128 path cost 2000
> member: bge0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>        ifmaxaddr 0 port 5 priority 128 path cost 200000

Why have you added the physical interface to the bridge? AFAICT you
don't need to: a bridge will bridge epairs just fine, and as you
explained in that blog post you have to route rather than bridge into
the tunnel, since the tunnel isn't an Ethernet device.

> Here's the relevant epair device for the jail whose IPv6 stack is working:
> 
> # jexec "ClamAV_Dev" ifconfig epair1b
> epair1b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 1500
> options=8<VLAN_MTU>
> ether 02:fb:c0:00:16:0b
> inet6 2001:470:8142:1::3 prefixlen 64
> inet6 fe80::fb:c0ff:fe00:160b%epair1b prefixlen 64 scopeid 0x2
> inet 10.7.1.172 netmask 0xfffffe00 broadcast 10.7.1.255
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
> status: active
> 
> Here's the relevant epair device for the jail whose IPv6 stack isn't
> working:
> 
> # jexec "Dev Template" ifconfig epair0b
> epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 1500
> options=8<VLAN_MTU>
> ether 02:80:03:00:14:0b
> inet6 2001:470:8142:1::5 prefixlen 64 tentative
> inet6 fe80::80:3ff:fe00:140b%epair0b prefixlen 64 tentative scopeid 0x2
> inet 10.7.1.92 netmask 0xfffffe00 broadcast 10.7.1.255
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

I suspect the addresses are only marked tentative because the interface
has been marked IFDISABLED. This causes all current addresses to be
marked tentative, because the kernel isn't allowed to send or receive
IPv6 packets and so can't defend the addresses any more.

Is it possible something in the jail's startup scripts is causing the
interface to be marked IFDISABLED after the inet6 address has been
assigned? Some of the functions in network.subr mark interfaces
IFDISABLED automatically if they don't think they have IPv6 addresses.

> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
> status: active
> 
> I brought up the "Dev Template" jail after bringing up the ClamAV_Dev jail.
> If there's any other output you'd like to see, let me know. If you're
> confused about my setup, visit my blog post about the subject here:
> http://0xfeedface.org/blog/lattera/2013-01-12/tunneled-ipv6-freebsd-jails
> 
> I'm curious to know if I've got a legit bug or if it's something I'm doing
> wrong. The one thing I haven't tried is setting up rtadvd on the bridge.
> That'd be kindof interesting, since my physical NIC is a member on the
> bridge. I'd rather not dish out IPv6 addresses for all devices on the
> network (a network with lots of devices I don't own or control).

As I said, I don't believe you need the physical interface on the
bridge, unless you have to for IPv4 (and you can't route or proxyarp
instead). However, before you can run rtadvd you will need to give the
bridge its proper link-local address, which probably also means locking
down its hardware address in rc.conf. Bridges don't get auto link-local
addresses, for reasons I've never entirely understood, and RAs have to
use ll addresses.

You will need to set up routing so that packets coming in through the
tunnel destined for the jails get routed out of the bridge, and packets
coming in on the bridge destined for the IPv6 Internet get routed out of
the tunnel. Probably that will have happened already, just by assigning
an inet6 address and prefixlen to the bridge and the default inet6 route
to the tunnel.

Ben