ifconfig VLAN cloning, renaming, configuring in rc.conf.local

Thu Jun 18 21:05:52 UTC 2015

[Please CC: me on any replies, as I check my inbox more frequently
than I check this list.]

I have three servers running 9.3-STABLE which are designed to be
able to exchange jails so that a virtual host can be readily moved
to a different physical host.  Because physical NIC names could
differ, the ezjail config files are set up to use an interface name
of 'public' or 'private' depending on which of the two physical NICs
each jail wants to use, and the rc.conf system is responsible to
configure the physical NICs with the correct 'public' and 'private' 
NICknames (pardon the expression).

The hitch is that one of the three machines is on a trunked switch
port, so that it can access multiple (two) VLANs through a single
physical NIC.

This works fine once I manually configure it, but I can't find a way
to adapt my rc.conf.local model to handle the VLAN setup
automatically at boot time.  I want to end up with a 'public'
interface on vlan 1 of the main physical NIC (and multiple IPs
configured), a 'vlan100' interface on vlan 100 of the main physical
NIC (with IPs configured), and a 'private' interface on the
secondary physical NIC (with IPs configured).

I use an identical rc.conf on the servers, and keep the nitty gritty
details in rc.conf.local, as follows:

rc.conf:

# most machine-specific stuff is in rc.conf.local
#
# these settings are common to all
#
moused_enable="YES"
gateway_enable="YES"
inetd_enable="YES"
sshd_enable="YES"
sshd_flags='-o "PermitRootLogin=without-password" -o "ListenAddress=$IP:22"'
zfs_enable="YES"
ezjail_enable="YES"
##eof##

Here is the problematic rc.conf.local:

hostname="trunked-server.example.com"

# vlan trunking on interface bce0:
# physical interface bce0 just needs to be up
ifconfig_bce0="up"

# We will clone two vlan interfaces:
cloned_interfaces="vlan1 vlan100"

# The details for those two cloned interfaces:
ifconfig_vlan1="vlan 1 vlandev bce0"
ifconfig_vlan100="vlan 100 vlandev bce0"

# Some interfaces get renamed, so that jails can find
# the "public" and "private" interfaces:
ifconfig_vlan1_name="public"
ifconfig_bce1_name="private"

# primary public IP:
IP="10.158.10.18"
MASK="/25"
defaultrouter="10.158.10.1"

# public interface IPs:

ipv4_addrs_public="
	${IP}${MASK}
	10.158.10.10/32
	10.158.10.31-47/32
"

ipv4_addrs_vlan100="
	10.158.2.5/27
"

# private interface IPs:

#ipv4_addrs_private="10.0.0.7/24"

firewall_enable="YES"            # Set to YES to enable firewall functionality
firewall_script="/root/fw.sh"

# jail settings:
jail_set_hostname_allow="NO" 
# jail_jail1_parameters="allow.raw_sockets=1 allow.sysvipc=1"
jail_parameters="
	allow.raw_sockets=1
	allow.mount.devfs=1
	allow.set_hostname=0
"
##eof##

Some things I have found:

As given above, the vlan interfaces don't get set up the way I want
them.  public gets created and has all the IPs, but is on vlan 0
with no parent device:

bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
	ether 00:1f:29:e1:22:f6
	inet6 fe80::21f:29ff:fee1:22f6%bce0 prefixlen 64 scopeid 0x1 
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
private: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
	ether 00:1f:29:e1:22:f4
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
public: flags=8003<UP,BROADCAST,MULTICAST> metric 0 mtu 1500
	ether 00:00:00:00:00:00
	inet 10.158.10.18 netmask 0xffffff80 broadcast 10.158.10.127
	inet 10.158.10.10 netmask 0xffffffff broadcast 10.158.10.10
	inet 10.158.10.31 netmask 0xffffffff broadcast 10.158.10.31
	inet 10.158.10.32 netmask 0xffffffff broadcast 10.158.10.32
	inet 10.158.10.33 netmask 0xffffffff broadcast 10.158.10.33
	inet 10.158.10.34 netmask 0xffffffff broadcast 10.158.10.34
	inet 10.158.10.35 netmask 0xffffffff broadcast 10.158.10.35
	inet 10.158.10.36 netmask 0xffffffff broadcast 10.158.10.36
	inet 10.158.10.37 netmask 0xffffffff broadcast 10.158.10.37
	inet 10.158.10.38 netmask 0xffffffff broadcast 10.158.10.38
	inet 10.158.10.39 netmask 0xffffffff broadcast 10.158.10.39
	inet 10.158.10.40 netmask 0xffffffff broadcast 10.158.10.40
	inet 10.158.10.41 netmask 0xffffffff broadcast 10.158.10.41
	inet 10.158.10.42 netmask 0xffffffff broadcast 10.158.10.42
	inet 10.158.10.43 netmask 0xffffffff broadcast 10.158.10.43
	inet 10.158.10.44 netmask 0xffffffff broadcast 10.158.10.44
	inet 10.158.10.45 netmask 0xffffffff broadcast 10.158.10.45
	inet 10.158.10.46 netmask 0xffffffff broadcast 10.158.10.46
	inet 10.158.10.47 netmask 0xffffffff broadcast 10.158.10.47
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	vlan: 0 parent interface: <none>
vlan100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=103<RXCSUM,TXCSUM,TSO4>
	ether 00:1f:29:e1:22:f6
	inet 10.158.2.5 netmask 0xffffffe0 broadcast 10.158.2.31
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
	vlan: 100 parent interface: bce0
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

My first thought was to clone the vlan 1 interface as 'public' directly,
instead of cloning it as 'vlan1' and then renaming it (although, is that
specifically not supported?)  However, ifconfig doesn't seem to like
that syntax:

# ifconfig public create vlan 1 vlandev bce0
ifconfig: SIOCIFCREATE2: Invalid argument

What do I need to do to get the vlan1 interface cloned properly,
configured with the proper IPs, and renamed as 'public'?

Thank you!  Please let me know if I can supply additional information.

Jim