netgraph network setup for jail(8) vnet jails.
Joe
fbsd8 at a1poweruser.com
Sat May 18 15:38:26 UTC 2013
Hello list
I cant get to the internet using this netgraph setup script.
I sure would appreciate giving this console log a look over for
errors. My netgraph knowledge level is not sufficient to see what is
wrong. The goal is to run this script to setup and break down a netgraph
network for a single vnet jail at a time. rl0 is the real nic interface
device name of the nic facing the internet. This box is on my lan and
the gateway box does NAT for all lan boxes. The host running this script
can ping the internet ok.
Thank you very much for your help.
The host's kernel has modules with vimage & ipfw compiled in.
From the host
# /root >ifconfig
rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
options=2008<VLAN_MTU,WOL_MAGIC>
ether 00:0c:6e:09:8b:74
inet 10.0.10.5 netmask 0xfffffff8 broadcast 10.0.10.7
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
The jails config file
# /root >cat /usr/local/etc/vnet/vdir4
vdir4 {
host.hostname = "vdir4";
path = "/usr/jails/vdir4";
mount.fstab = "/usr/local/etc/fstab/vdir4";
vnet;
persist;
}
The netgraph script
# /root >cat /usr/local/bin/vnet.ng.test
#!/bin/sh
# snip comments for displaying here
# This script is based on this /usr/share/examples/netgraph/virtual.lan
# Give the name of ethernet interface.
ETHER_INTF="rl0"
# List the names of virtual nodes and their IP addresses. Use ':'
# character to separate node name from node IP address and netmask.
#TARGET_TOPOLOGY="c1|10.0.2.20/24 c2|10.0.2.21/24 c3|10.0.2.22/24"
TARGET_TOPOLOGY="vdir4|10.0.2.20/24"
# MAC manufacturer prefix. This can be modified according to needs.
MAC_PREFIX="00:1d:92"
# Temporary file is important for proper execution of script.
TEMP_FILE="/var/tmp/virtual.lan.tmp"
virtual_lan_start() {
# Load netgraph KLD's as necessary.
for KLD in ng_ether ng_bridge ng_eiface; do
if ! kldstat -v | grep -qw ${KLD}; then
echo -n "Loading ${KLD}.ko... "
kldload ${KLD} || exit 1
echo "done"
fi
done
# Reset all interfaces and jails. If temporary file can not be found
# script assumes that there is no previous configuration.
if [ ! -e ${TEMP_FILE} ]; then
echo "No previous configuration(${TEMP_FILE}) found to clean-up."
else
echo -n "Cleaning previous configuration..."
virtual_lan_stop
echo "done"
fi
# Create temporary file for usage. This file includes generated
# interface names and jail names. All bridges, interfaces and jails
# are written to file while created. In clean-up process written
# objects are cleaned (i.e. removed) from system.
if [ -e ${TEMP_FILE} ]; then
touch ${TEMP_FILE}
fi
echo -n "Verifying ethernet interface existence..."
# Verify ethernet interface exist.
if ! ngctl info ${ETHER_INTF}: >/dev/null 2>&1; then
echo "Error: interface ${ETHER_INTF} does not exist"
exit 1
fi
ifconfig ${ETHER_INTF} up || exit 1
echo "done"
# Get current number of bridge interfaces in the system. This number
# is used to create a name for new bridge.
BRIDGE_COUNT=`ngctl l | grep bridge | wc -l | sed -e "s/ //g"`
BRIDGE_NAME="bridge${BRIDGE_COUNT}"
# Create new ng_bridge(4) node and attach it to the ethernet interface.
# Connect ng_ether:lower hook to bridge:link0 when creating bridge and
# connect ng_ether:upper hook to bridge:link1 after bridge name is set.
echo "Creating bridge interface: ${BRIDGE_NAME}..."
ngctl mkpeer ${ETHER_INTF}: bridge lower link0 || exit 1
ngctl name ${ETHER_INTF}:lower ${BRIDGE_NAME} || exit 1
ngctl connect ${ETHER_INTF}: ${BRIDGE_NAME}: upper link1 || exit 1
echo "Bridge ${BRIDGE_NAME} is created and ${ETHER_INTF} is connected."
# In the above code block two hooks are connected to bridge interface,
# therefore LINKNUM is set to 2 indicating total number of connected
# hooks on the bridge interface.
LINKNUM=2
# Write name of the bridge to temp file. Clean-up procedure will use
# this name to shutdown bridge interface.
echo "bridge ${BRIDGE_NAME}" > ${TEMP_FILE}
# Attach vnet jail.
for NODE in ${TARGET_TOPOLOGY}; do
# Virtual nodes are defined in TARGET_TOPOLOGY variable. They
# have the form of 'nodeName|IPaddr'. Below two lines split
# node definition to get node name and node IP.
NODE_NAME=`echo ${NODE} | awk -F"|" '{print $1}'`
NODE_IP=`echo ${NODE} | awk -F"|" '{print $2}'`
# Create virtual node (jail) with given name
echo -n "Creating virtual node (jail) ${NODE_NAME}..."
jail -f "/usr/local/etc/vnet/${NODE_NAME}" -c ${NODE_NAME}
if [ $? -ne 0 ]; then
echo "Error: /usr/sbin/jail failed to start jail ${NODE_NAME}."
virtual_lan_stop
exit 2
fi
echo "done"
# Write name of the jail to temp file. Clean-up procedure will
# use this name to remove jail.
echo "node ${NODE_NAME}" >> ${TEMP_FILE}
# Create a ng_eiface object for virtual node. ng_eiface
# object has a hook that can be connected to one of bridge
# links. After creating interface get its automatically
# generated name for further usage.
echo "Creating eiface interface for virtual node ${NODE_NAME}."
ngctl mkpeer eiface ether ether
EIFACE=`ngctl l | grep ngeth | tail -n 1| awk '{print $2}'`
echo "Interface ${EIFACE} is created."
# Write name of the interface to temp file. Clean-up procedure
# will use this name to shutdown interface.
echo "interface ${EIFACE}" >> ${TEMP_FILE}
# Move virtual interface to virtual node. Note that Interface
# name will not be changed at the end of this movement. Moved
# interface can be seen at the output of ifconfig command in
# jail: 'jexec jailname ifconfig'
echo "Moving ${EIFACE} to ${NODE_NAME}"
ifconfig ${EIFACE} vnet ${NODE_NAME}
# Make lo0 interface localhost.
jexec ${NODE_NAME} ifconfig lo0 localhost
# Generate a random mac address for virtual interface. First
# three octets can be changed by user. Last three octets are
# generated randomly.
M4=`od -An -N2 -i /dev/random | sed -e 's/ //g' | \
awk '{ print $1 % 256 }'`
M5=`od -An -N2 -i /dev/random | sed -e 's/ //g' | \
awk '{ print $1 % 256 }'`
M6=`od -An -N2 -i /dev/random | sed -e 's/ //g' | \
awk '{ print $1 % 256 }'`
MAC=`printf ${MAC_PREFIX}:%02x:%02x:%02x ${M4} ${M5} ${M6}`
# Set the link address (mac address) of virtual interface in
# virtual node to randomly generated MAC.
echo "Setting MAC address of ${EIFACE} to '${MAC}'"
jexec ${NODE_NAME} ifconfig ${EIFACE} link $MAC
# Either IPv4 or IPv6 can be used in this script. Ifconfig
# IP setting syntax differs slightly for two IP versions.
# For version 4 'inet' keyword is used whereas for version 6
# 'inet6' is used. Below line tries to decide which IP version
# is given and sets IPVER to 'inet' or 'inet6'.
IPVER=`echo ${NODE_IP} | awk -F"." '{ split($4,last,"/"); \
if( NF==4 && $1>0 && $1<256 && $2<256 && $3<256 && \
last[1]<256) print "inet"; else print "inet6"}'`
# Set IP address of virtual interface in virtual node.
echo "Setting IP address of ${EIFACE} to '${NODE_IP}'"
jexec ${NODE_NAME} ifconfig ${EIFACE} ${IPVER} ${NODE_IP}
# Connect virtual interface to bridge interface. Syntax is :
# ngctl connect INTERFACE: BRIDGE: INTERFACE_HOOK EMPTY_LINK.
# Interface has one hook named 'ether' and below line connects
# ether hook to bridge's first unconnected link.
echo -n "Connecting ${EIFACE}:ether to ${BRIDGE_NAME}:link${LINKNUM}..."
ngctl connect ${EIFACE}: ${BRIDGE_NAME}: ether link${LINKNUM} \
|| exit 1
echo "done"
# Now, bridge has one more connected link thus link count is
# incremented.
LINKNUM=`expr ${LINKNUM} + 1`
done
echo "Virtual LAN established successfully!"
}
# Stop routine.
virtual_lan_stop() {
if [ ! -e ${TEMP_FILE} ]; then
echo "Nothing to stop! ${TEMP_FILE}: temp file not found"
else
echo -n "Shutdown bridge interface.."
OBJECTS=`cat ${TEMP_FILE} | grep bridge | awk '{print $2}'`
for BRIDGE in ${OBJECTS}; do
ngctl shutdown ${BRIDGE}: >/dev/null 2>&1
done
echo "done"
echo -n "Shutdown all eiface interfaces..."
OBJECTS=`cat ${TEMP_FILE} | grep interface | awk '{print $2}'`
for INTERFACE in ${OBJECTS}; do
ngctl shutdown ${INTERFACE}: >/dev/null 2>&1
done
echo "done"
echo -n "Removing all jails..."
OBJECTS=`cat ${TEMP_FILE} | grep node | awk '{print $2}'`
for NODE in ${OBJECTS}; do
jail -f "/usr/local/etc/vnet/${NODE}" -r ${NODE}
done
echo "done"
echo "Removing tempfile ${TEMP_FILE}"
rm ${TEMP_FILE}
fi
echo "Virtual LAN objects removed successfully!"
}
# Main entry point.
case $# in
1)
case $1 in
start)
echo -n "Creating default target topology:"
echo " ${TARGET_TOPOLOGY}"
virtual_lan_start
;;
stop)
if [ ! -e ${TEMP_FILE} ]; then
echo -n "Noting to stop! ${TEMP_FILE}:"
echo " temp file not found"
else
virtual_lan_stop
fi
;;
help)
virtual_lan_usage
exit 1
;;
*)
virtual_lan_usage
exit 1
esac
;;
2)
case $1 in
start)
TARGET_TOPOLOGY=$2
echo -n "Creating target topology:"
echo "${TARGET_TOPOLOGY}"
virtual_lan_start
;;
*)
virtual_lan_usage
exit 1
esac
;;
*)
virtual_lan_usage
exit 1
esac
# /root >vnet.ng.test start
Creating default target topology: vdir4|10.0.2.20/24
Loading ng_ether.ko... done
Loading ng_bridge.ko... done
Loading ng_eiface.ko... done
No previous configuration(/var/tmp/virtual.lan.tmp) found to clean-up.
Verifying ethernet interface existence...done
Creating bridge interface: bridge0...
Bridge bridge0 is created and rl0 is connected.
Creating virtual node (jail) vdir4...vdir4: created
done
Creating eiface interface for virtual node vdir4.
Interface ngeth0 is created.
Moving ngeth0 to vdir4
Setting MAC address of ngeth0 to '00:1d:92:df:92:8e'
Setting IP address of ngeth0 to '10.0.2.20/24'
Connecting ngeth0:ether to bridge0:link2...done
Virtual LAN established successfully!
# /root >ngctl ls -l
There are 5 total nodes:
Name: rl0 Type: ether ID: 00000001 Num hooks: 2
Local hook Peer name Peer type Peer ID Peer
hook
---------- --------- --------- -------
---------
upper bridge0 bridge 00000006 link1
lower bridge0 bridge 00000006 link0
Name: ipfw0 Type: ether ID: 00000002 Num hooks: 0
Name: bridge0 Type: bridge ID: 00000006 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer
hook
---------- --------- --------- -------
---------
link2 ngeth0 eiface 0000000a ether
link1 rl0 ether 00000001 upper
link0 rl0 ether 00000001 lower
Name: ngeth0 Type: eiface ID: 0000000a Num hooks: 1
Local hook Peer name Peer type Peer ID Peer
hook
---------- --------- --------- -------
---------
ether bridge0 bridge 00000006 link2
Name: ngctl1513 Type: socket ID: 0000000d Num hooks: 0
# /root >jexec vdir4 tcsh
vdir4 / >ping -c 1 8.8.178.135
PING 8.8.178.135 (8.8.178.135): 56 data bytes
ping: sendto: No route to host
--- 8.8.178.135 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss
vdir4 / >exit
exit
# /root >vnet.ng.test stop
Shutdown bridge interface..done
Shutdown all eiface interfaces...done
Removing all jails...vdir4: removed
done
Removing tempfile /var/tmp/virtual.lan.tmp
Virtual LAN objects removed successfully!
# /root >jls
JID IP Address Hostname Path
# /root >ngctl ls -l
There are 3 total nodes:
Name: ngctl1540 Type: socket ID: 00000010 Num hooks: 0
Name: rl0 Type: ether ID: 00000001 Num hooks: 0
Name: ipfw0 Type: ether ID: 00000002 Num hooks: 0
More information about the freebsd-questions
mailing list