netgraph network setup for jail(8) vnet jails.
Teske, Devin
Devin.Teske at fisglobal.com
Sat May 18 23:29:56 UTC 2013
Sorry for top-post, but just wanted to add a quick note:
The output of "ngctl dot" would be very helpful to others in debugging your setup.
--
Devin
On May 18, 2013, at 8:38 AM, Joe wrote:
> Hello list
>
> I cant get to the internet using this netgraph setup script.
> I sure would appreciate giving this console log a look over for
> errors. My netgraph knowledge level is not sufficient to see what is
> wrong. The goal is to run this script to setup and break down a netgraph
> network for a single vnet jail at a time. rl0 is the real nic interface
> device name of the nic facing the internet. This box is on my lan and
> the gateway box does NAT for all lan boxes. The host running this script can ping the internet ok.
>
> Thank you very much for your help.
>
>
>
>
>
> The host's kernel has modules with vimage & ipfw compiled in.
>
> From the host
> # /root >ifconfig
> rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> options=2008<VLAN_MTU,WOL_MAGIC>
> ether 00:0c:6e:09:8b:74
> inet 10.0.10.5 netmask 0xfffffff8 broadcast 10.0.10.7
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (100baseTX <full-duplex>)
> status: active
> plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8
> inet 127.0.0.1 netmask 0xff000000
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
>
> The jails config file
> # /root >cat /usr/local/etc/vnet/vdir4
> vdir4 {
> host.hostname = "vdir4";
> path = "/usr/jails/vdir4";
> mount.fstab = "/usr/local/etc/fstab/vdir4";
> vnet;
> persist;
> }
>
> The netgraph script
> # /root >cat /usr/local/bin/vnet.ng.test
> #!/bin/sh
> # snip comments for displaying here
> # This script is based on this /usr/share/examples/netgraph/virtual.lan
>
> # Give the name of ethernet interface.
> ETHER_INTF="rl0"
>
> # List the names of virtual nodes and their IP addresses. Use ':'
> # character to separate node name from node IP address and netmask.
>
> #TARGET_TOPOLOGY="c1|10.0.2.20/24 c2|10.0.2.21/24 c3|10.0.2.22/24"
> TARGET_TOPOLOGY="vdir4|10.0.2.20/24"
>
> # MAC manufacturer prefix. This can be modified according to needs.
> MAC_PREFIX="00:1d:92"
>
> # Temporary file is important for proper execution of script.
> TEMP_FILE="/var/tmp/virtual.lan.tmp"
>
> virtual_lan_start() {
>
> # Load netgraph KLD's as necessary.
>
> for KLD in ng_ether ng_bridge ng_eiface; do
> if ! kldstat -v | grep -qw ${KLD}; then
> echo -n "Loading ${KLD}.ko... "
> kldload ${KLD} || exit 1
> echo "done"
> fi
> done
>
> # Reset all interfaces and jails. If temporary file can not be found
> # script assumes that there is no previous configuration.
>
> if [ ! -e ${TEMP_FILE} ]; then
> echo "No previous configuration(${TEMP_FILE}) found to clean-up."
> else
> echo -n "Cleaning previous configuration..."
> virtual_lan_stop
> echo "done"
> fi
>
> # Create temporary file for usage. This file includes generated
> # interface names and jail names. All bridges, interfaces and jails
> # are written to file while created. In clean-up process written
> # objects are cleaned (i.e. removed) from system.
>
> if [ -e ${TEMP_FILE} ]; then
> touch ${TEMP_FILE}
> fi
>
> echo -n "Verifying ethernet interface existence..."
> # Verify ethernet interface exist.
> if ! ngctl info ${ETHER_INTF}: >/dev/null 2>&1; then
> echo "Error: interface ${ETHER_INTF} does not exist"
> exit 1
> fi
>
> ifconfig ${ETHER_INTF} up || exit 1
> echo "done"
>
> # Get current number of bridge interfaces in the system. This number
> # is used to create a name for new bridge.
> BRIDGE_COUNT=`ngctl l | grep bridge | wc -l | sed -e "s/ //g"`
> BRIDGE_NAME="bridge${BRIDGE_COUNT}"
>
> # Create new ng_bridge(4) node and attach it to the ethernet interface.
> # Connect ng_ether:lower hook to bridge:link0 when creating bridge and
> # connect ng_ether:upper hook to bridge:link1 after bridge name is set.
>
> echo "Creating bridge interface: ${BRIDGE_NAME}..."
> ngctl mkpeer ${ETHER_INTF}: bridge lower link0 || exit 1
> ngctl name ${ETHER_INTF}:lower ${BRIDGE_NAME} || exit 1
> ngctl connect ${ETHER_INTF}: ${BRIDGE_NAME}: upper link1 || exit 1
> echo "Bridge ${BRIDGE_NAME} is created and ${ETHER_INTF} is connected."
>
> # In the above code block two hooks are connected to bridge interface,
> # therefore LINKNUM is set to 2 indicating total number of connected
> # hooks on the bridge interface.
> LINKNUM=2
>
> # Write name of the bridge to temp file. Clean-up procedure will use
> # this name to shutdown bridge interface.
> echo "bridge ${BRIDGE_NAME}" > ${TEMP_FILE}
>
>
> # Attach vnet jail.
> for NODE in ${TARGET_TOPOLOGY}; do
>
> # Virtual nodes are defined in TARGET_TOPOLOGY variable. They
> # have the form of 'nodeName|IPaddr'. Below two lines split
> # node definition to get node name and node IP.
>
> NODE_NAME=`echo ${NODE} | awk -F"|" '{print $1}'`
> NODE_IP=`echo ${NODE} | awk -F"|" '{print $2}'`
>
> # Create virtual node (jail) with given name
> echo -n "Creating virtual node (jail) ${NODE_NAME}..."
>
> jail -f "/usr/local/etc/vnet/${NODE_NAME}" -c ${NODE_NAME}
> if [ $? -ne 0 ]; then
> echo "Error: /usr/sbin/jail failed to start jail ${NODE_NAME}."
> virtual_lan_stop
> exit 2
> fi
> echo "done"
>
> # Write name of the jail to temp file. Clean-up procedure will
> # use this name to remove jail.
>
> echo "node ${NODE_NAME}" >> ${TEMP_FILE}
>
> # Create a ng_eiface object for virtual node. ng_eiface
> # object has a hook that can be connected to one of bridge
> # links. After creating interface get its automatically
> # generated name for further usage.
>
> echo "Creating eiface interface for virtual node ${NODE_NAME}."
> ngctl mkpeer eiface ether ether
> EIFACE=`ngctl l | grep ngeth | tail -n 1| awk '{print $2}'`
> echo "Interface ${EIFACE} is created."
>
> # Write name of the interface to temp file. Clean-up procedure
> # will use this name to shutdown interface.
>
> echo "interface ${EIFACE}" >> ${TEMP_FILE}
>
> # Move virtual interface to virtual node. Note that Interface
> # name will not be changed at the end of this movement. Moved
> # interface can be seen at the output of ifconfig command in
> # jail: 'jexec jailname ifconfig'
>
> echo "Moving ${EIFACE} to ${NODE_NAME}"
> ifconfig ${EIFACE} vnet ${NODE_NAME}
>
> # Make lo0 interface localhost.
> jexec ${NODE_NAME} ifconfig lo0 localhost
>
> # Generate a random mac address for virtual interface. First
> # three octets can be changed by user. Last three octets are
> # generated randomly.
> M4=`od -An -N2 -i /dev/random | sed -e 's/ //g' | \
> awk '{ print $1 % 256 }'`
> M5=`od -An -N2 -i /dev/random | sed -e 's/ //g' | \
> awk '{ print $1 % 256 }'`
> M6=`od -An -N2 -i /dev/random | sed -e 's/ //g' | \
> awk '{ print $1 % 256 }'`
>
> MAC=`printf ${MAC_PREFIX}:%02x:%02x:%02x ${M4} ${M5} ${M6}`
>
> # Set the link address (mac address) of virtual interface in
> # virtual node to randomly generated MAC.
> echo "Setting MAC address of ${EIFACE} to '${MAC}'"
> jexec ${NODE_NAME} ifconfig ${EIFACE} link $MAC
>
> # Either IPv4 or IPv6 can be used in this script. Ifconfig
> # IP setting syntax differs slightly for two IP versions.
> # For version 4 'inet' keyword is used whereas for version 6
> # 'inet6' is used. Below line tries to decide which IP version
> # is given and sets IPVER to 'inet' or 'inet6'.
>
> IPVER=`echo ${NODE_IP} | awk -F"." '{ split($4,last,"/"); \
> if( NF==4 && $1>0 && $1<256 && $2<256 && $3<256 && \
> last[1]<256) print "inet"; else print "inet6"}'`
>
> # Set IP address of virtual interface in virtual node.
> echo "Setting IP address of ${EIFACE} to '${NODE_IP}'"
> jexec ${NODE_NAME} ifconfig ${EIFACE} ${IPVER} ${NODE_IP}
>
> # Connect virtual interface to bridge interface. Syntax is :
> # ngctl connect INTERFACE: BRIDGE: INTERFACE_HOOK EMPTY_LINK.
> # Interface has one hook named 'ether' and below line connects
> # ether hook to bridge's first unconnected link.
>
> echo -n "Connecting ${EIFACE}:ether to ${BRIDGE_NAME}:link${LINKNUM}..."
>
> ngctl connect ${EIFACE}: ${BRIDGE_NAME}: ether link${LINKNUM} \
> || exit 1
> echo "done"
>
> # Now, bridge has one more connected link thus link count is
> # incremented.
> LINKNUM=`expr ${LINKNUM} + 1`
> done
> echo "Virtual LAN established successfully!"
>
> }
>
>
> # Stop routine.
> virtual_lan_stop() {
>
> if [ ! -e ${TEMP_FILE} ]; then
> echo "Nothing to stop! ${TEMP_FILE}: temp file not found"
> else
> echo -n "Shutdown bridge interface.."
> OBJECTS=`cat ${TEMP_FILE} | grep bridge | awk '{print $2}'`
> for BRIDGE in ${OBJECTS}; do
> ngctl shutdown ${BRIDGE}: >/dev/null 2>&1
> done
> echo "done"
>
> echo -n "Shutdown all eiface interfaces..."
> OBJECTS=`cat ${TEMP_FILE} | grep interface | awk '{print $2}'`
> for INTERFACE in ${OBJECTS}; do
> ngctl shutdown ${INTERFACE}: >/dev/null 2>&1
> done
> echo "done"
>
> echo -n "Removing all jails..."
> OBJECTS=`cat ${TEMP_FILE} | grep node | awk '{print $2}'`
> for NODE in ${OBJECTS}; do
> jail -f "/usr/local/etc/vnet/${NODE}" -r ${NODE}
> done
> echo "done"
>
> echo "Removing tempfile ${TEMP_FILE}"
> rm ${TEMP_FILE}
> fi
> echo "Virtual LAN objects removed successfully!"
>
> }
>
>
> # Main entry point.
>
> case $# in
> 1)
> case $1 in
> start)
> echo -n "Creating default target topology:"
> echo " ${TARGET_TOPOLOGY}"
> virtual_lan_start
> ;;
> stop)
>
> if [ ! -e ${TEMP_FILE} ]; then
> echo -n "Noting to stop! ${TEMP_FILE}:"
> echo " temp file not found"
> else
> virtual_lan_stop
> fi
> ;;
> help)
> virtual_lan_usage
> exit 1
> ;;
> *)
> virtual_lan_usage
> exit 1
>
> esac
> ;;
> 2)
> case $1 in
> start)
> TARGET_TOPOLOGY=$2
> echo -n "Creating target topology:"
> echo "${TARGET_TOPOLOGY}"
> virtual_lan_start
> ;;
> *)
> virtual_lan_usage
> exit 1
> esac
> ;;
>
> *)
> virtual_lan_usage
> exit 1
> esac
>
>
> # /root >vnet.ng.test start
> Creating default target topology: vdir4|10.0.2.20/24
> Loading ng_ether.ko... done
> Loading ng_bridge.ko... done
> Loading ng_eiface.ko... done
> No previous configuration(/var/tmp/virtual.lan.tmp) found to clean-up.
> Verifying ethernet interface existence...done
> Creating bridge interface: bridge0...
> Bridge bridge0 is created and rl0 is connected.
> Creating virtual node (jail) vdir4...vdir4: created
> done
> Creating eiface interface for virtual node vdir4.
> Interface ngeth0 is created.
> Moving ngeth0 to vdir4
> Setting MAC address of ngeth0 to '00:1d:92:df:92:8e'
> Setting IP address of ngeth0 to '10.0.2.20/24'
> Connecting ngeth0:ether to bridge0:link2...done
> Virtual LAN established successfully!
>
>
> # /root >ngctl ls -l
> There are 5 total nodes:
> Name: rl0 Type: ether ID: 00000001 Num hooks: 2
> Local hook Peer name Peer type Peer ID Peer
> hook
> ---------- --------- --------- -------
> ---------
> upper bridge0 bridge 00000006 link1
>
> lower bridge0 bridge 00000006 link0
>
> Name: ipfw0 Type: ether ID: 00000002 Num hooks: 0
> Name: bridge0 Type: bridge ID: 00000006 Num hooks: 3
> Local hook Peer name Peer type Peer ID Peer
> hook
> ---------- --------- --------- -------
> ---------
> link2 ngeth0 eiface 0000000a ether
>
> link1 rl0 ether 00000001 upper
>
> link0 rl0 ether 00000001 lower
>
> Name: ngeth0 Type: eiface ID: 0000000a Num hooks: 1
> Local hook Peer name Peer type Peer ID Peer
> hook
> ---------- --------- --------- -------
> ---------
> ether bridge0 bridge 00000006 link2
>
> Name: ngctl1513 Type: socket ID: 0000000d Num hooks: 0
>
>
> # /root >jexec vdir4 tcsh
> vdir4 / >ping -c 1 8.8.178.135
> PING 8.8.178.135 (8.8.178.135): 56 data bytes
> ping: sendto: No route to host
>
> --- 8.8.178.135 ping statistics ---
> 1 packets transmitted, 0 packets received, 100.0% packet loss
> vdir4 / >exit
> exit
>
>
> # /root >vnet.ng.test stop
> Shutdown bridge interface..done
> Shutdown all eiface interfaces...done
> Removing all jails...vdir4: removed
> done
> Removing tempfile /var/tmp/virtual.lan.tmp
> Virtual LAN objects removed successfully!
>
> # /root >jls
> JID IP Address Hostname Path
>
> # /root >ngctl ls -l
> There are 3 total nodes:
> Name: ngctl1540 Type: socket ID: 00000010 Num hooks: 0
> Name: rl0 Type: ether ID: 00000001 Num hooks: 0
> Name: ipfw0 Type: ether ID: 00000002 Num hooks: 0
>
>
>
>
> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freebsd.org"
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
More information about the freebsd-questions
mailing list