jails and multple interfaces
Jeffrey Williams
jeff at sailorfej.net
Wed Jan 31 12:29:09 UTC 2007
Oliver Fromme wrote:
> Jeffrey Williams wrote:
> > I am trying to set a jail hosting server to support multiple jails for
> > development testing.
> >
> > The server has two network interfaces, I am configuring one for host
> > server to use, and the other with several aliased IPs, one for each of
> > the jail servers.
> >
> > All the services running on the host are configured to bind to the host
> > IP on the first interface.
> >
> > The crux is both interfaces on the same network, I am seeing the
> > expected arp errors (e.g. kernel: arp: x.x.x.x is on int0 but got reply
> > on int1), now I know I set the sysctl variable
> > net.link.ether.inet.log_arp_wrong_iface=0 to get rid of these messages,
> > but what I want to know if there are any other problems I am going to
> > have having both interfaces live on the same network.
>
> What exactly are your inetrface configurations and netmasks
> (ifconfig output might be useful)?
>
> You say that both NICs are on teh same network. Does that
> mean they're connected to teh same switch? That's generally
> not a good idea. It doesn't buy you anything (unless you
> use VLAN technology or other additional measures).
>
> > Also even though
> > I have the jail host's services all binding to the first interfaces ip,
> > there is not guarantee that network traffic originating from the jail
> > host will only use its primary interface/IP, is their anyway to ensure
> > that the jail host does not try to talk through the interface being used
> > by the jails?
>
> Any network traffic originating from a jail is guaranteed
> to use the jail's IP address. The interface that will be
> used is the one according to your routing table entry for
> that IP address. (Unless you use things like IPFW FWD
> or similar to redirect the packets somewhere else.)
>
> Best regards
> Oliver
>
> PS: Be very careful when binding services to localhost
> (127.0.0.1) within the jail. They will listen on the jail's
> official IP address instead! For that reason I often
> configure an addition address on lo0 (e.g. 127.0.0.2)
> and use that one for internal-only traffic such as DNS
> and mail between host and jails.
>
Yes, both NIC's are on the same network, connected to the same switch, I
not just using the jail's as a sandbox for a couple of services, rather
I am using a virtual servers, ideally I would like to have a separate
NIC for eash jail, just like I would do if I were setting these up in
vmware.
It is currently setup and running, at the moment, as near as I can tell
I am not having any IP routing issues, all the appropriate configs are
below. I don't believe (I could be wrong, definitely tell me if I am) I
am going to have and IP issues (i.e. layer 3), what I am concerned about
are the potential ethernet issues ARP/RARP (i.e. layer 2). I was
getting the expected arp errors, but I did set the
net.link.ether.inet.log_arp_wrong_iface to 0, so I am not logging the
errors, of course that doesn't mean the errors went away, they just
aren't clogging the log files now. What I want to know is, if there are
any deleterious effects from having two ethernet interfaces on the same
network segment/switch.
Frankly I would be very concerned if there wasn't a way to pull this
off, while I realize jails were not originally intended to be used as a
mechanism to provide virtual servers, it is such an obvious application
of the jail functionality, and it is perfectly reasonable to want to
have a dedicated NIC for each virtual server, with out having to place
each one on a separate network segment.
Configurations:
############rc.conf
# Please make all changes to this file, not to /etc/defaults/rc.conf.
# This file now contains just the overrides from /etc/defaults/rc.conf.
defaultrouter="192.168.10.1"
hostname="dev.inside.mydomain.com"
ifconfig_em0="inet 192.168.10.41 netmask 255.255.255.0"
ifconfig_vr0="inet 192.168.10.70 netmask 255.255.255.0"
ifconfig_vr0_alias0="192.168.10.71 netmask 255.255.255.0"
ifconfig_vr0_alias1="192.168.10.72 netmask 255.255.255.0"
ifconfig_vr0_alias2="192.168.10.73 netmask 255.255.255.0"
ifconfig_vr0_alias3="192.168.10.74 netmask 255.255.255.0"
syslogd_flags="-s -b 192.168.10.41"
sendmail_enable="NO"
moused_enable="YES"
sshd_enable="YES"
usbd_enable="YES"
ntpd_enable="YES"
rpcbind_enable="NO"
inetd_flags="-wW -a 192.168.10.41"
jail_enable="YES"
jail_list="test1 test2 test3 test4 test5"
jail_set_hostname_allow="NO"
jail_exec_start="/bin/sh /etc/rc"
jail_exec_stop="/bin/sh /etc/rc.shutdown"
jail_devfs_enable="YES"
jail_test1_rootdir="/jails/test1"
jail_test1_hostname="test1.inside.mydomain.com"
jail_test1_ip="192.168.10.70"
jail_test2_rootdir="/jails/test2"
jail_test2_hostname="test2.inside.mydomain.com"
jail_test2_ip="192.168.10.71"
jail_test3_rootdir="/jails/test3"
jail_test3_hostname="test3.inside.mydomain.com"
jail_test3_ip="192.168.10.72"
jail_test4_rootdir="/jails/test4"
jail_test4_hostname="test4.inside.mydomain.com"
jail_test4_ip="192.168.10.73"
jail_test5_rootdir="/jails/test5"
jail_test5_hostname="test5.inside.mydomain.com"
jail_test5_ip="192.168.10.74"
################# netstat -r
Routing tables
Internet:
Destination Gateway Flags Refs Use Netif Expire
default gateway UGS 0 25149 em0
localhost localhost UH 0 1230 lo0
192.168.10 link#1 UC 0 0 em0
gateway link#1 UHLW 2 0 em0
scope 00:13:d3:95:44:80 UHLW 1 22959 em0 559
ns1 00:13:d3:95:43:6c UHLW 1 43 em0 980
ns2 00:13:d3:95:43:90 UHLW 1 25 em0 562
dev 00:11:25:57:e8:a1 UHLW 1 0 lo0
test1 00:13:46:e7:04:bc UHLW 1 24 em0 980
test2 00:13:46:e7:04:bc UHLW 1 24 em0 978
test3 00:13:46:e7:04:bc UHLW 1 24 em0 981
test4 00:13:46:e7:04:bc UHLW 1 24 em0 981
test5 00:13:46:e7:04:bc UHLW 1 24 em0 981
Internet6:
Destination Gateway Flags Netif Expire
localhost.inside localhost.inside UHL lo0
fe80::%lo0 fe80::1%lo0 U lo0
fe80::1%lo0 link#4 UHL lo0
ff01:4:: fe80::1%lo0 UC lo0
ff02::%lo0 fe80::1%lo0 UC lo0
######################## ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=b<RXCSUM,TXCSUM,VLAN_MTU>
inet 192.168.10.41 netmask 0xffffff00 broadcast 192.168.10.255
ether 00:11:25:57:e8:a1
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
vr0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet 192.168.10.70 netmask 0xffffff00 broadcast 192.168.10.255
inet 192.168.10.71 netmask 0xffffff00 broadcast 192.168.10.255
inet 192.168.10.72 netmask 0xffffff00 broadcast 192.168.10.255
inet 192.168.10.73 netmask 0xffffff00 broadcast 192.168.10.255
inet 192.168.10.74 netmask 0xffffff00 broadcast 192.168.10.255
ether 00:13:46:e7:04:bc
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
plip0: flags=108810<POINTOPOINT,SIMPLEX,MULTICAST,NEEDSGIANT> mtu 1500
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
Thanks
Jeff
More information about the freebsd-stable
mailing list