Changes in Jails from FreeBSD 6 to FreeBSD 9 -- particularly, networking and routing

[Chad Leigh Shire.Net LLC]

Hi All

OK, so I have a server that has been running FreeBSD 6.1 and a bunch of jails, providing a few limited services.  I am migrating these from real hardware and FreeBSD 6.1 with jail running, to a Xen based VPS running FreeBSD 9.0-R with a kernel rebuild from a GENERIC kernel to GENERIC plus the Xen pci device.  There is one network device on the new server and it shares all addresses and the default route goes out it.

Because jails in FBSD 6 shared a network stack, I could have a public network x.x.x.0/24 and public address on the host machine, and a default route in that network as well, and use a 192.168.1.0/24 address aliased on the same network interface as the IP for my jail.  When doing that, from inside the jail, I could still reach the internet since it shared the route with the underlying  machine.


That seems to have changed on FBSD 9.  Now, if I add in the 192.168.1.0/24 address and run a jail on it, with the host machine in a public network/address/route as described above, from inside the jail I CANNOT reach the internet (it is not a resolver issue as services going to numeric addresses also fail).   However, the jail with the private 192.168.1.0/24 address CAN reach the host machines services even if it cannot get out onto the internet.  And the HOST machine can access services on the jail running on the private IP address.

(The purpose of the jail is to provide services to other jails and hosts on the same public network [all VPS on the same public vlan] and NOT to provide services to the internet.  Things like local ldap or a local dns etc.  But the private jail still needs to reach the internet for things like name servers it needs to access that are outside of the public network the host lives in.  So I don't care if the internet itself can reach the private jail, just the local jails and hosts it co-exists with.   The answer shouldn't be natd etc (was not needed in 6.0 and I am not sharing one public address with a range of private jails behind it).



If I launch the jail with an address from the same public range as the host, it works fine.  The jail can access the internet fine and vice versa.  The host can access the jail services as well.

If I launch the jail with a private address, the jail cannot reach the internet.  It can reach the host in the public network, but not other machines in the same public network (ie, the other VPS I have running which are all in the same public network).

If I launch the jail with both a private address and a public address, it can reach the internet and other VPS on the same public network.  I may have to end up doing that and just not having any services run on the public IP but I'd rather avoid using up an address like that.

What changes happened in the jails between FBSD 6 and FBSD 9 that would give the symptoms I have been experiencing?

Thanks
Chad