deploy multiple vnets with VIMAGE/VNET + Production Ready?

[Ernie Luzar]

Roger Marquis wrote:
> Ernie Luzar wrote:
>> the kernel to included vimage. Enabling pf or ipf firewalls cause the
>> host to crash. ipfw firewall does not cause a crash but has next to no
>> real life usage on vimage.
> 
> Considering we have had ipfw/vimage/netgraph jails for several years I'd
> be interested in your data sources.

The source is personal experience. Tested 9.3 & 10.0 with ipfw running 
in vnet/vimage jails. At that time ipfw was logging to the host and not 
to the vimage jail. Definitely a security violation.

You know I give you a lot of credit for risking things on vnet/vimage 
jails in your shop. Most management just wouldn't take that risk.

> 
>> When stopping vimage jails there is a problem with memory loss.
> 
> Have you tested this, on a recent release?

NO why would I when release notes didn't say anything about vimage 
changes or pf, ipf firewall becoming vimage aware.

> 
>> You need a high proficiency in coding netgraph which
>> is used to tie the hosts network to each vimage jail.
> 
> This certainly used to be true and IMO has been a significant barrier to
> netgraph usage but the scripts in head/share/examples/jails/ are
> at least helpful.
> 

I checked out those examples. Hardly any comments about what is 
happening or why their being done. All they are is a starting point to 
experiment doing trial and error testing

>> Needs a public network with multiple static ip address & registered 
>> domain names even to test it.
> 
> How are you implementing vimage that needs a registered domain name?
> 

Maybe the real question is how do you drive un-solicited public traffic 
to your vnet/vimage jail without them. The real point here is, are you 
talking about a production config or some home play ground? There is no 
need for a vnet/vimage jail setup just for some server on the lan 
restricted to local usage only. The power of vnet/image comes to shine 
when used by a ISP or hosting company. There you have customers with 
static ip address and domain names. They have what looks like a real 
FreeBSd system to use when in reality its just one jail of many.


>> There are a few write ups about how to configure vet/vimage jails, but
>> their out of date. IE: 8.x & 9.x releases which are at EOL [end of life,
>> unsupported].
> 
> Vimage gets little attention.  Unfortunately the mapping of non-vimage
> localhost interfaces to the primary external interface isn't noted 
> nearly enough either.  These are weaknesses in bsd jails, the latter a
> non-trivial security issue on many non-vimage systems considering
> daemons like sendmail are installed and listening on "localhost" by
> default.
> 

After learning the usage of the jail(8) command doing testing the manual 
way, I found it to be so tedious keeping all the many different jail 
config options and command formats in my head, mistakes were common. 
qjail changed all that. Its so user friendly. In qjail sendmail is 
disabled by default and the cron status reports run faster because all 
the sendmail status checks are turned off.

I disagree with you about the security issue of using localhost. Running 
sendmail in a non-vimage jail using its default config listening on 
localhost is still contained in the jail. Localhost is internally 
converted to the jails assigned ip address by jail(8). Why do you think 
this is a non-trivial security issue?

>> Going down this road will make the shop totally dependent on you and your
>> ability. A mega size pay bump is in your future. The shop will be 
>> fubar-ed
>> if you die or get hurt requiring a hospital stay and long recovery.
> 
> Potentially true of any Unix or Linux application in my experience.
> Have you tried vimage with epair/if_bridge instead of netgraph?  It's
> considerably simpler though the documentation is almost as conflicting
> and insufficient.
> 

Yes epair/if_bridge is way simpler, but far less flexible when you want 
to re-point your public network ip address to different jails as 
circumstances change. Yep netgraph documentation sucks big time.

My time for playing around is very limited. I'll wait for 11.0 to be 
published and see what the "release notes" say about vimage and the 
firewalls becoming vimage aware. Also will be checking the closed bugs 
for vimage to see what has been fixed. Then I will make up my mind about 
giving vimage another ride. But qjail will be the tool I use to perform 
the test ride.

http://freshbsd.org/search?branch=HEAD&project=freebsd&q=vimage+OR+vnet

shows 286 commits for vnet/vimage. This worries me that there has not 
been a call for vnet/vimage testers of -current. Just have to wait and 
see what happens. Maybe letting other vnet/vimage users lead the way 
with what is a bleeding edge version of vimage is the conservative way 
to approach this. I just think about zfs and how many releases 
containing zfs bug fixes before it became reliable. Its been many years 
and FreeBSD releases since vimage first became available as a kernel 
compile option. There is no way to know if vimage development will 
continue or even if bugs will be addresses. Vimage is not enjoying paid 
support.

I do hope vnet/vimage has finally become of age and reliable for 
production like the non-vimage jails have become.