Using jail.conf array parameters in exec.* commands

[Ernie Luzar]

On release 10.3 or maybe 10.2 there were Devin Teske's examples of 
vnet/netgraph setups in these directories.
/usr/share/examples/netgraph
/usr/share/examples/jails

11.0 does not have them any more.

If I recall correctly, some of the netgraph commands need the jid 
number, which means the vnet jail has to be started. But here "started" 
does not have the normal meaning. A vnet jail can be created, jail -c, 
and as long as there is no "exec.start sh /etc/rc" command in the 
jail.conf definition there is nothing running in the jail. Now your 
script has a jid to use and can setup the netghaph world that will allow 
the vnet jail to commutate. After all that stuff is complete, a jexec 
command can be issued to start services in the jail. "sh /etc/rc" is all 
you need to fire up services in the vnet jail. Any userland application 
like apache have to be installed the normal way and have its 
apache24_enable="YES" statement id the vnet's jail rc.conf file.

Here's a general observation. During the 10.x releases I had a netgraph 
vnet script that Devin Teske posted to questions or jail list that 
worked. I think it was the same one that was in 10.x 
/usr/share/examples/jails directory. But when I tried it on 11.0 it 
stopped functioning giving a error on some netctl command. For me 
netgraph is just to hard. That is why I use the bridge/epair method.




Jeff Kletsky wrote:
> Thanks again for your thoughts on this.
> 
> I *am* using "pure" jail(8) and jail.conf(5) techniques and have been
> for many years now.  What I'd like to get to is a robust way to start
> jails the way I have been
> 
>     # jail -c some_jail
> 
> and just have it work reliably, especially when there is setup that
> needs to be done on the host to enable the smooth running of the jail.
> 
> Preferably, *all* configuration of the jail and its connectivity is
> done in jail.conf, be it /etc/jail.conf or a jail-specific one.  I'm
> trying to avoid going back to having to define another service that
> wraps a call to jail(8) which would just start the jail with persist
> set, then have to set up networking in the wrapper, jexec /etc/rc,
> and then deal with shutdown of a jail created with persist set.
> 
> I agree that *usually* the reason a jail won't start is
> misconfiguration. However, there are other conditions that can occur,
> such as lack of resources. If you want to more gracefully respond to
> this, the wrapper script would need either to parse the appropriate
> jail.conf, or to have all the pertinent information available in
> another form.  Having individual jail.conf files for each jail at
> least makes easier to parse, at the expense of not being able to
> define global and regional jail properties that are inherited across
> the appropriate jails.
> 
> 
> The network has to be up and connected *before* /etc/rc runs,
> especially where services in the jail need network interfaces present
> to bind to specific addresses, to mount network file systems, or have
> access to critical services, such as DNS (for example, nginx will fail
> to start if it can't resolve proxy host names).
> 
> Since there is no jail vnet or jail ID (number) available, you can't
> have jail(8) run needed operations in the jail.conf-declared
> exec.prestart command, including, for example:
> 
>     * ifconfig interface vnet jail
>     * ipfw add action proto from src to dst jail prisonID
>     * ipfw add lookup jail table_name
> 
> By the time jail(8) will run the jail.conf declared exec.poststart,
> exec.start has already run to completion inside the jail.
> 
> 
> How do you handle getting the network up *before* /etc/rc or the
> specific service is started in the jail?
> 
> 
> I unfortunately suspect you're right that I can't use the existing
> jail(8) and jail.conf(5) approach without wrapping the whole thing in
> a script.  The hooks, even for networking, don't seem to be there.
> 
> 
> Jeff
> 
> 
> 
> On 2/17/17 3:01 PM, Ernie Luzar wrote:
> 
>  >Lets make this simple. Do not use the "service jail jailname start"
>  >command to start / stop your jails.
> 
>  >Your mixing legacy rc.conf jail method with jail.conf method. All
>  >ways use the jail(8) command itself to start/stop your jails. If you
>  >do this in a script then you can check the jail resulting return code
>  >to determine if the jail start/stop failed. But there is no
>  >information to tell you why it failed. In all most all cases it's
>  >caused by jail.conf parameters syntax coding error or invalid value
>  >content. Really pretty simple to determine cause by looking at the
>  >jail.conf content for the offending vnet jail.
> 
>  >Change your mind set from thinking you have to use the exec.* hooks
>  >to configure the vnet jails netgraph network setup.
> 
>  >Just have individual jail.conf files for each vnet jail with no vnet
>  >interface defined.
> 
>  >Now you can start the jail with just the standard exec.start line and
>  >standard exec.stop line. Once your script has issued the jail(8)
>  >command to start the jail then follow it with all the netgraph
>  >commands to enable its network. The vnet jail it self has no
>  >knowledge of any network connectivity at start up, you can wrap
>  >either bridge/epair or netgraph around it and it don't care.
> 
>  >This was learned the hard way.
> 
>  > Jeff Kletsky wrote:
>  >> Thanks for the suggestion of trying to use 'ifconfig interface vnet 
> jail'
>  >> in the scripts themselves.
>  >>
>  >> I'll get my scripts up once I've got them running again confidently
>  >> and can get proper licensing on them.
>  >>
>  >> TL;DR
>  >>
>  >> * Is there a clean way to "catch" failures in jail(8) creation after
>  >>   exec.prestart completes, such as vnet.interface failing?
>  >>
>  >> * Is there a good way to execute commands in the host environment once
>  >>   jail(8) brings up the jail, but before exec.start runs?
>  >> [...]
>