Jail fails to unmount a directory

[Ernie Luzar]

list+org.freebsd.jail at io7m.com wrote:
> On 2016-10-31T18:45:31 +0000
> list+org.freebsd.jail at io7m.com wrote:
> 
>> On 2016-10-31T14:07:16 -0400
>> Ernie Luzar <luzar722 at gmail.com> wrote:
>>> The correct way to stop the jail is to issue this command from the host 
>>> command line
>>>
>>> host# jail -v -f com.example.service0.conf -r com_example_service0  
>> Hello.
>>
>> The problem with this is that jails can end up in this broken, undead
>> state. It seems rather nasty that the created process could crash,
>> leaving a nonexistent jail with no processes that nevertheless can't be
>> restarted without first "destroying" it, even though it appears to have
>> already gone! How is an administrator even supposed to know that a jail
>> that doesn't appear in the output of `jls` must be destroyed before it
>> can be started?
>>

If after issuing the jail start command, "jls" doesn't show the jail 
running, that is how the administrator KNOWS the jails jail.conf has an 
error and he has to manually inspect things to identify the error and 
fix things that were completed before the jail died, so it can be 
cleanly started again.

> 
> And, in fact, having just tried it, it doesn't work:
> 
> host# jail -v -f com.example.service0.conf -c com_example_service0
> com_example_service0: run command: /sbin/ifconfig em0 inet 127.0.0.2 netmask 255.255.255.255 alias
> com_example_service0: run command: /sbin/mount -t nullfs -o ro,noauto /usr/jail/base /usr/jail/com.example.service0/base
> com_example_service0: jail_set(JAIL_CREATE) persist name=com_example_service0 host.hostname=com.example.service0 ip4.addr=127.0.0.2 path=/usr/jail/com.example.service0
> com_example_service0: created
> com_example_service0: run command in jail: /bin/sh
> # exit
> com_example_service0: jail_set(JAIL_UPDATE) jid=14 nopersist
> 
> host# jail -v -f com.example.service0.conf -r com_example_service0
> jail: "com_example_service0" not found
> 
> host# jail -v -f com.example.service0.conf -c com_example_service0
> com_example_service0: run command: /sbin/ifconfig em0 inet 127.0.0.2 netmask 255.255.255.255 alias
> com_example_service0: run command: /sbin/mount -t nullfs -o ro,noauto /usr/jail/base /usr/jail/com.example.service0/base
> mount_nullfs: /usr/jail/com.example.service0/base: Resource deadlock avoided
> jail: com_example_service0: /sbin/mount -t nullfs -o ro,noauto /usr/jail/base /usr/jail/com.example.service0/base: failed
> com_example_service0: run command: /sbin/ifconfig em0 inet 127.0.0.2 netmask 255.255.255.255 -alias
> 
> host#

What is that # exit   doing there?

Your jail is not being completely started. It finds something wrong with 
the jail(8) jail.conf definition options and dies.

Do a jls command before and after starting the jail to help document 
what your doing.

The jail console log does not show a log of what the jail is doing 
during startup. This has been a problem since the very beginning of the 
jail function in Freebsd.

I think your last test above still had the nullfs file mounted. You have 
to manually dismount it before trying to start the jail again. A host 
reboot will do that for you if umount command does not work.

Try changing The /usr/jail/com.example.service0.fstab from

/usr/jail/base /usr/jail/com.example.service0/base nullfs ro,noauto 0 0

to
/usr/jail/base /usr/jail/com.example.service0/base nullfs ro 0 0

In your first post you said
"/usr/jail/base is a directory containing binaries.". How did you 
populate this directory and with what?

The "jail -v -f com.example.service0.conf -r com_example_service0" will 
work if the jail in question is really running and shows in the jls output.

I suggest you try using sysutils/qjail port for your jail admin utility. 
It's very user friendly. qjail2 for 9.x systems or qjail4 for 10.x 
systems or qjail for 11.x systems. "pkg install" command is even quicker.