some (random) jails are not started after reboot

[James Gritton]

On 2016-10-11 18:46, Miroslav Lachman wrote:
> James Gritton wrote on 2016/10/06 18:11:
>> On 2016-10-06 06:01, Miroslav Lachman wrote:
> 
> [...]
> 
>>> This is production machine so I can't play with it too much. But does
>>> anybody else seen it too?
>>> How can I debug it on next reboot?
>> 
>> Two handy things.  One is to make sure every jail has an 
>> exec.consolelog
>> file set, so if it gets partway through startup and then goes away, 
>> you
>> can see what it did up until then.  Second is the "-v" flag to 
>> jail(8).
>> Add it to jail_flags in your rc.conf, and then comment out the bit in
>> /etc/rc.d/jail that removes the temp file for jail(8)'s output 
>> ($_tmp).
>> Watch out for parallel starts - they just went to /dev/null until 
>> recently.
> 
> Thank you for your advice.
> I added -v flag and commented out rm $_tmp
> We are not using parallel start.
> I rebooted server tonight and another jail didn't start.
> 
> Nothing was logged to jail's console log file
> 
> But I found this in tmp file:
> 
> # grep costa jail.rwe8dsFF
> costa: run command as root: /sbin/mount -t nullfs -o ro
> /vol0/jail/_basejail /vol0/jail/costa/basejail
> mount_nullfs: /vol0/jail/costa/basejail: Operation not supported by 
> device
> jail: costa: /sbin/mount -t nullfs -o ro /vol0/jail/_basejail
> /vol0/jail/costa/basejail: failed
> 
> 
> This is strange, because all jails are using the same _basejail and I
> can manually mount this directory
> 
> root at cage ~/# /sbin/mount -t nullfs -o ro /vol0/jail/_basejail
> /vol0/jail/costa/basejail
> 
> root at cage ~/# mount | grep costa
> tank/vol0/jail/costa on /vol0/jail/costa (zfs, local, noatime, 
> nfsv4acls)
> /vol0/jail/_basejail on /vol0/jail/costa/basejail (nullfs, local, 
> read-only)
> 
> root at cage ~/# umount /vol0/jail/costa/basejail
> 
> 
> And I can start the jail by rc script as well
> 
> 
> root at cage ~/# service jail start costa
> Starting jails: costa.
> 
> 
> Do you have any idea what is going on?

I don't know the cause, but I've seen similar problems before with 
nullfs mounts.  A possible  workaround is to replace the mount with an 
exec.prestart script that tries the mount and if it fails waits a few 
seconds and tries again (also an exec.poststop command to unmount it).  
I haven't tried such a thing myself, but then I've never had this 
problem either.  It might have to do with the main jail mount being zfs, 
but that's just a guess.

- Jamie