some (random) jails are not started after reboot

[Miroslav Lachman]

James Gritton wrote on 2016/10/06 18:11:
> On 2016-10-06 06:01, Miroslav Lachman wrote:

[...]

>> This is production machine so I can't play with it too much. But does
>> anybody else seen it too?
>> How can I debug it on next reboot?
>
> Two handy things.  One is to make sure every jail has an exec.consolelog
> file set, so if it gets partway through startup and then goes away, you
> can see what it did up until then.  Second is the "-v" flag to jail(8).
> Add it to jail_flags in your rc.conf, and then comment out the bit in
> /etc/rc.d/jail that removes the temp file for jail(8)'s output ($_tmp).
> Watch out for parallel starts - they just went to /dev/null until recently.

Thank you for your advice.
I added -v flag and commented out rm $_tmp
We are not using parallel start.
I rebooted server tonight and another jail didn't start.

Nothing was logged to jail's console log file

But I found this in tmp file:

# grep costa jail.rwe8dsFF
costa: run command as root: /sbin/mount -t nullfs -o ro 
/vol0/jail/_basejail /vol0/jail/costa/basejail
mount_nullfs: /vol0/jail/costa/basejail: Operation not supported by device
jail: costa: /sbin/mount -t nullfs -o ro /vol0/jail/_basejail 
/vol0/jail/costa/basejail: failed


This is strange, because all jails are using the same _basejail and I 
can manually mount this directory

root at cage ~/# /sbin/mount -t nullfs -o ro /vol0/jail/_basejail 
/vol0/jail/costa/basejail

root at cage ~/# mount | grep costa
tank/vol0/jail/costa on /vol0/jail/costa (zfs, local, noatime, nfsv4acls)
/vol0/jail/_basejail on /vol0/jail/costa/basejail (nullfs, local, read-only)

root at cage ~/# umount /vol0/jail/costa/basejail


And I can start the jail by rc script as well


root at cage ~/# service jail start costa
Starting jails: costa.


Do you have any idea what is going on?

Miroslav Lachman