Jail management

[Miroslav Lachman]

Aristedes Maniatis wrote on 02/22/2016 12:26:
> On 22/02/2016 9:56pm, Miroslav Lachman wrote:
>> I don't know your environment and your FreeBSD jails skills but it seems you think jails are something complex and "magic". It is not.
> ...
>> Just don't be afraid of writing simple shell scripts :)
>
>
> You are right, and perhaps I should just bite the bullet. I am afraid of only two things.
>
> * upgrade the basejail with FreeBSD upgrades. I am sure this is a simple bit of chroot magic, but freebsd-update is a bit of a black box to me.

I tried it few years ago and it had some problems that doesn't fit well 
in to my environment, then I moved all our servers to own buildserver 
with make buildkernel + buildworld and then installworld through NFS in 
to destionation. Faster, safer and predictable solution.
(I had problems with freebsd-update even on bare metal systems, not in 
jails)

> * nullfs. I've never used it before and need to play with it more

Nullfs is easy. You can "mount" one directory to another.

If you have /vol0/jail/_basejail and jails in /vol0/jail/alpha, 
/vol0/jail/beta
Then you can do
mkdir /vol0/jail/alpha/basejail
mkdir /vol0/jail/beta/basejail

mount -t nullfs /vol0/jail/_basejail /vol0/jail/alpha/basejail
mount -t nullfs /vol0/jail/_basejail /vol0/jail/beta/basejail

Your basejail contains

# ls -1 /vol0/jail/_basejail
UPDATED
bin
boot
lib
libexec
rescue
sbin
usr

an jails (alpha, beta and you new jail template) contains symlinks to 
these directories

# ls -lg /vol0/jail/alpha/
-rw-r--r--   1 root  wheel   798 Jan 13  2015 .cshrc
-rw-r--r--   2 root  wheel   265 Jan 13  2015 .profile
-r--r--r--   1 root  wheel  6197 May 12  2015 COPYRIGHT
drwxr-xr-x   9 root  wheel    10 May 12  2015 basejail
lrwxr-xr-x   1 root  wheel    13 Jan 13  2015 bin -> /basejail/bin
lrwxr-xr-x   1 root  wheel    14 Jan 13  2015 boot -> /basejail/boot
dr-xr-xr-x   7 root  wheel   512 Oct 18 17:52 dev
lrwxr-xr-x   1 root  wheel    12 Jan 20  2015 develop -> /usr/develop
drwxr-xr-x  20 root  wheel   105 Nov 12 19:37 etc
lrwxr-xr-x   1 root  wheel     8 Jan 13  2015 home -> usr/home
lrwxr-xr-x   1 root  wheel    13 Jan 13  2015 lib -> /basejail/lib
lrwxr-xr-x   1 root  wheel    17 Jan 13  2015 libexec -> /basejail/libexec
dr-xr-xr-x   2 root  wheel     2 Jan 13  2015 proc
lrwxr-xr-x   1 root  wheel    16 Jan 13  2015 rescue -> /basejail/rescue
drwxr-xr-x  10 root  wheel    29 May 12  2015 root
lrwxr-xr-x   1 root  wheel    14 Jan 13  2015 sbin -> /basejail/sbin
lrwxr-xr-x   1 root  wheel    11 Jan 13  2015 sys -> usr/src/sys
drwxrwxrwt   9 root  wheel    10 Feb 22 03:43 tmp
drwxr-xr-x   7 root  wheel    17 Jan 20  2015 usr
drwxr-xr-x  22 root  wheel    22 Oct 18 17:52 var


Nullfs mounts can be specified in fstab files

# cat /etc/fstab.alpha
/vol0/jail/_basejail /vol0/jail/alpha/basejail nullfs ro 0 0

# cat /etc/fstab.beta
/vol0/jail/_basejail /vol0/jail/beta/basejail nullfs ro 0 0


So if jails are running, you wil see this

tank/vol0/jail/alpha on /vol0/jail/alpha (zfs, local, noatime, nfsv4acls)
tank/vol0/jail/beta on /vol0/jail/beta (zfs, local, noatime, nfsv4acls)

/vol0/jail/_basejail on /vol0/jail/alpha/basejail (nullfs, local, read-only)
/vol0/jail/_basejail on /vol0/jail/beta/basejail (nullfs, local, read-only)

And you can have gamma with another basejail called _basejail93 mounted as

tank/vol0/jail/gamma on /vol0/jail/gamma (zfs, local, noatime, nfsv4acls)

/vol0/jail/_basejail93 on /vol0/jail/gamma/basejail (nullfs, local, 
read-only)

Migrate this jail to _basejail is just a matter of change one line if 
fstab.gamma

All commong settings are in /etc/jail.conf

It can be something like this

## Typical static defaults:
## Use the rc scripts to start and stop jails.  Mount jail's /dev.
exec.start = "/bin/sh /etc/rc";
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.clean;
exec.system_user   = "root";
exec.jail_user     = "root";
mount.devfs;
devfs_ruleset      = 4;
enforce_statfs     = 1;
#allow.set_hostname = false;
#allow.mount;
allow.set_hostname = 0;
allow.sysvipc      = 0;
allow.raw_sockets  = 0;

## Dynamic wildcard parameter:
## Base the path off the jail name.
path            = "/vol0/jail/$name";
exec.consolelog = "/var/log/jail/$name.console";
mount.fstab     = "/etc/fstab.$name";

## Alpha
alpha {
         host.hostname = "alpha.example.com";
         ip4.addr      = 10.10.10.20;
         allow.sysvipc = 1;
}

## Beta
beta {
         host.hostname   = "beta.example.com";
         ip4.addr        = 10.10.10.30;
}

## Gamma
gamma {
         host.hostname   = "gamma.example.com";
         ip4.addr        = 10.10.10.40;
}


> As for shell scripts: my only goal in life is to write *fewer* shell scripts. My adoption of saltstack was spurred by shell everywhere, mostly not under version control. So less shell and more python centrally managed and versioned is my dream.

I understand this approach. You can look at it as your own port 
(package) and not as unversioned shell script.  :)


Miroslav Lachman