Jail management

[Martin "eto" Misuth]

On Mon, 22 Feb 2016 22:26:02 +1100
Aristedes Maniatis <ari at ish.com.au> wrote:

> 
> You are right, and perhaps I should just bite the bullet. I am afraid of only
> two things.
> 

Hi I am jails newcomer as well, and this is kinda report of how far I got.
But depending on how much you want bet on this technology, I strongly suggest
killing some serious time with jails. It's really worth it, if you intend to
use them more extensively on FreeBSD, eg. besides "standard" add few lines to
rc.conf using ezjail or whatever.

For all intent and purposes jail is actually just glorified chroot (although 
VNET ones are running separate TCP/IP stacks and I have not have had time to
look at them). And thus in actual usage jails are pretty "simple" as most things
on FreeBSD are (which is benefit). I considered them magical too in the past.

In the end it seems same base rules as for managing chroots apply. Although
chroot is with us for long time, I don't see much easily accessible
information on how to efficiently build and manage chroots themselves and large
numbers of chroots concurently. As such the rule is, if there are mentions of
chroot parameter in manpages of tool it can be used as both jail and chroot
(operating directly on jail tree) management tool.

Seems like at certain point most admins doing this kind of work absorb enough
knowledge to build chroots on their own, either by some custom scripts stack or
with help of package manager and other tools.

Several observations: 
 - you can assemble your jail tree at any location, although most setups pick
   certain path and use that as jail "hive"
 - if you have your given jail tree mounted in place, you can always chroot to
   it, without spawning it as jail (this might have security implications!)
   to test programs are dynamically linking together or whatever
 - if jail is offline, you can pkg "do" (install whatever) into it with chroot
   parameter (offline here is just to avoid locking and general clashing
   with "running" jail)
 - you can pkg "do" (install, whatever) into running jail as well, with jail
   parameter
 - you can use literally any mounted fs under jail's root, and unless you allow
   given jail to do mounts and stat for fs information, it sees it as one
   single / it cannot modify
 - not sure about Miroslav's problems with freebsd-update, but it seems to work
   pretty well with -basedir /jail/tree parameter nowadays (there might be
   corner cases)
 - you can have older jail-base run on newest kernel (other way around is not
   possible)
 - you can kill many files in given jail to get bare minimal running setup 
   (this seems completely driven by gut, from what I gathered, as some things
   might have un-obvious dependencies)
 - you can mount many things into jail read-only (this makes them more rigid
   and harder to "manage" "live")
 - jails can have limits on number of procs living in them and can be
   allowed to be nested(!) (jail-in-jail)
 - with rctl you can cap resources per jail
 - you don't need to run all daemons usually run on host jail
   - depending on your setup you can get rid of native syslog and use something
     lighter
 - jails have their own user databases, but if you do this, uids don't map
   to usernames when viewed from host (ps, htop etc)
   - imho simplest possible setup with "live" uid database (eg not copying
     files around) was for me to use nss ldap extension, that way all jails
     and host see same synced uids (if you store them in single ldap
     space, this might have security implications as users in other jails
     can see others in the system)
   - as ports tree uses same uids on every machine it made sense to me to import
     those into main ldap setup, then have per jail subtree with jail local
     users this requires from you to adjust nss ldap search rules

As the idea of jail is having chroot like tree with more protections, once you
add ZFS into the mix, number possibilities starts exploding :
 - by various ways of nesting your jailtree ZFS datasets 
   - you can get to instant "full jails snapshots". Once snapshotted, given
     your connection limitations and zfs skillset, jails can be sent over to 
     other zfs pools (within same machine or between multiple machines) by
     virtues of 'zfs send'
   - similarly you can implement automatic incremental snapshots
 - you can instruct zfs to attach given dataset to jail (from there jail can
   control it's own snapshots or other zfs operations from inside)
 - taking advantage of FreeBSD's / and /usr/local split you can reuse (by zfs
   clone) same "root" dataset in multiple jails and treat is as easily
   discardable
   - this allows you to keep /usr/local subtree and swap roots around
     after freebsd-update-ing template (by cloning)

With last most interesting option you can insert nullfs into the mix
this allows you to share some part of host tree within multiple jails.
Keep in mind nullfs-es don't mount recursively and this becomes managemnt 
issue with deeper mounts.

So now because it's so easy, when you made 60+ jails starts the fun
 - all these jails need love to keep them up-to date
 - packages in these jails need to be kept up-to date
 - fs partitioning or zfs nesting you picked might turn out to not be
   exactly optimal
 - if you use fstab option of jail.conf, unmounting and re-mounting
   might sometimes fail
 - some daemons (I suspect native syslog being one of them for me) might 
   leave jail in dying state because of what I guess, are TCP/IP and unix
   socket timeouts, this might, make "remount" with custom fstab 
   impossible in quick succession (after teardown)
 - as mentioned nullfs mounts don't nest so you might need some kind of
   script glue if you want to use that

So with all tools mentioned in this thread, you get many of those decisions made
for you and you can mostly use your hive only in the way tool's author
envisioned. 

The more you use it directly more convoluted and flexible setups are
possible provided you can manage them.

Generally, the more static (all jail mounts at system boot + persistent jails
with fixed jail ids) setup is, more reliably it operates, but more rigid it
becomes to change. 

I personally don't run heavy load hosts. My main use currently is to test
various combinations of http servers with different versions of php-s
with various versions of our php stuff.