Jail management

[Aristedes Maniatis]

On 25/02/2016 7:52am, Mark Felder wrote:
> 
> 
> On Sun, Feb 21, 2016, at 19:13, Aristedes Maniatis wrote:

>> * It is hard to reproduce the environment exactly, matching the
>> application to the same version of Java that was available at the time of
>> deployment. Again I'm fighting against the pkg system which always wants
>> the latest version.
> 
> The package system *could* handle this, but it doesn't fit our design.
> We aren't like RedHat/Debian where we "freeze" packages at a certain
> version at the OS release and then backport only changes. With that
> method different versions of packages will just work with everything
> else in the system. With FreeBSD's ports system it's really a rolling
> release as the entire ports tree moves together. Mixing packages build
> from different checkouts of the ports tree is dangerous and not
> guaranteed to work.

Hi Mark

Yes, that makes sense and I've frequently struggled in Linux systems to get all the bits to match each other properly. So the FreeBSD solution here works.

But for me, where I use poudriere and roll my own custom packages, versioning becomes complicated. I'd need to either create a new poudriere jail for every release of my software, or go down the path of snap-shotting the entire jail (I'm choosing the second).


> I don't use ezjail. It doesn't upgrade well, and changes to the base
> jail require you stop all your jails. FreeBSD fat jails are so small
> (300MB?) it's not worth it in my opinion. I simply wrote a shell script
> to create fat jails and another script to handle updating them all.
> They're all treated like full servers/VMs, and configs/roles are managed
> with Ansible/Salt/etc.

That's a good point. And after all the excellent advice here, this is probably what I'll do:


1. Discard salt-minions inside every jail. That's become more trouble to look after as the number of jails grows. That's also really tricky to handle in salt when I want to fail over jails to another host. Since then every jail is running in two places and that confuses salt.

2. Create a single master/template jail. That might include the basejail or perhaps I'll keep the nullfs thing which works fine.

3. With every release of my software, upgrade that jail using 'pkg upgrade', test and 'zfs snapshot pool/template at v8.10'. I'll keep accumulating snapshots for every release, bundling up the changes to my software plus all the dependency packages and config.

4. When I want to upgrade a customer jail, I stop the jail and:

 # zfs destroy pool/customerJail
 # zfs clone pool/template at v8.10 pool/customerJail

I'll have salt help with automating this of course. By using the zfs clone command, even the 300Mb of FreeBSD userland takes zero bytes to clone. These commands should really take less than a second to execute, so the upgrade speed is great.

5. Then salt will write out the appropriate customer specific configuration into the newly cloned jail (for us that's just 2-3 files)

6. Start jail


Where having no basejail falls down is when the host system goes from FreeBSD 10 to 11, then I'll need to upgrade every jail but I'll have no way to go backwards to an older version once this is done since the old snapshot will include the old userland. That's probably not a problem for something that is rare.

Also, all the above works only because we use logstash and therefore don't care about losing all the logs inside each jail with every upgrade. I guess syslog would do the same thing for other people.


I hope this little summary helps other people facing similar challenges.


Ari


-- 
-------------------------->
Aristedes Maniatis
CEO, ish
https://www.ish.com.au
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-jail/attachments/20160225/8c66891d/attachment.sig>