Re: bastille : poudriere not working in jail: jail: jail:_set: Operation not permitted!

On Mon, 28 Feb 2022 16:15:45 +0100
FreeBSD User <freebsd@walstatt-de.de> wrote:

> Hello folks,
> 
> we run at least two poudriere build systems on recent CURRENT boxes
> and one of these poudriere build systems is working within a jail -
> setup via FreeBSD's /etc/jail.conf and by misusing the port ezjail
> for copying/deploying our self-compiled jail binary. The poudriere
> jail uses ZFS and is, to make it short, working like a charme.
> 
> Now we try to setup another poudriere, but this time the base is
> XigmaNAS 12.3.0.4/9009, which is based upon 12.X-RELENG, utilizing
> "bastille". Bastille is up to date (in terms od the XigmaNAS plugin).
> 
> Following the setup we used on the native CURRENT "jailed poudriere"
> builder and also following this reference (for those who want to
> check on this)
> 
> https://www.mimar.rs/blog/host-your-own-services-with-freebsd-jails-part-3-poudriere
> 
> which seems quite recent and with the exception, that we use "vnet"
> on all of our systems for jails and so does XigmaNAS.
> 
> Starting a building process via poudriere ends up with
> 
> 
> # poudriere bulk -p head -z default -j 123-amd64 -f
> /usr/local/etc/poudriere.d/zeit4-default.pkglist [00:00:00] Creating
> the reference jail... done [00:00:01] Mounting system devices for
> 123-amd64-head-default [00:00:01] Warning: Using packages from
> previously failed, or uncommitted, build:
> /mnt/poudriere/data/packages/123-amd64-head-default/.building
> [00:00:01] Mounting ports from: /mnt/poudriere/ports/head [00:00:01]
> Mounting packages from:
> /mnt/poudriere/data/packages/123-amd64-head-default [00:00:01]
> Mounting distfiles from: /mnt/poudriere/ports/distfiles [00:00:01]
> Copying /var/db/ports from:
> /usr/local/etc/poudriere.d/head-amd64-head-default-options [00:00:02]
> Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
> /etc/resolv.conf ->
> /mnt/poudriere/data/.m/123-amd64-head-default/ref/etc/resolv.conf
> [00:00:02] Starting jail 123-amd64-head-default jail: jail_set:
> Operation not permitted [00:00:02] Cleaning up [00:00:02] Unmounting
> file systems
> 
> poudriere jail -l:
> 
> # poudriere jail -l
> JAILNAME VERSION ARCH METHOD TIMESTAMP PATH
> 123-amd64 12.3-RELEASE amd64
> url=https://download.freebsd.org/releases/a ... 3-RELEASE/ 2022-02-24
> 14:14:25 /mnt/poudriere/jails/123-amd64 130-amd64 13.0-RELEASE amd64
> url=https://download.freebsd.org/releases/a ... 0-RELEASE/ 2022-02-24
> 14:11:32 /mnt/poudriere/jails/130-amd64
> 
> The jail.conf for this specific jail is as follows:
> 
> [...]
> pulverfass-001 {
> devfs_ruleset = 13;
> enforce_statfs = 1;
> exec.clean;
> exec.consolelog =
> /mnt/extensions/bastille/logs/pulverfass-001_console.log; exec.start
> = '/bin/sh /etc/rc'; exec.stop = '/bin/sh /etc/rc.shutdown';
> host.hostname = XXXXXXXXX;
> mount.devfs;
> mount.fstab = /mnt/extensions/bastille/jails/pulverfass-001/fstab;
> path = /mnt/extensions/bastille/jails/pulverfass-001/root;
> securelevel = 0;
> 
> vnet;
> vnet.interface = e0b_bastille4;
> exec.prestart += "jib addm bastille4 igb0";
> exec.prestart += "ifconfig e0a_bastille4 description \"vnet host
> interface for Bastille jail pulverfass-001\""; exec.poststop += "jib
> destroy bastille4";
> 
> allow.mount;
> allow.mount.fdescfs;
> allow.mount.devfs;
> allow.mount.tmpfs;
> allow.mount.nullfs;
> allow.mount.procfs;
> allow.mount.linsysfs;
> allow.mount.linprocfs;
> allow.mount.zfs;
> 
> allow.chflags;
> allow.raw_sockets;
> allow.socket_af;
> allow.sysvipc;
> 
> linux = new;
> 
> exec.created += "/sbin/zfs jail ${name} BUNKER00/poudriere";
> exec.start += "/sbin/zfs mount -a";
> exec.poststop += "/sbin/zfs unjail BUNKER00/poudriere";
> 
> }
> [...]
> 
> Tracking the execution of the build process by issuing
> 
> poudriere -x bulk ...
> 
> and examin the resulting trace doesn' tgive me any hint, the error
> reported above immediately occurs when the jail is about to be
> started:
> 
> + set -u +x
> + jail -c persist 'name=123-amd64-head-default'
> 'path=/mnt/poudriere/data/.m/ \ 123-amd64-head-default/ref'
> 'host.hostname=basehost.local.domain' \ 'ip4.addr=127.0.0.1'
> 'ip6.addr=::1' allow.chflags allow.sysvipc jail: jail_set: Operation
> not permitted
> + exit_handler
> [...]
> 
> Searching the net revealed some issues with setting IP4 and IP6 in
> poudriere, but those findings are dated back to 2017 and 2014 and I
> guess this is solved right now.
> 
> The difference between our manually jail.conf driven setup and the
> XigmaNAS/bastille based one is, bastille uses jib/netgraph based
> seutups of the vnet and the ip4/ip6 is setup from rc.conf, while we
> use epair in the other world and the ip is setup from withing the
> jail definition in jail.conf.
> 
> I'm out of ideas here and after two days of trial and error and
> trying to understand what's going on lost ... Any hints or tipps?
> 
> Thanks in advance,
> 
> O. Hartmann

Hi Oliver,

I don't see `children.max` set in any of the configuration you shared
above.

Cheers
Michael

-- 
Michael Gmelin