From nobody Mon Feb 28 16:11:27 2022 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id C744E19DB893; Mon, 28 Feb 2022 16:11:43 +0000 (UTC) (envelope-from grembo@freebsd.org) Received: from mail.evolve.de (mail.evolve.de [213.239.217.29]) (using TLSv1.3 with cipher TLS_CHACHA20_POLY1305_SHA256 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "mail.evolve.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K6lkP6DKhz3Fdr; Mon, 28 Feb 2022 16:11:41 +0000 (UTC) (envelope-from grembo@freebsd.org) Received: by mail.evolve.de (OpenSMTPD) with ESMTP id 051ec02a; Mon, 28 Feb 2022 16:11:32 +0000 (UTC) Received: by mail.evolve.de (OpenSMTPD) with ESMTPSA id d2f374be (TLSv1.3:AEAD-CHACHA20-POLY1305-SHA256:256:NO); Mon, 28 Feb 2022 16:11:29 +0000 (UTC) Date: Mon, 28 Feb 2022 17:11:27 +0100 From: Michael Gmelin To: FreeBSD User Cc: FreeBSD virtualization , FreeBSD CURRENT Subject: Re: bastille : poudriere not working in jail: jail: jail:_set: Operation not permitted! Message-ID: <20220228171127.2469a57d.grembo@freebsd.org> In-Reply-To: <20220228161545.251fe0d8@hermann> References: <20220228161545.251fe0d8@hermann> X-Face: $wrgCtfdVw_H9WAY?S&9+/F"!41z'L$uo*WzT8miX?kZ~W~Lr5W7v?j0Sde\mwB&/ypo^}> +a'4xMc^^KroE~+v^&^#[B">soBo1y6(TW6#UZiC]o>C6`ej+i Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAJFBMVEWJBwe5BQDl LASZU0/LTEWEfHbyj0Txi32+sKrp1Mv944X8/fm1rS+cAAAACXBIWXMAAAsTAAAL EwEAmpwYAAAAB3RJTUUH3wESCxwC7OBhbgAAACFpVFh0Q29tbWVudAAAAAAAQ3Jl YXRlZCB3aXRoIFRoZSBHSU1QbbCXAAAAAghJREFUOMu11DFvEzEUAGCfEhBVFzuq AKkLd0O6VrIQsLXVSZXoWE5N1K3DobBBA9fQpRWc8OkWouaIjedWKiyREOKs+3PY fvalCNjgLVHeF7/3bMtBzV8C/VsQ8tecEgCcDgrzjekwKZ7TwsJZd/ywEKwwP+ZM 8P3drTsAwWn2mpWuDDuYiK1bFs6De0KUUFw0tWxm+D4AIhuuvZqtyWYeO7jQ4Aea 7jUqI+ixhQoHex4WshEvSXdood7stlv4oSuFOC4tqGcr0NjEqXgV4mMJO38nld4+ xKNxRDon7khyKVqY7YR4d+Cg0OMrkWXZOM7YDkEfKiilCn1qYv4mighZiynuHHOA Wq9QJq+BIES7lMFUtcikMnkDGHUoncA+uHgrP0ctIEqfwLHzeSo+eUA66AqzwN6n 2ZHJhw6Qh/PoyC/QENyEyC/AyNjq74Bs+3UH0xYwzDUC4B97HgLocg1QLYgDDO1v f3UX9Y307Ew4AHh67YAFFsxEpkXwpXY3eIgMhAAE3R19L919nNnuD2wlPcDE3UeT L2ytEICQib9BXgS2fU8PrD82ToYO1OEmMSnYTjSqSv9wdC0tPYC+rQRQD9ESnldF CyqfmiYW+tlALt8gH2xrMdC/youbjzPXEun+/ReXsMCDyve3dZc09fn2Oas8oXGc Jj6/fOeK5UmSMPmf/jL+GD8BEj0k/Fn6IO4AAAAASUVORK5CYII= List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4K6lkP6DKhz3Fdr X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=softfail (mx1.freebsd.org: 213.239.217.29 is neither permitted nor denied by domain of grembo@freebsd.org) smtp.mailfrom=grembo@freebsd.org X-Spamd-Result: default: False [-1.17 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEFALL_USER(0.00)[grembo]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_MEDIUM(-0.50)[-0.495]; NEURAL_HAM_LONG(-0.58)[-0.576]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; R_SPF_SOFTFAIL(0.00)[~all:c]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MID_CONTAINS_FROM(1.00)[]; SUBJECT_ENDS_EXCLAIM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current,freebsd-virtualization]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:213.239.192.0/18, country:DE]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On Mon, 28 Feb 2022 16:15:45 +0100 FreeBSD User wrote: > Hello folks, > > we run at least two poudriere build systems on recent CURRENT boxes > and one of these poudriere build systems is working within a jail - > setup via FreeBSD's /etc/jail.conf and by misusing the port ezjail > for copying/deploying our self-compiled jail binary. The poudriere > jail uses ZFS and is, to make it short, working like a charme. > > Now we try to setup another poudriere, but this time the base is > XigmaNAS 12.3.0.4/9009, which is based upon 12.X-RELENG, utilizing > "bastille". Bastille is up to date (in terms od the XigmaNAS plugin). > > Following the setup we used on the native CURRENT "jailed poudriere" > builder and also following this reference (for those who want to > check on this) > > https://www.mimar.rs/blog/host-your-own-services-with-freebsd-jails-part-3-poudriere > > which seems quite recent and with the exception, that we use "vnet" > on all of our systems for jails and so does XigmaNAS. > > Starting a building process via poudriere ends up with > > > # poudriere bulk -p head -z default -j 123-amd64 -f > /usr/local/etc/poudriere.d/zeit4-default.pkglist [00:00:00] Creating > the reference jail... done [00:00:01] Mounting system devices for > 123-amd64-head-default [00:00:01] Warning: Using packages from > previously failed, or uncommitted, build: > /mnt/poudriere/data/packages/123-amd64-head-default/.building > [00:00:01] Mounting ports from: /mnt/poudriere/ports/head [00:00:01] > Mounting packages from: > /mnt/poudriere/data/packages/123-amd64-head-default [00:00:01] > Mounting distfiles from: /mnt/poudriere/ports/distfiles [00:00:01] > Copying /var/db/ports from: > /usr/local/etc/poudriere.d/head-amd64-head-default-options [00:00:02] > Appending to make.conf: /usr/local/etc/poudriere.d/make.conf > /etc/resolv.conf -> > /mnt/poudriere/data/.m/123-amd64-head-default/ref/etc/resolv.conf > [00:00:02] Starting jail 123-amd64-head-default jail: jail_set: > Operation not permitted [00:00:02] Cleaning up [00:00:02] Unmounting > file systems > > poudriere jail -l: > > # poudriere jail -l > JAILNAME VERSION ARCH METHOD TIMESTAMP PATH > 123-amd64 12.3-RELEASE amd64 > url=https://download.freebsd.org/releases/a ... 3-RELEASE/ 2022-02-24 > 14:14:25 /mnt/poudriere/jails/123-amd64 130-amd64 13.0-RELEASE amd64 > url=https://download.freebsd.org/releases/a ... 0-RELEASE/ 2022-02-24 > 14:11:32 /mnt/poudriere/jails/130-amd64 > > The jail.conf for this specific jail is as follows: > > [...] > pulverfass-001 { > devfs_ruleset = 13; > enforce_statfs = 1; > exec.clean; > exec.consolelog = > /mnt/extensions/bastille/logs/pulverfass-001_console.log; exec.start > = '/bin/sh /etc/rc'; exec.stop = '/bin/sh /etc/rc.shutdown'; > host.hostname = XXXXXXXXX; > mount.devfs; > mount.fstab = /mnt/extensions/bastille/jails/pulverfass-001/fstab; > path = /mnt/extensions/bastille/jails/pulverfass-001/root; > securelevel = 0; > > vnet; > vnet.interface = e0b_bastille4; > exec.prestart += "jib addm bastille4 igb0"; > exec.prestart += "ifconfig e0a_bastille4 description \"vnet host > interface for Bastille jail pulverfass-001\""; exec.poststop += "jib > destroy bastille4"; > > allow.mount; > allow.mount.fdescfs; > allow.mount.devfs; > allow.mount.tmpfs; > allow.mount.nullfs; > allow.mount.procfs; > allow.mount.linsysfs; > allow.mount.linprocfs; > allow.mount.zfs; > > allow.chflags; > allow.raw_sockets; > allow.socket_af; > allow.sysvipc; > > linux = new; > > exec.created += "/sbin/zfs jail ${name} BUNKER00/poudriere"; > exec.start += "/sbin/zfs mount -a"; > exec.poststop += "/sbin/zfs unjail BUNKER00/poudriere"; > > } > [...] > > Tracking the execution of the build process by issuing > > poudriere -x bulk ... > > and examin the resulting trace doesn' tgive me any hint, the error > reported above immediately occurs when the jail is about to be > started: > > + set -u +x > + jail -c persist 'name=123-amd64-head-default' > 'path=/mnt/poudriere/data/.m/ \ 123-amd64-head-default/ref' > 'host.hostname=basehost.local.domain' \ 'ip4.addr=127.0.0.1' > 'ip6.addr=::1' allow.chflags allow.sysvipc jail: jail_set: Operation > not permitted > + exit_handler > [...] > > Searching the net revealed some issues with setting IP4 and IP6 in > poudriere, but those findings are dated back to 2017 and 2014 and I > guess this is solved right now. > > The difference between our manually jail.conf driven setup and the > XigmaNAS/bastille based one is, bastille uses jib/netgraph based > seutups of the vnet and the ip4/ip6 is setup from rc.conf, while we > use epair in the other world and the ip is setup from withing the > jail definition in jail.conf. > > I'm out of ideas here and after two days of trial and error and > trying to understand what's going on lost ... Any hints or tipps? > > Thanks in advance, > > O. Hartmann Hi Oliver, I don't see `children.max` set in any of the configuration you shared above. Cheers Michael -- Michael Gmelin