From nobody Mon Feb 28 16:11:00 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A394719D9EFB; Mon, 28 Feb 2022 16:11:04 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-lj1-x22f.google.com (mail-lj1-x22f.google.com [IPv6:2a00:1450:4864:20::22f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K6ljg568Pz3F3S; Mon, 28 Feb 2022 16:11:03 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by mail-lj1-x22f.google.com with SMTP id r20so18120829ljj.1; Mon, 28 Feb 2022 08:11:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=Ba9jPzhu9XMVHjHyx/w8NF08TqK9t+60oP+1ZhuShXo=; b=ILDAw1CZVjI/20mJhQLHSv6wNbjYUlqnuImzlXNYiHKK808/Ys/v4B8QEcZjb0l8f6 Et65oh+DqEGg/iy9AfQ5WIAb7hQzbIYUVwi0B4EGDOZYaexH7Oo3/IzZSe03mvv58xDY lXn9GB/xLIQILRAORxN0iSkHQP+FU9kje9ErIM5lq4mtVSD0NoowT154LDM/Dt+mPgg2 FCL9HZY0sO7TXyFShXHnZ0394zPo0OO49iEXl0LMu7LwL/gGrYGNVngRqXbAsNkhnm+b 6+LdoRK9Q/+gInsHXPWzzA3Cow/izhcG/h1p09tFmxB2/qGzNByB7Gu0t0jN9qfC8O5K iT1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=Ba9jPzhu9XMVHjHyx/w8NF08TqK9t+60oP+1ZhuShXo=; b=QCevoaCpUVavJWDF26+vxHBYUsR72muAuJP0WpnwEEFbW82xclkRbmoi1AmIHPEuzZ WM8/RgxdJUaGbfBRySaF9REJCcmXbV+79TXEAkddVJKDi8AGAf1brsLvxWr5vq7fGnpA JdBomCE8kpf7GWLVlJSawSMLQFzMp0ZrPYJyczUm/ZPX62qqbWzxjgOddnMHW/W8pIbD GkZ4sxHwGbH9W24UwmErXOpVzRza3zX/2JTHWfQjOqARq1omt3+SOKv/vJvoNihto2jq 8MdlWyf6YusL3zRsIg/UD2AWsDP1OlTb6wVbGAn7n7eIpgs58EaNaWr6VB46PQZ6eWE6 ALSQ== X-Gm-Message-State: AOAM533dt/B/XQi3BIAxYZvYWF76V8mhn8/IFw9FNPx5uTK8SZt0F63v GjTeW6wPDzwGGaXGKaNlfcJQIEfel3T5mImsZ+k= X-Google-Smtp-Source: ABdhPJxbMODoZrzDzaFe9QQCNTGzBFgJGekQjgYVNPQWoW+9NmjgWnP0N4U/dYjHfkwlY4l73OHyT979Pop7t8Fx56g= X-Received: by 2002:a2e:9cd5:0:b0:246:3ec0:7505 with SMTP id g21-20020a2e9cd5000000b002463ec07505mr15058457ljj.434.1646064661550; Mon, 28 Feb 2022 08:11:01 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Received: by 2002:a05:6520:4da0:b0:1a1:e0c6:12e5 with HTTP; Mon, 28 Feb 2022 08:11:00 -0800 (PST) In-Reply-To: <20220228161545.251fe0d8@hermann> References: <20220228161545.251fe0d8@hermann> From: Mateusz Guzik Date: Mon, 28 Feb 2022 17:11:00 +0100 Message-ID: Subject: Re: bastille : poudriere not working in jail: jail: jail:_set: Operation not permitted! To: FreeBSD User Cc: FreeBSD virtualization , FreeBSD CURRENT Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4K6ljg568Pz3F3S X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=ILDAw1CZ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2a00:1450:4864:20::22f as permitted sender) smtp.mailfrom=mjguzik@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22f:from]; SUBJECT_ENDS_EXCLAIM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization,freebsd-current]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N There is a number of places which can return EPERM, to paste an example: if (gotslevel) { if (slevel < ppr->pr_securelevel) { error = EPERM; goto done_deref_locked; } } if (gotchildmax) { if (childmax >= ppr->pr_childmax) { error = EPERM; goto done_deref_locked; } } if (gotenforce) { if (enforce < ppr->pr_enforce_statfs) { error = EPERM; goto done_deref_locked; } } I see in your config you have enforce_statfs = 1; , perhaps that's smaller than what the host has and in that case you would get the error. Ultimately, while cumbersome, you can add printf to the code to find out which case is giving you the error. The real thing to do here would be implement something like SET_ERROR from illumos, which would allow to immediately pin point where the problem is coming from. On 2/28/22, FreeBSD User wrote: > Hello folks, > > we run at least two poudriere build systems on recent CURRENT boxes and one > of these > poudriere build systems is working within a jail - setup via FreeBSD's > /etc/jail.conf and > by misusing the port ezjail for copying/deploying our self-compiled jail > binary. The > poudriere jail uses ZFS and is, to make it short, working like a charme. > > Now we try to setup another poudriere, but this time the base is XigmaNAS > 12.3.0.4/9009, > which is based upon 12.X-RELENG, utilizing "bastille". Bastille is up to > date (in terms > od the XigmaNAS plugin). > > Following the setup we used on the native CURRENT "jailed poudriere" builder > and also > following this reference (for those who want to check on this) > > https://www.mimar.rs/blog/host-your-own-services-with-freebsd-jails-part-3-poudriere > > which seems quite recent and with the exception, that we use "vnet" on all > of our systems > for jails and so does XigmaNAS. > > Starting a building process via poudriere ends up with > > > # poudriere bulk -p head -z default -j 123-amd64 -f > /usr/local/etc/poudriere.d/zeit4-default.pkglist [00:00:00] Creating the > reference > jail... done [00:00:01] Mounting system devices for 123-amd64-head-default > [00:00:01] Warning: Using packages from previously failed, or uncommitted, > build: > /mnt/poudriere/data/packages/123-amd64-head-default/.building [00:00:01] > Mounting ports > from: /mnt/poudriere/ports/head [00:00:01] Mounting packages from: > /mnt/poudriere/data/packages/123-amd64-head-default [00:00:01] Mounting > distfiles from: > /mnt/poudriere/ports/distfiles [00:00:01] Copying /var/db/ports from: > /usr/local/etc/poudriere.d/head-amd64-head-default-options [00:00:02] > Appending to > make.conf: /usr/local/etc/poudriere.d/make.conf /etc/resolv.conf -> > /mnt/poudriere/data/.m/123-amd64-head-default/ref/etc/resolv.conf [00:00:02] > Starting > jail 123-amd64-head-default jail: jail_set: Operation not permitted > [00:00:02] Cleaning up > [00:00:02] Unmounting file systems > > poudriere jail -l: > > # poudriere jail -l > JAILNAME VERSION ARCH METHOD TIMESTAMP PATH > 123-amd64 12.3-RELEASE amd64 url=https://download.freebsd.org/releases/a ... > 3-RELEASE/ > 2022-02-24 14:14:25 /mnt/poudriere/jails/123-amd64 130-amd64 13.0-RELEASE > amd64 > url=https://download.freebsd.org/releases/a ... 0-RELEASE/ 2022-02-24 > 14:11:32 > /mnt/poudriere/jails/130-amd64 > > The jail.conf for this specific jail is as follows: > > [...] > pulverfass-001 { > devfs_ruleset = 13; > enforce_statfs = 1; > exec.clean; > exec.consolelog = /mnt/extensions/bastille/logs/pulverfass-001_console.log; > exec.start = '/bin/sh /etc/rc'; > exec.stop = '/bin/sh /etc/rc.shutdown'; > host.hostname = XXXXXXXXX; > mount.devfs; > mount.fstab = /mnt/extensions/bastille/jails/pulverfass-001/fstab; > path = /mnt/extensions/bastille/jails/pulverfass-001/root; > securelevel = 0; > > vnet; > vnet.interface = e0b_bastille4; > exec.prestart += "jib addm bastille4 igb0"; > exec.prestart += "ifconfig e0a_bastille4 description \"vnet host interface > for Bastille > jail pulverfass-001\""; exec.poststop += "jib destroy bastille4"; > > allow.mount; > allow.mount.fdescfs; > allow.mount.devfs; > allow.mount.tmpfs; > allow.mount.nullfs; > allow.mount.procfs; > allow.mount.linsysfs; > allow.mount.linprocfs; > allow.mount.zfs; > > allow.chflags; > allow.raw_sockets; > allow.socket_af; > allow.sysvipc; > > linux = new; > > exec.created += "/sbin/zfs jail ${name} BUNKER00/poudriere"; > exec.start += "/sbin/zfs mount -a"; > exec.poststop += "/sbin/zfs unjail BUNKER00/poudriere"; > > } > [...] > > Tracking the execution of the build process by issuing > > poudriere -x bulk ... > > and examin the resulting trace doesn' tgive me any hint, the error reported > above > immediately occurs when the jail is about to be started: > > + set -u +x > + jail -c persist 'name=123-amd64-head-default' > 'path=/mnt/poudriere/data/.m/ \ > 123-amd64-head-default/ref' 'host.hostname=basehost.local.domain' \ > 'ip4.addr=127.0.0.1' 'ip6.addr=::1' allow.chflags allow.sysvipc > jail: jail_set: Operation not permitted > + exit_handler > [...] > > Searching the net revealed some issues with setting IP4 and IP6 in > poudriere, but those > findings are dated back to 2017 and 2014 and I guess this is solved right > now. > > The difference between our manually jail.conf driven setup and the > XigmaNAS/bastille > based one is, bastille uses jib/netgraph based seutups of the vnet and the > ip4/ip6 is > setup from rc.conf, while we use epair in the other world and the ip is > setup from > withing the jail definition in jail.conf. > > I'm out of ideas here and after two days of trial and error and trying to > understand > what's going on lost ... Any hints or tipps? > > Thanks in advance, > > O. Hartmann > > -- Mateusz Guzik