From nobody Fri Nov 15 23:30:46 2024 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XqtYf5TZ1z5dhsR; Fri, 15 Nov 2024 23:30:46 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XqtYf5Jdwz3xh3; Fri, 15 Nov 2024 23:30:46 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731713446; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=hmy+KSGSAlUPBycIgCrS7Un7GQcvOMvK3ZCk8afmNuE=; b=S/m3pzquGGokFLRWbJz1p3Nq2KtQ+muRdnMtDzXj99gIc03IOsBcBDXsRZkm7VIGg4OHPn a5StTzXi2lB4X3PVe2WTboAVPMRHKGFDVSYJBCvStm+q0aGjwFZ56LKU0wHNKyXaLjP8rP y/FkB5+ym02EUNjikL28dHTH69JywQkaViB0zpySOJFxOGwIxGdqt4iIDCBvua5blza730 wBIc6F2R0obeUP4luDqZdnAKb+MPiycDD9dvVY0ciXipJQhRojjGGUhamoFNS/pB7dFeqg TjyZOxZGkpdq0S7AD0AaiHbFB9l1J1Zz8nGh8qzcaAQblyvYra4Jrc868Cw80A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731713446; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=hmy+KSGSAlUPBycIgCrS7Un7GQcvOMvK3ZCk8afmNuE=; b=Wb7KycQMJa05xEHtG9rDRPUqghKA4uz12L9ZBSHaCaplvN+3bH8kDDtHu5gNkPB/aoO1YH skv1abJqenUVwpd+pYoiyXxP6RJuFRQZMIO/jjAj+rX6W3infPCV16M6O5jDHOg1RWOPrc 02pvIqWksYL309jJo/HPVy0S2Glm/LDqCMoNVhCr/2cZfDA2OhpILcqLf4ev1MLFPSTn0q 32pvRdUDbeJigObK8gurC/hNKqqAnVumXyQM5Km/DDHuho72wcxJMiWY7BOiyr30+z7XHA 7hVeXW5f4upK5BG0kuwbk3V/bG0YCzWEffxwjobWS3nxIiZr9PzAVCsGDVvCTg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731713446; a=rsa-sha256; cv=none; b=KlC75yrouUevCX95KxRmUoZSgpBs5klSPAM0LvwLIV0Nvatq2tR3ZSzm8cZk87RHmxTbiQ 5duzhvD3n1Jk7e0DTYPvVMOLb2VwZcCcP5gyKGfjNjdgLI+TyRv3XJuQ+gMYnF6LezlYcD WcydGoz9osTJoQTbT9cgIsJsiv/nc1Ao9E6pXgE60v9lIp2hsctnKADwOCWzAglKs5OtMP bLMskLmc5H7XNbRHoXupBR8ockcOWFKh9FSFOcGNPx0eE7dlAV/fqjlcFHhbDQxKgBYaLH 7xE0ep0HDD7FAYYq7vinnqvvNUe7l+7GSRexDk72z3Sv8lNplPGd+aOUxrChRA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XqtYf4s63zk42; Fri, 15 Nov 2024 23:30:46 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 4AFNUkYE041877; Fri, 15 Nov 2024 23:30:46 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 4AFNUkE7041874; Fri, 15 Nov 2024 23:30:46 GMT (envelope-from git) Date: Fri, 15 Nov 2024 23:30:46 GMT Message-Id: <202411152330.4AFNUkE7041874@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Colin Percival Subject: git: 8a688fcc242e - releng/14.2 - release: add optional OCI images List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: cperciva X-Git-Repository: src X-Git-Refname: refs/heads/releng/14.2 X-Git-Reftype: branch X-Git-Commit: 8a688fcc242e1b4eea32fe97aa0f16a18ed21246 Auto-Submitted: auto-generated The branch releng/14.2 has been updated by cperciva: URL: https://cgit.FreeBSD.org/src/commit/?id=8a688fcc242e1b4eea32fe97aa0f16a18ed21246 commit 8a688fcc242e1b4eea32fe97aa0f16a18ed21246 Author: Doug Rabson AuthorDate: 2024-08-14 15:39:24 +0000 Commit: Colin Percival CommitDate: 2024-11-15 23:22:37 +0000 release: add optional OCI images This adds three OCI archive format files to the release containing FreeBSD base images suitable for static linked, dynamic linked and shell workloads. The shell image also contains pkg-bootstrap and can be easily extended by installing packages (including pkgbase packages). Approved by: re (cperciva) Reviewed by: dch, cpersiva, jlduran, zlei Differential Revision: https://reviews.freebsd.org/D46759 MFC after: 2 days (cherry picked from commit d03c82c28da86e0812b98b051d24ae5980804ad7) (cherry picked from commit 6686056ca3547c8280bd2423c59c661c2f501409) --- release/Makefile | 11 ++++--- release/Makefile.oci | 34 +++++++++++++++++++ release/release.conf.sample | 4 +++ release/release.sh | 47 ++++++++++++++++++++++++++- release/scripts/make-oci-image.sh | 63 ++++++++++++++++++++++++++++++++++++ release/tools/oci-image-dynamic.conf | 11 +++++++ release/tools/oci-image-minimal.conf | 19 +++++++++++ release/tools/oci-image-static.conf | 42 ++++++++++++++++++++++++ share/examples/Makefile | 6 ++++ share/examples/oci/Containerfile.pkg | 27 ++++++++++++++++ share/examples/oci/README | 7 ++++ share/man/man7/release.7 | 14 +++++++- 12 files changed, 279 insertions(+), 6 deletions(-) diff --git a/release/Makefile b/release/Makefile index be0abf84de23..4f95d3ba32ec 100644 --- a/release/Makefile +++ b/release/Makefile @@ -7,14 +7,16 @@ # memstick: Builds memory stick image (memstick.img) # mini-memstick: Builds minimal memory stick image (mini-memstick.img) # ftp: Sets up FTP distribution area (ftp) -# release: Invokes real-release, vm-release, and cloudware-release targets +# release: Invokes real-release, vm-release, cloudware-release and oci-release targets # real-release: Build all media and FTP distribution area # vm-release: Build all virtual machine image targets # cloudware-release: Build all cloud hosting provider targets -# install: Invokes the release-install and vm-install targets +# oci-release: Build all OCI container images +# install: Invokes the release-install, vm-install and oci-install targets # release-install: Copies all release installation media into ${DESTDIR} # vm-install: Copies all virtual machine images into ${DESTDIR} # cloud-install: Copies non-uploaded cloud images into ${DESTDIR} +# oci-install: Copies all OCI container images into ${DESTDIR} # # Variables affecting the build process: # WORLDDIR: location of src tree -- must have built world and default kernel @@ -319,7 +321,7 @@ ftp: packagesystem mkdir -p ftp cp *.txz MANIFEST ftp -release: real-release vm-release cloudware-release +release: real-release vm-release cloudware-release oci-release ${MAKE} -C ${.CURDIR} ${.MAKEFLAGS} release-done true @@ -330,7 +332,7 @@ real-release: ${MAKE} -C ${.CURDIR} ${.MAKEFLAGS} obj ${MAKE} -C ${.CURDIR} ${.MAKEFLAGS} ${RELEASE_TARGETS} -install: release-install vm-install .WAIT cloud-install +install: release-install vm-install oci-install .WAIT cloud-install release-install: .if defined(DESTDIR) && !empty(DESTDIR) @@ -350,3 +352,4 @@ release-install: .include "${.CURDIR}/Makefile.inc1" .include "${.CURDIR}/Makefile.vm" +.include "${.CURDIR}/Makefile.oci" diff --git a/release/Makefile.oci b/release/Makefile.oci new file mode 100644 index 000000000000..461c95f49636 --- /dev/null +++ b/release/Makefile.oci @@ -0,0 +1,34 @@ +# +# +# +# Makefile for building OCI container images. +# + +.if defined(WITH_OCIIMAGES) && !empty(WITH_OCIIMAGES) +OCI_IMAGES= static dynamic minimal +.endif + +oci-install: +.if defined(WITH_OCIIMAGES) && !empty(WITH_OCIIMAGES) + mkdir -p ${DESTDIR}/ociimages +. for _IMG in ${OCI_IMAGES} + cp -p ${.OBJDIR}/container-image-${_IMG}.txz ${DESTDIR}/ociimages +. endfor +.endif + +OCI_TARGETS= +OCI_DEPS_static= +OCI_DEPS_dynamic= container-image-static.txz +OCI_DEPS_minimal= container-image-dynamic.txz + +.for _IMG in ${OCI_IMAGES} +OCI_TARGETS+= container-image-${_IMG}.txz +container-image-${_IMG}.txz: ${OCI_DEPS_${_IMG}} + sh ${.CURDIR}/scripts/make-oci-image.sh ${.CURDIR} ${REVISION} ${BRANCH} ${TARGET_ARCH} ${_IMG} + skopeo copy \ + containers-storage:localhost/freebsd${REVISION:R}-${_IMG}:latest \ + oci-archive:${.OBJDIR}/container-image-${_IMG}.tar:freebsd${REVISION:R}-${_IMG}:${REVISION}-${BRANCH}-${TARGET_ARCH} + ${XZ_CMD} < ${.OBJDIR}/container-image-${_IMG}.tar > ${.OBJDIR}/container-image-${_IMG}.txz +.endfor + +oci-release: ${OCI_TARGETS} diff --git a/release/release.conf.sample b/release/release.conf.sample index 351496dcf6a4..1dd2f2504e30 100644 --- a/release/release.conf.sample +++ b/release/release.conf.sample @@ -114,3 +114,7 @@ PORTBRANCH="main" ## If WITH_CLOUDWARE is set to a non-empty value, this is a list of providers ## to create disk images. #CLOUDWARE="EC2 GCE ORACLE VAGRANT-VIRTUALBOX VAGRANT-VMWARE" + +## If WITH_OCIIMAGES is set to a non-empty value, build Open Container +## Initiative (OCI) base images as part of the release. +#WITH_OCIIMAGES= diff --git a/release/release.sh b/release/release.sh index cee0fbd5643d..d6752e016994 100755 --- a/release/release.sh +++ b/release/release.sh @@ -120,6 +120,9 @@ env_setup() { # cloud providers as part of the release. WITH_CLOUDWARE= + # Set to non-empty to build OCI images as part of the release + WITH_OCIIMAGES= + return 0 } # env_setup() @@ -195,7 +198,8 @@ env_check() { RELEASE_RMAKEFLAGS="${ARCH_FLAGS} ${RELEASE_FLAGS} \ KERNCONF=\"${KERNEL}\" ${CONF_FILES} ${SRCPORTS} \ WITH_DVD=${WITH_DVD} WITH_VMIMAGES=${WITH_VMIMAGES} \ - WITH_CLOUDWARE=${WITH_CLOUDWARE} XZ_THREADS=${XZ_THREADS}" + WITH_CLOUDWARE=${WITH_CLOUDWARE} WITH_OCIIMAGES=${WITH_OCIIMAGES} \ + XZ_THREADS=${XZ_THREADS}" return 0 } # env_check() @@ -288,6 +292,44 @@ extra_chroot_setup() { fi fi + if [ ! -z "${WITH_OCIIMAGES}" ]; then + # Install buildah and skopeo from ports if the ports tree is available; + # otherwise install the pkg. + if [ -d ${CHROOTDIR}/usr/ports ]; then + # Trick the ports 'run-autotools-fixup' target to do the right + # thing. + _OSVERSION=$(chroot ${CHROOTDIR} /usr/bin/uname -U) + REVISION=$(chroot ${CHROOTDIR} make -C /usr/src/release -V REVISION) + BRANCH=$(chroot ${CHROOTDIR} make -C /usr/src/release -V BRANCH) + UNAME_r=${REVISION}-${BRANCH} + GITUNSETOPTS="CONTRIB CURL CVS GITWEB GUI HTMLDOCS" + GITUNSETOPTS="${GITUNSETOPTS} ICONV NLS P4 PERL" + GITUNSETOPTS="${GITUNSETOPTS} SEND_EMAIL SUBTREE SVN" + GITUNSETOPTS="${GITUNSETOPTS} PCRE PCRE2" + PBUILD_FLAGS="OSVERSION=${_OSVERSION} BATCH=yes" + PBUILD_FLAGS="${PBUILD_FLAGS} UNAME_r=${UNAME_r}" + PBUILD_FLAGS="${PBUILD_FLAGS} OSREL=${REVISION}" + PBUILD_FLAGS="${PBUILD_FLAGS} WRKDIRPREFIX=/tmp/ports" + PBUILD_FLAGS="${PBUILD_FLAGS} DISTDIR=/tmp/distfiles" + for _PORT in sysutils/buildah sysutils/skopeo; do + eval chroot ${CHROOTDIR} env ${PBUILD_FLAGS} make -C \ + /usr/ports/${_PORT} \ + FORCE_PKG_REGISTER=1 deinstall install clean distclean + done + else + eval chroot ${CHROOTDIR} env ASSUME_ALWAYS_YES=yes \ + pkg install -y sysutils/buildah sysutils/skopeo + eval chroot ${CHROOTDIR} env ASSUME_ALWAYS_YES=yes \ + pkg clean -y + fi + # Use the vfs storage driver so that this works whether or not + # the build directory is on ZFS. The images are small so the + # performance difference is negligible. + eval chroot ${CHROOTDIR} sed -I .bak -e '/^driver/s/zfs/vfs/' /usr/local/etc/containers/storage.conf + # Remove any stray images from previous builds + eval chroot ${CHROOTDIR} buildah rmi -af + fi + if [ ! -z "${EMBEDDEDPORTS}" ]; then _OSVERSION=$(chroot ${CHROOTDIR} /usr/bin/uname -U) REVISION=$(chroot ${CHROOTDIR} make -C /usr/src/release -V REVISION) @@ -323,6 +365,9 @@ chroot_build_target() { fi eval chroot ${CHROOTDIR} make -C /usr/src ${RELEASE_WMAKEFLAGS} buildworld eval chroot ${CHROOTDIR} make -C /usr/src ${RELEASE_KMAKEFLAGS} buildkernel + if [ ! -z "${WITH_OCIIMAGES}" ]; then + eval chroot ${CHROOTDIR} make -C /usr/src ${RELEASE_WMAKEFLAGS} packages + fi return 0 } # chroot_build_target diff --git a/release/scripts/make-oci-image.sh b/release/scripts/make-oci-image.sh new file mode 100644 index 000000000000..a139a38d1930 --- /dev/null +++ b/release/scripts/make-oci-image.sh @@ -0,0 +1,63 @@ +#! /bin/sh + +# Build an Open Container Initiative (OCI) container image + +curdir=$1; shift +rev=$1; shift +branch=$1; shift +arch=$1; shift +image=$1; shift + +major=${rev%.*} +minor=${rev#*.} + +abi=FreeBSD:${major}:${arch} + +echo "Building OCI freebsd${major}-${image} image for ${abi}" + +. ${curdir}/tools/oci-image-${image}.conf + +init_workdir() { + local abi=$1; shift + local workdir=$(mktemp -d -t oci-images) + + mkdir ${workdir}/repos + cat > ${workdir}/repos/base.conf < /dev/null + mtree -deU -p $m/var -f /etc/mtree/BSD.var.dist > /dev/null + mtree -deU -p $m/usr -f /etc/mtree/BSD.usr.dist > /dev/null + mtree -deU -p $m/usr/include -f /etc/mtree/BSD.include.dist > /dev/null + mtree -deU -p $m/usr/lib -f /etc/mtree/BSD.debug.dist > /dev/null + install_packages ${abi} ${workdir} $m FreeBSD-caroot FreeBSD-zoneinfo + cp /etc/master.passwd $m/etc + pwd_mkdb -p -d $m/etc $m/etc/master.passwd || return $? + cp /etc/group $m/etc || return $? + cp /etc/termcap.small $m/etc/termcap.small || return $? + cp /etc/termcap.small $m/usr/share/misc/termcap || return $? + env DESTDIR=$m /usr/sbin/certctl rehash + # Generate a suitable repo config for pkgbase + case ${branch} in + CURRENT|STABLE|BETA*) + repo=base_latest + ;; + *) + repo=base_release_${minor} + ;; + esac + mkdir -p $m/usr/local/etc/pkg/repos + cat > $m/usr/local/etc/pkg/repos/base.conf <