Podman within JAIL(nested containers)

Go to: [ bottom of page ] [ top of archives ] [ this month ]
From: peter garshtja <peter.garshtja_at_ambient-md.com>
Date: Tue, 13 Feb 2024 12:23:25 UTC
Greetings,

I have been trying to use podman in a long living jail container on FreeBSD
13.2 release.
The long living container is configured with linux emulation

> *zroot/bastille/jails/podman/root on / (zfs, local, noatime, nfsv4acls)
> zroot/bastille/jails/podman/root/containers on /var/db/containers (zfs,
> local, noatime, nfsv4acls) devfs on /compat/linux/dev (devfs) tmpfs on
> /compat/linux/dev/shm (tmpfs, local) fdescfs on /compat/linux/dev/fd
> (fdescfs) linprocfs on /compat/linux/proc (linprocfs, local) linsysfs on
> /compat/linux/sys (linsysfs, local) /tmp on /compat/linux/tmp (nullfs,
> local, noatime, nosuid, nfsv4acls) /usr/home on /compat/linux/home (nullfs,
> local, noatime, nfsv4acls) /usr/local/bastille/releases/13.2-RELEASE on
> /.bastille (nullfs, local, noatime, read-only, nfsv4acls) devfs on /dev
> (devfs) fdescfs on /dev/fd (fdescfs)*


ATM, I am trying to figure what configuration is missing on the system that
causes:

> *Error pulling candidate docker.io/library/alpine:latest
> <http://docker.io/library/alpine:latest>: copying system image from
> manifest list: writing blob: adding layer with blob
> "sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8":
> ApplyLayer stdout: stderr: operation not permitted exit status 1*



The full log

> *root@podman:~ # podman --log-level debug run --rm --os=linux
> docker://docker.io/alpine <http://docker.io/alpine> cat /etc/os-release
> INFO[0000] podman filtering at log level debug DEBU[0000] Called
> run.PersistentPreRunE(podman --log-level debug run --rm --os=linux
> docker://docker.io/alpine <http://docker.io/alpine> cat /etc/os-release)
> DEBU[0000] Using conmon: "/usr/local/bin/conmon" DEBU[0000] Initializing
> boltdb state at /var/db/containers/storage/libpod/bolt_state.db DEBU[0000]
> Using graph driver zfs DEBU[0000] Using graph root
> /var/db/containers/storage DEBU[0000] Using run root
> /var/run/containers/storage DEBU[0000] Using static dir
> /var/db/containers/storage/libpod DEBU[0000] Using tmp dir /var/run/libpod
> DEBU[0000] Using volume path /var/db/containers/storage/volumes DEBU[0000]
> Using transient store: false DEBU[0000] [graphdriver] trying provided
> driver "zfs" DEBU[0000] ID:36a2c4c9-eeba-406a-b1e4-0da02dcc28be START
> /sbin/zfs list -rHp -t filesystem -o
> name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset
> zroot/bastille/jails/podman/root/containers storage-driver=zfs DEBU[0000]
> ID:36a2c4c9-eeba-406a-b1e4-0da02dcc28be FINISH storage-driver=zfs
> DEBU[0000] Initializing event backend file DEBU[0000] Configured OCI
> runtime youki initialization failed: no valid executable found for OCI
> runtime youki: invalid argument DEBU[0000] Configured OCI runtime krun
> initialization failed: no valid executable found for OCI runtime krun:
> invalid argument DEBU[0000] Configured OCI runtime crun-wasm initialization
> failed: no valid executable found for OCI runtime crun-wasm: invalid
> argument DEBU[0000] Configured OCI runtime runc initialization failed: no
> valid executable found for OCI runtime runc: invalid argument DEBU[0000]
> Configured OCI runtime kata initialization failed: no valid executable
> found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI
> runtime runsc initialization failed: no valid executable found for OCI
> runtime runsc: invalid argument DEBU[0000] Configured OCI runtime crun
> initialization failed: no valid executable found for OCI runtime crun:
> invalid argument DEBU[0000] Configured OCI runtime runj initialization
> failed: no valid executable found for OCI runtime runj: invalid argument
> DEBU[0000] Using OCI runtime "/usr/local/bin/ocijail" INFO[0000] Setting
> parallel job count to 13 DEBU[0000] Successfully loaded 1 networks
> DEBU[0000] Pulling image docker://docker.io/alpine
> <http://docker.io/alpine> (policy: missing) DEBU[0000] Looking up image
> "docker.io/library/alpine:latest <http://docker.io/library/alpine:latest>"
> in local containers storage DEBU[0000] Normalized platform linux/amd64 to
> {amd64 linux [] } DEBU[0000] Trying "docker.io/library/alpine:latest
> <http://docker.io/library/alpine:latest>" ... DEBU[0000] reference
> "[zfs@/var/db/containers/storage+/var/run/containers/storage]docker.io/library/alpine:latest
> <http://docker.io/library/alpine:latest>" does not resolve to an image ID
> DEBU[0000] Trying "docker.io/library/alpine:latest
> <http://docker.io/library/alpine:latest>" ... DEBU[0000] reference
> "[zfs@/var/db/containers/storage+/var/run/containers/storage]docker.io/library/alpine:latest
> <http://docker.io/library/alpine:latest>" does not resolve to an image ID
> DEBU[0000] Trying "docker.io/library/alpine:latest
> <http://docker.io/library/alpine:latest>" ... DEBU[0000] Enforcing pull
> policy to "newer" to pull custom platform (arch: "", os: "linux", variant:
> "") - local image may mistakenly specify wrong platform DEBU[0000] Loading
> registries configuration "/usr/local/etc/containers/registries.conf"
> DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0000]
> Attempting to pull candidate docker.io/library/alpine:latest
> <http://docker.io/library/alpine:latest> for
> docker.io/library/alpine:latest DEBU[0000
> <http://docker.io/library/alpine:latest%0DDEBU%5B0000>] parsed reference
> into
> "[zfs@/var/db/containers/storage+/var/run/containers/storage]docker.io/library/alpine:latest
> <http://docker.io/library/alpine:latest>" Trying to pull
> docker.io/library/alpine:latest.
> <http://docker.io/library/alpine:latest.>.. DEBU[0000] Copying source image
> //alpine:latest to destination image
> [zfs@/var/db/containers/storage+/var/run/containers/storage]docker.io/library/alpine:latest
> DEBU[0000 <http://docker.io/library/alpine:latest%0DDEBU%5B0000>] Using
> registries.d directory /usr/local/etc/containers/registries.d DEBU[0000]
> Trying to access "docker.io/library/alpine:latest
> <http://docker.io/library/alpine:latest>" DEBU[0000] No credentials
> matching docker.io/library/alpine <http://docker.io/library/alpine> found
> in /root/.config/containers/auth.json DEBU[0000] No credentials matching
> docker.io/library/alpine <http://docker.io/library/alpine> found in
> /root/.config/containers/auth.json DEBU[0000] No credentials matching
> docker.io/library/alpine <http://docker.io/library/alpine> found in
> /root/.docker/config.json DEBU[0000] No credentials matching
> docker.io/library/alpine <http://docker.io/library/alpine> found in
> /root/.dockercfg DEBU[0000] No credentials for docker.io/library/alpine
> <http://docker.io/library/alpine> found DEBU[0000] No signature storage
> configuration found for docker.io/library/alpine:latest
> <http://docker.io/library/alpine:latest>, using built-in default
> file:///var/lib/containers/sigstore DEBU[0000] Looking for TLS certificates
> and private keys in /usr/local/etc/docker/certs.d/docker.io
> <http://docker.io> DEBU[0000] GET https://registry-1.docker.io/v2/
> DEBU[0000 <https://registry-1.docker.io/v2/%0DDEBU%5B0000>] Ping
> https://registry-1.docker.io/v2/ <https://registry-1.docker.io/v2/> status
> 401 DEBU[0000] GET
> https://auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io
> DEBU[0000
> <https://auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io%0DDEBU[0000>]
> GET https://registry-1.docker.io/v2/library/alpine/manifests/latest
> DEBU[0000
> <https://registry-1.docker.io/v2/library/alpine/manifests/latest%0DDEBU%5B0000>]
> Content-Type from manifest GET is
> "application/vnd.docker.distribution.manifest.list.v2+json" DEBU[0000]
> Using SQLite blob info cache at
> /var/lib/containers/cache/blob-info-cache-v1.sqlite DEBU[0000] Source is a
> manifest list; copying (only) instance
> sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0 for
> current system DEBU[0000] GET
> https://registry-1.docker.io/v2/library/alpine/manifests/sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0
> DEBU[0000
> <https://registry-1.docker.io/v2/library/alpine/manifests/sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0%0DDEBU%5B0000>]
> Content-Type from manifest GET is
> "application/vnd.docker.distribution.manifest.v2+json" DEBU[0000]
> IsRunningImageAllowed for image docker:docker.io/library/alpine:latest
> DEBU[0000 <http://docker.io/library/alpine:latest%0DDEBU%5B0000>] Using
> default policy section DEBU[0000] Requirement 0: allowed DEBU[0000]
> Overall: allowed DEBU[0000] Downloading
> /v2/library/alpine/blobs/sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd
> DEBU[0000] GET
> https://registry-1.docker.io/v2/library/alpine/blobs/sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd
> <https://registry-1.docker.io/v2/library/alpine/blobs/sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd>
> Getting image source signatures DEBU[0000] Reading
> /var/lib/containers/sigstore/library/alpine@sha256=6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0/signature-1
> DEBU[0000] Not looking for sigstore attachments: disabled by configuration
> DEBU[0000] Manifest has MIME type
> application/vnd.docker.distribution.manifest.v2+json, ordered candidate
> list [application/vnd.docker.distribution.manifest.v2+json,
> application/vnd.docker.distribution.manifest.v1+prettyjws,
> application/vnd.oci.image.manifest.v1+json,
> application/vnd.docker.distribution.manifest.v1+json] DEBU[0000] ... will
> first try using the original manifest unmodified DEBU[0000] Checking if we
> can reuse blob
> sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8:
> general substitution = true, compression for MIME type
> "application/vnd.docker.image.rootfs.diff.tar.gzip" = true DEBU[0000]
> Failed to retrieve partial blob: format not supported on this system
> DEBU[0000] Downloading
> /v2/library/alpine/blobs/sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8
> DEBU[0000] GET
> https://registry-1.docker.io/v2/library/alpine/blobs/sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8
> <https://registry-1.docker.io/v2/library/alpine/blobs/sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8>
> Copying blob 4abcf2066143 [--------------------------------------] 0.0b /
> 3.3MiB (skipped: 0.0b = 0.00%) Copying blob 4abcf2066143
> [--------------------------------------] 0.0b / 3.3MiB | 0.0 b/s Copying
> blob 4abcf2066143 done | Copying blob 4abcf2066143 done | DEBU[0001]
> ID:62d93b96-1b16-4703-8999-a2ba584f1bc5 FINISH storage-driver=zfs
> DEBU[0001] ID:1871d56d-a96a-4a0d-8355-6688f206d776 START /sbin/zfs list -Hp
> -o
> name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset
> zroot/bastille/jails/podman/root/containers/d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820
> storage-driver=zfs Copying blob 4abcf2066143 done | DEBU[0001]
> mount("zroot/bastille/jails/podman/root/containers/d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820",
> "/var/db/containers/storage/zfs/graph/d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820",
> "") storage-driver=zfs DEBU[0001] Start untar layer ERRO[0001] While
> applying layer: ApplyLayer stdout: stderr: operation not permitted exit
> status 1 DEBU[0001]
> unmount("/var/db/containers/storage/zfs/graph/d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820")
> storage-driver=zfs DEBU[0001] ID:acefec41-353b-4871-a2e7-a60a7b239d94 START
> /sbin/zfs destroy -r
> zroot/bastille/jails/podman/root/containers/d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820
> storage-driver=zfs DEBU[0001] ID:acefec41-353b-4871-a2e7-a60a7b239d94
> FINISH storage-driver=zfs DEBU[0001] Error pulling candidate
> docker.io/library/alpine:latest <http://docker.io/library/alpine:latest>:
> copying system image from manifest list: writing blob: adding layer with
> blob
> "sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8":
> ApplyLayer stdout: stderr: operation not permitted exit status 1 Error:
> copying system image from manifest list: writing blob: adding layer with
> blob
> "sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8":
> ApplyLayer stdout: stderr: operation not permitted exit status 1 DEBU[0001]
> Shutting down engines *


Please advise.
Thanks,
Petru