poudriere-devel getting varying "Killed" results, seemingly only when a builder is reused, amd64 and aarch64

I recently updated ports on multiple systems (amd64m aarch64)
to:

# ~/fbsd-based-on-what-commit.sh 
branch: main
merge-base: 1af1b4caa25a2d4dcee07939369e988c996bffe5
merge-base: CommitDate: 2021-09-06 00:39:11 +0000
1af1b4caa25a (HEAD -> main, freebsd/main, freebsd/HEAD) www/webkit2-gtk3: convert 3f23fec4c93a to upstream fix
n557108 (--first-parent --count for merge-base)

it seemd to go okay but include a poudriere-devel update that would
be put to use the next time I built (each machine).


NOTEs:
For the following, it turns out that the first poudrier bulk
commands accidentally had -r indicated. That is abnormal for
my builds. But retries without the -r still got problems.


I'm now trying an update to (examples from before retry):

# ~/fbsd-based-on-what-commit.sh 
branch: main
merge-base: b5815e7648a8e5307a20a234befa00e34306319d
merge-base: CommitDate: 2021-09-06 22:25:00 +0000
b5815e7648a8 (HEAD -> main, freebsd/main, freebsd/HEAD) net/mpich: unbreak optimized runtime after 88e134883dd2
n557201 (--first-parent --count for merge-base)

and the systems doing the builds are getting the likes of:


NOTE: Some do not seem to be Killed during extraction,
      others are.
NOTE: There is variation in the exact step whre the Killed
      happens for the various build attempts of the same port.
      (This note spans retries.)
NOTE: There is variation in which ports have some step killed
      from retry to retry. (This note spans retries.)


amd64 (Thread Ripper 1950X, 128 GiByte RAM) exmaples:

[amd64_ZFS] |   `-- Extracting llvm12-12.0.1_4: .....Killed
ln: /usr/local/poudriere/data/logs/bulk/13_0R-amd64-default/2021-09-06_16h12m10s/logs/errors/qt5-widgets-5.15.2_3.log: File exists
build of x11-toolkits/qt5-widgets | qt5-widgets-5.15.2_3 ended at Mon Sep  6 16:20:09 PDT 2021
build time: 00:00:26
!!! build failure encountered !!!

[amd64_ZFS] `-- Extracting python38-3.8.11: .......... done
Killed
ln: /usr/local/poudriere/data/logs/bulk/13_0R-amd64-default/2021-09-06_16h12m10s/logs/errors/poppler-21.09.0.log: File exists
build of graphics/poppler | poppler-21.09.0 ended at Mon Sep  6 16:22:00 PDT 2021
build time: 00:00:28
!!! build failure encountered !!!


aarch64 (HoneyComb, 64 GiByte RAM) examples:

===>   libwacom-1.5 depends on shared library: libintl.so - found (/usr/local/lib/libintl.so)
===========================================================================
=======================<phase: configure      >============================
===== env: NO_DEPENDS=yes USER=root UID=0 GID=0
===>  Configuring for libwacom-1.5
Killed
ln: /usr/local/poudriere/data/logs/bulk/13_0R-CA72-default/2021-09-06_16h11m41s/logs/errors/libwacom-1.5.log: File exists
build of x11/libwacom | libwacom-1.5 ended at Mon Sep  6 16:13:36 PDT 2021
build time: 00:00:24
!!! build failure encountered !!!

[CA72_ZFS] |   `-- Extracting libxcb-1.14_1: ....Killed
ln: /usr/local/poudriere/data/logs/bulk/13_0R-CA72-default/2021-09-06_16h11m41s/logs/errors/gstreamer1-libav-1.16.2.log: File exists
build of multimedia/gstreamer1-libav | gstreamer1-libav-1.16.2 ended at Mon Sep  6 16:22:49 PDT 2021
build time: 00:00:25
!!! build failure encountered !!!

[CA72_ZFS] Extracting cmake-3.21.0: Killed
ln: /usr/local/poudriere/data/logs/bulk/13_0R-CA72-default/2021-09-06_16h11m41s/logs/errors/poppler-21.09.0.log: File exists
build of graphics/poppler | poppler-21.09.0 ended at Mon Sep  6 16:33:03 PDT 2021
build time: 00:00:24
!!! build failure encountered !!!

The new builds are of a smaller number of ports than built last
time, with lots of overlap in what was to be built.

All of the these failures get the "ln: . . . : File exists" notes.

The systems are unchanged, still at:

# uname -apKU
FreeBSD amd64_ZFS 13.0-RELEASE-p4 FreeBSD 13.0-RELEASE-p4 #1 releng/13.0-n244760-940681634ee1-dirty: Mon Aug 30 11:28:31 PDT 2021     root@amd64_ZFS:/usr/obj/BUILDs/13_0R-amd64-nodbg-clang/usr/13_0R-src/amd64.amd64/sys/GENERIC-NODBG  amd64 amd64 1300139 1300139

# uname -apKU
FreeBSD CA72_16Gp_ZFS 13.0-RELEASE-p4 FreeBSD 13.0-RELEASE-p4 #4 releng/13.0-n244760-940681634ee1-dirty: Mon Aug 30 11:35:45 PDT 2021     root@CA72_16Gp_ZFS:/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R-src/arm64.aarch64/sys/GENERIC-NODBG-CA72  arm64 aarch64 1300139 1300139


From after retry without -r:

amd64:

[amd64_ZFS] `-- Extracting libXi-1.7.10,1: Killed
ln: /usr/local/poudriere/data/logs/bulk/13_0R-amd64-default/2021-09-06_16h54m44s/logs/errors/qt5-printsupport-5.15.2_1.log: File exists
 encountered !!!

NOTE: The missing text above is missing in the log file, it is
not a copy/paste error.

[amd64_ZFS] |   `-- Extracting llvm12-12.0.1_4: .....Killed
ln: /usr/local/poudriere/data/logs/bulk/13_0R-amd64-default/2021-09-06_17h49m31s/logs/errors/qt5-assistant-5.15.2.log: File exists
!!! build failure encountered !!!


Note: a build on aarch64 that targeted armv7 got:

[CA72_ZFS] |   `-- Extracting python38-3.8.11: ...Killed
ln: /usr/local/poudriere/data/logs/bulk/13_0R-CA7-default/2021-09-06_16h55m02s/logs/errors/libinput-1.16.4.log: File exists
build of x11/libinput | libinput-1.16.4 ended at Mon Sep  6 16:57:25 PDT 2021
build time: 00:00:28

[CA72_ZFS] |   `-- Extracting p11-kit-0.24.0: ..Killed
ln: /usr/local/poudriere/data/logs/bulk/13_0R-CA7-default/2021-09-06_16h55m02s/logs/errors/gstreamer1-libav-1.16.2.log: File exists
build of multimedia/gstreamer1-libav | gstreamer1-libav-1.16.2 ended at Mon Sep  6 17:05:46 PDT 2021
build time: 00:00:25
!!! build failure encountered !!!

[CA72_ZFS] Extracting cmake-3.21.0: Killed
ln: /usr/local/poudriere/data/logs/bulk/13_0R-CA7-default/2021-09-06_16h55m02s/logs/errors/poppler-21.09.0.log: File exists
build of graphics/poppler | poppler-21.09.0 ended at Mon Sep  6 17:15:09 PDT 2021
build time: 00:00:26
!!! build failure encountered !!!


So far whatever is of the form:

[??] [00:00:00] Building . . .

has completed fine, even if it failed in prior attempts (but
later in the overall builder reusage sequence).

So, for example, each retry manages to build more ports, including
some that previously failed.

Some ports did build in reuse of the same builder (such as [01]).
The number of reuses before getting a failure varied.


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)