FYI: poudriere-devel builder got stuck in kqread during from scratch bulk -a

Go to: [ bottom of page ] [ top of archives ] [ this month ]
From: Mark Millard <marklmi_at_yahoo.com>
Date: Thu, 16 Nov 2023 04:01:56 UTC
Tail of the poudriere-devel Building and Finished notices just before only
the stuck builder was left "running":

. . .
[85:21:50] [27] [02:06:01] Finished databases/mongodb60 | mongodb60-6.0.11: Success
[85:34:00] [28] [03:23:06] Finished biology/ncbi-cxx-toolkit | ncbi-cxx-toolkit-27.0.0_1: Success
[85:46:31] [30] [08:19:30] Finished cad/kicad-library-packages3d | kicad-library-packages3d-7.0.2_2: Success
[87:07:02] [03] [13:00:45] Finished emulators/libretro-mame | libretro-mame-20220124_1: Success

# poudriere status -b
[main-amd64-bulk_a-default] [2023-11-11_17h59m25s] [parallel_build:] Queued: 34683 Built: 33807 Failed: 173   Skipped: 382   Ignored: 320   Fetched: 0     Tobuild: 1      Time: 88:17:59
 ID  TOTAL        ORIGIN   PKGNAME                PHASE PHASE    TMPFS    CPU% MEM%
[05] 17:27:25 ftp/curlie | curlie-1.6.7_15 check-sanity 17:27:15 1.28 GiB          
=>> Logs: /usr/local/poudriere/data/logs/bulk/main-amd64-bulk_a-default/2023-11-11_17h59m25s

# ps -oetime -alxdww # but left vs. right part split out, left side first (with 2 notes) . . .
. . .
  3-16:18:11   0 83812  1765  8  20  0  22128 6856 select   I+    0      4:33.84 |
    17:27:35   0 48809 83812 26  68  0  22128 3044 kqread   I     0      0:00.05 | (<<==build_pkg)
    17:27:25   0 50788 48809 25  88  0      0    0 -        Z     0      0:00.03 | (<<==defunct)
  3-16:16:40   0 76609 83812 19  34  0  18544 3608 piperd   I+    0     16:45.55 |
  3-16:18:09   0 84173 83812 13  68  0  15472 2864 nanslp   S+    0    184:47.04 |
. . .

Then right side . . .

. . .
| `-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -jmain-amd64-bulk_a -a
|   |-- sh: poudriere[main-amd64-bulk_a-default][05]: build_pkg (curlie-1.6.7_15) (sh)
|   | `-- <defunct>
|   |-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -jmain-amd64-bulk_a -a
|   `-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -jmain-amd64-bulk_a -a
. . .

Later forcing a failure of the builder that was stuck for curlie:

. . .
[main-amd64-bulk_a-default] [2023-11-11_17h59m25s] [committing:] Queued: 34683 Built: 33807 Failed: 174   Skipped: 382   Ignored: 320   Fetched: 0     Tobuild: 0      Time: 88:46:04
[88:46:05] Logs: /usr/local/poudriere/data/logs/bulk/main-amd64-bulk_a-default/2023-11-11_17h59m25s
[88:46:07] Cleaning up
[88:46:07] Unmounting file systems


Context, for reference:

# uname -apKU
FreeBSD amd64-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT #126 main-n266130-d521abdff236-dirty: Tue Oct 24 18:17:40 PDT 2023     root@amd64-ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG amd64 amd64 1500002 1500002

# ~/fbsd-based-on-what-commit.sh -C /usr/ports
6ec8e3450b29 (HEAD -> main, freebsd/main, freebsd/HEAD) devel/sdts++: Mark DEPRECATED
Author:     Muhammad Moinur Rahman <bofh@FreeBSD.org>
Commit:     Muhammad Moinur Rahman <bofh@FreeBSD.org>
CommitDate: 2023-10-21 19:01:38 +0000
branch: main
merge-base: 6ec8e3450b29462a590d09fb0b07ed214d456bd5
merge-base: CommitDate: 2023-10-21 19:01:38 +0000
n637598 (--first-parent --count for merge-base)

ThreadRipper 1950X system, 32 hardware threads, 128 GiBytes
of RAM, ZFS. USE_TMPFS=all in use.

The style of bulk -a build has high load averages for much
of the bulk -a compared to the hardware thread count. Largest
swap space use observed was 123513 MiBytes, observed via a
hacked top that tracks some "Maximum Observed" figures for
some of top's monitoring data. For reference:

245564Mi MaxObs(Act+Wir+Lndry+SwapUsed)

Note: max(A+B) <= max(A) + max(B)

===
Mark Millard
marklmi at yahoo.com