Re: poudriere and the user ... is it mostly a lost idea?
- In reply to: Dennis Clarke : "Re: poudriere and the user ... is it mostly a lost idea?"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Fri, 17 Jan 2025 20:26:47 UTC
On Jan 17, 2025, at 10:30, Dennis Clarke <dclarke@blastwave.org> wrote:
> On 1/15/25 21:20, Mark Millard wrote:
>> Dennis Clarke <dclarke_at_blastwave.org> wrote on
>> Date: Wed, 15 Jan 2025 15:16:58 UTC :
>>> Over the past month or so I see endless fails in builds for the big
>>> three user facing window manager things. This means that a simple user
>>> type person can not get a desktop. Really? Yes really. For at least a
>>> month or more you can not build KDE5 nor LXDE nor XFCE desktop. . . .
>> Here you seem to have leaped from your context's bulk build
>> problems to most everyone else's bulk builds of similar
>> software having similar bulk build problems.
>
> I apologize for the rant. Clearly a rant. Pure frustration as I try
> to do some testing of the big RELEASE stuff from 13.4 up to 15-CURRENT.
> It was a hand waving rant wherein I have seens build failures for weeks
> and weeks and it really feels like a hit or miss throw a dart good luck
> and spin the wheel maybe you are a winner today situation.
>
>> (Since Jan-7 I'm I'm temporarily without access to the FreeBSD
>> systems that I normally do. Also, I do not normally build those
>> specific ports. So I do not have evidence about those from my
>> own activities.)
>>
>
> I have plenty of logs.
If you can provide access to examples, I might be able to take
a look during breaks from other activities and see if I notice
any patterns to the failures. (If anyone one else would look,
I've no idea.) No time frame promises.
Another question is if you have also been getting console log
notices, such as OOM kills and the like. Logs from inside
builders may be necessary context but not sufficient context
for some types of failures.
> Piles of them. Perhaps the problem is that I
> am building on a 15-CURRENT machine which has poudriere jails like so :
All the official FreeBSD build machines boot main [so: 15 as stands]
but use jails that are for specific releases --or are for main.
(Sometimes there are rare runs of a STABLE instead of a RELEASE.)
So, of itself, using main to build for other versions is the most
tested way of building.
But you might not be using the same vintage of main as the build
servers are using.
And, such statements do not get into other issues such as resources
on the build systems or the configuration of the bulk run. Those
are less likely to be match.
> titan# poudriere jails -l
> JAILNAME VERSION ARCH METHOD TIMESTAMP PATH
> 134amd64 13.4-RELEASE-p2 1304000 3f40d5821eca amd64 git+https 2025-01-10 10:42:08 /poudriere/jails/134amd64
> 142amd64 14.2-RELEASE 1402000 c8918d6c7412 amd64 git+https 2024-12-03 12:50:29 /poudriere/jails/142amd64
> 140amd64 14.2-STABLE 1402501 e6de39be80e2 amd64 git+https 2025-01-13 21:36:43 /poudriere/jails/140amd64
> 150amd64 15.0-CURRENT 1500030 amd64 src=/usr/src 2025-01-12 07:44:29 /poudriere/jails/150amd64
> titan#
https://lists.freebsd.org/archives/dev-commits-src-branches/2025-January/date.html
shows:
Mon, 13 Jan 2025
• git: e6de39be80e2 - stable/14 - MFC: libmagic: Unbreak for older FreeBSD releases. Xin LI
This is also why there is the "501" in the 1402501: The 5?? indicates, in
part, that it is a STABLE branch that is in the jail instead of a releng/*
branch. The 14 indicates FreeBSD 14, the 02 indicates that it is 14.2
specifically. So: some vintage of stable/14 after releng/14.2 was branched
off of stable/14 .
releng is not a branch of itself that is used. There are separate branches
(I'm not listing in time order):
. . .
releng/13.4
releng/14.0
releng/14.1
releng/14.2
. . .
NOTE: https://cgit.freebsd.org/ is having problems currently --so I'm not
quoting web page material above.
> The one called 140stable is a bit strange given that I built it with the
> branch called "releng" for 14 and what I get is 14.2-STABLE.
There is a branch stable/14 . After the likes of releng/14.2 being branched
off it it, stable/14 is modified to report 14.2-STABLE.
stable/14 changes content frequently. On days where it has are commits,
there usually multiple commits it it that day. Some days may not have
commits. So far this month: about 110 commits (on 2025-Jan-17). One of
them is a revert of an earlier commit.
(stable/13 has 33 commits, the older supported stable/* tends to have
less activity.)
STABLE does not here mean "changes infrequently". Only RELEASE builds
tend to have that.
stable/14 activity at this point is preparation for releng/14.3 , so
sort of a 14.2+ approximation.
> Whatever
> that is. I had the silly notion that something called "STABLE" is a good
> place to build packages.
stable/14 is a development branch that should be more stable than main:
far less experimentation and normally testing in main before being
MFC'd to stable/14 . stable/14 is supported. main is not.
Similar comments apply to stable/13 .
See: https://www.freebsd.org/security/#sup
> A stable is where one may keep horses. Maybe
> goats. Other than that I really do not know if building packages in that
> jail would be of any value compared to the 142amd64 jail. Who knows?
> I surely do not.
FreeBSD does not officially build and distribute packages for
stable/14 or stable/13 . So building packages based on stable/14
is supported but not as tested.
(There can be rare tests of building packages for the likes of
stable/14 but such are not distributed.)
> I tend to kick off something like this :
>
> titan# ls -lApbtr /poudriere/data/packages/
> total 60
> drwxr-xr-x 3 root wheel 15 Jan 15 21:40 134amd64-latest/
> drwxr-xr-x 3 root wheel 15 Jan 16 07:20 150amd64-2025Q1/
> drwxr-xr-x 3 root wheel 15 Jan 16 07:23 140amd64-2025Q1/
> drwxr-xr-x 3 root wheel 15 Jan 16 10:15 142amd64-latest/
> drwxr-xr-x 3 root wheel 15 Jan 16 10:36 142amd64-2025Q1/
> drwxr-xr-x 3 root wheel 15 Jan 16 14:00 150amd64-latest/
> drwxr-xr-x 3 root wheel 15 Jan 16 14:09 134amd64-2025Q1/
> titan#
> titan# /usr/bin/time -p idprio 0 poudriere bulk -r -j 140amd64 -p 2025Q1 -f /root/pkg.list
> [00:00:00] Creating the reference jail... done
> [00:00:00] Mounting system devices for 140amd64-2025Q1
> [00:00:01] Stashing existing package repository
> [00:00:01] Mounting ccache from: /var/cache/ccache
> [00:00:01] Mounting ports from: /poudriere/ports/2025Q1
> [00:00:01] Mounting packages from: /poudriere/data/packages/140amd64-2025Q1
> [00:00:01] Mounting distfiles from: /poudriere/distfiles
> /etc/resolv.conf -> /poudriere/data/.m/140amd64-2025Q1/ref/etc/resolv.conf
> [00:00:01] Starting jail 140amd64-2025Q1
> Updating /var/run/os-release done.
> [00:00:01] Will build as nobody:nobody (65534:65534)
> [00:00:01] Logs: /poudriere/data/logs/bulk/140amd64-2025Q1/2025-01-16_14h18m02s
> [00:00:01] Loading MOVED for /poudriere/data/.m/140amd64-2025Q1/ref/usr/ports
> [00:00:02] Ports supports: FLAVORS SUBPACKAGES SELECTED_OPTIONS
> [00:00:02] Inspecting ports tree for modifications to git checkout... no
> [00:00:03] Ports top-level git hash: 1bbe39c25
> [00:00:03] Gathering ports metadata
> [00:00:08] Calculating ports order and dependencies
> [00:00:10] Trimming IGNORED and blacklisted ports
> [00:00:10] Sanity checking the repository
> [00:00:10] Checking packages for incremental rebuild needs
> [00:00:13] Deleting rsync-3.4.0.pkg: new version: 3.4.1
> [00:00:14] Deleting mariadb1011-server-10.11.10_1.pkg: missing dependency: rsync-3.4.0
> [00:00:14] Deleting mariadb114-server-11.4.4.pkg: missing dependency: rsync-3.4.0
> [00:00:15] Deleting stale symlinks... done
> [00:00:15] Deleting empty directories... done
> [00:00:15] Unqueueing existing packages
> [00:00:16] Unqueueing orphaned build dependencies
> [00:00:16] Sanity checking build queue
> [00:00:16] Processing PRIORITY_BOOST
> [00:00:16] Balancing pool
> [140amd64-2025Q1] [2025-01-16_14h18m02s] [balancing_pool] Queued: 13 Built: 0 Failed: 0 Skipped: 0 Ignored: 0 Fetched: 0 Tobuild: 13 T ime: 00:00:15
> [00:00:16] Recording filesystem state for prepkg... done
> [00:00:16] Building 13 packages using up to 13 builders
> [00:00:16] Hit CTRL+t at any time to see build progress and stats
> [00:00:16] [01] [00:00:00] Builder starting
> [00:00:16] [02] [00:00:00] Builder starting
> [00:00:16] [01] [00:00:00] Builder started
> [00:00:16] [01] [00:00:00] Building x11-toolkits/libwnck3 | libwnck3-3.36.0_2
> [00:00:16] [02] [00:00:00] Builder started
> [00:00:16] [02] [00:00:00] Building net/rsync | rsync-3.4.1
> [00:00:26] [02] [00:00:10] Finished net/rsync | rsync-3.4.1: Success
> [00:00:26] [02] [00:00:00] Building databases/mariadb114-server | mariadb114-server-11.4.4
> [00:00:26] [03] [00:00:00] Builder starting
> [00:00:27] [03] [00:00:01] Builder started
> [00:00:27] [03] [00:00:00] Building databases/mariadb1011-server | mariadb1011-server-10.11.10_1
> [00:00:36] [01] [00:00:20] Finished x11-toolkits/libwnck3 | libwnck3-3.36.0_2: Failed: build
So the log for libwnck3-3.36.0_2 would hopefully give more specific
information about what failed within its build attempt.
I'll note that this stopped well before building any of your "big
three": a prerequisite failed before getting that far. There can be
a large number of prerequisites, some of which may be used by more
than one of your "big three": one failure could block multiple of
the "big three".
> [00:00:36] [01] [00:00:20] Skipping x11/libxfce4windowing | libxfce4windowing-4.20.0: Dependent port x11-toolkits/libwnck3 | libwnck3-3.36.0 _2 failed
> [00:00:36] [01] [00:00:20] Skipping x11-fm/thunar | thunar-4.20.1: Dependent port x11-toolkits/libwnck3 | libwnck3-3.36.0_2 failed
> [00:00:36] [01] [00:00:20] Skipping x11-wm/xfce4 | xfce-4.20: Dependent port x11-toolkits/libwnck3 | libwnck3-3.36.0_2 failed
> [00:00:36] [01] [00:00:20] Skipping x11-wm/xfce4-desktop | xfce4-desktop-4.20.0: Dependent port x11-toolkits/libwnck3 | libwnck3-3.36.0_2 fa iled
> [00:00:36] [01] [00:00:20] Skipping deskutils/xfce4-notifyd | xfce4-notifyd-0.9.6_1: Dependent port x11-toolkits/libwnck3 | libwnck3-3.36.0_ 2 failed
> [00:00:36] [01] [00:00:20] Skipping x11-wm/xfce4-panel | xfce4-panel-4.20.0: Dependent port x11-toolkits/libwnck3 | libwnck3-3.36.0_2 failed
> [00:00:36] [01] [00:00:20] Skipping sysutils/xfce4-power-manager | xfce4-power-manager-4.20.0: Dependent port x11-toolkits/libwnck3 | libwnc k3-3.36.0_2 failed
> [00:00:36] [01] [00:00:20] Skipping x11-wm/xfce4-session | xfce4-session-4.20.0: Dependent port x11-toolkits/libwnck3 | libwnck3-3.36.0_2 fa iled
> [00:00:36] [01] [00:00:20] Skipping x11-wm/xfce4-wm | xfce4-wm-4.20.0: Dependent port x11-toolkits/libwnck3 | libwnck3-3.36.0_2 failed
> [00:02:51] [02] [00:02:25] Finished databases/mariadb114-server | mariadb114-server-11.4.4: Success
> [00:02:52] [03] [00:02:25] Finished databases/mariadb1011-server | mariadb1011-server-10.11.10_1: Success
> [00:02:53] Stopping 13 builders
> [00:02:54] Creating pkg repository
> Creating repository in /tmp/packages: 100%
> Packing files for repository: 100%
> [00:03:12] Committing packages to repository: /poudriere/data/packages/140amd64-2025Q1/.real_1737037274 via .latest symlink
> [00:03:12] Removing old packages
> [00:03:12] Built ports: net/rsync databases/mariadb114-server databases/mariadb1011-server
> [00:03:12] Failed ports: x11-toolkits/libwnck3:build
> [00:03:12] Skipped ports: deskutils/xfce4-notifyd sysutils/xfce4-power-manager x11-fm/thunar x11-wm/xfce4 x11-wm/xfce4-desktop x11-wm/xfce4- panel x11-wm/xfce4-session x11-wm/xfce4-wm x11/libxfce4windowing
> [140amd64-2025Q1] [2025-01-16_14h18m02s] [committing] Queued: 13 Built: 3 Failed: 1 Skipped: 9 Ignored: 0 Fetched: 0 Tobuild: 0 Time: 00:03:11
> [00:03:12] Logs: /poudriere/data/logs/bulk/140amd64-2025Q1/2025-01-16_14h18m02s
> [00:03:12] Cleaning up
> [00:03:12] Unmounting file systems
> real 192.36
> user 518.74
> sys 219.50
> titan#
>
>
> Then straight away after that :
> titan# /usr/bin/time -p idprio 0 poudriere bulk -r -j 140amd64 -p 2025Q1 -f /root/pkg.list
Note to readers: That is the same command as above repeated, up to
a white space difference.
> [00:00:00] Updating portstree "2025Q1" with git+https... done
> [00:00:00] Creating the reference jail... done
> [00:00:00] Mounting system devices for 140amd64-2025Q1
> [00:00:00] Stashing existing package repository
> [00:00:00] Mounting ccache from: /var/cache/ccache
> [00:00:00] Mounting ports from: /poudriere/ports/2025Q1
> [00:00:00] Mounting packages from: /poudriere/data/packages/140amd64-2025Q1
> [00:00:00] Mounting distfiles from: /poudriere/distfiles
> /etc/resolv.conf -> /poudriere/data/.m/140amd64-2025Q1/ref/etc/resolv.conf
> [00:00:00] Starting jail 140amd64-2025Q1
> Updating /var/run/os-release done.
> [00:00:00] Will build as nobody:nobody (65534:65534)
> [00:00:00] Logs: /poudriere/data/logs/bulk/140amd64-2025Q1/2025-01-17_07h00m48s
> [00:00:00] Loading MOVED for /poudriere/data/.m/140amd64-2025Q1/ref/usr/ports
> [00:00:01] Ports supports: FLAVORS SUBPACKAGES SELECTED_OPTIONS
> [00:00:01] Inspecting ports tree for modifications to git checkout... no
> [00:00:02] Ports top-level git hash: ea08b2ed7
> [00:00:02] Gathering ports metadata
> [00:00:07] Calculating ports order and dependencies
> [00:00:09] Trimming IGNORED and blacklisted ports
> [00:00:10] Sanity checking the repository
> [00:00:10] Checking packages for incremental rebuild needs
> [00:00:13] Deleting stale symlinks... done
> [00:00:13] Deleting empty directories... done
> [00:00:13] Unqueueing existing packages
> [00:00:14] Unqueueing orphaned build dependencies
> [00:00:14] Sanity checking build queue
> [00:00:14] Processing PRIORITY_BOOST
> [00:00:15] Balancing pool
> [140amd64-2025Q1] [2025-01-17_07h00m48s] [balancing_pool] Queued: 10 Built: 0 Failed: 0 Skipped: 0 Ignored: 0 Fetched: 0 Tobuild: 10 Time: 00:00:14
> [00:00:15] Recording filesystem state for prepkg... done
> [00:00:15] Building 10 packages using up to 10 builders
> [00:00:15] Hit CTRL+t at any time to see build progress and stats
> [00:00:15] [01] [00:00:00] Builder starting
> [00:00:15] [01] [00:00:00] Builder started
> [00:00:15] [01] [00:00:00] Building x11-toolkits/libwnck3 | libwnck3-3.36.0_2
> [00:00:33] [01] [00:00:18] Finished x11-toolkits/libwnck3 | libwnck3-3.36.0_2: Failed: build
So the failure for libwnck3-3.36.0_2 seems repeatable instead of
racy/random. This tends to make it easier to discover what is
wrong and how to have it fixed.
> [00:00:33] [01] [00:00:18] Skipping x11/libxfce4windowing | libxfce4windowing-4.20.0: Dependent port x11-toolkits/libwnck3 | libwnck3-3.36.0_2 failed
> [00:00:33] [01] [00:00:18] Skipping x11-fm/thunar | thunar-4.20.1: Dependent port x11-toolkits/libwnck3 | libwnck3-3.36.0_2 failed
> [00:00:33] [01] [00:00:18] Skipping x11-wm/xfce4 | xfce-4.20: Dependent port x11-toolkits/libwnck3 | libwnck3-3.36.0_2 failed
> [00:00:33] [01] [00:00:18] Skipping x11-wm/xfce4-desktop | xfce4-desktop-4.20.0: Dependent port x11-toolkits/libwnck3 | libwnck3-3.36.0_2 failed
> [00:00:33] [01] [00:00:18] Skipping deskutils/xfce4-notifyd | xfce4-notifyd-0.9.6_1: Dependent port x11-toolkits/libwnck3 | libwnck3-3.36.0_2 failed
> [00:00:33] [01] [00:00:18] Skipping x11-wm/xfce4-panel | xfce4-panel-4.20.0: Dependent port x11-toolkits/libwnck3 | libwnck3-3.36.0_2 failed
> [00:00:33] [01] [00:00:18] Skipping sysutils/xfce4-power-manager | xfce4-power-manager-4.20.0: Dependent port x11-toolkits/libwnck3 | libwnck3-3.36.0_2 failed
> [00:00:33] [01] [00:00:18] Skipping x11-wm/xfce4-session | xfce4-session-4.20.0: Dependent port x11-toolkits/libwnck3 | libwnck3-3.36.0_2 failed
> [00:00:33] [01] [00:00:18] Skipping x11-wm/xfce4-wm | xfce4-wm-4.20.0: Dependent port x11-toolkits/libwnck3 | libwnck3-3.36.0_2 failed
> [00:00:33] Stopping 10 builders
> [00:00:33] Creating pkg repository
> Creating repository in /tmp/packages: 100%
> Packing files for repository: 100%
> [00:00:52] Committing packages to repository: /poudriere/data/packages/140amd64-2025Q1/.real_1737097300 via .latest symlink
> [00:00:52] Removing old packages
> [00:00:52] Failed ports: x11-toolkits/libwnck3:build
> [00:00:52] Skipped ports: deskutils/xfce4-notifyd sysutils/xfce4-power-manager x11-fm/thunar x11-wm/xfce4 x11-wm/xfce4-desktop x11-wm/xfce4-panel x11-wm/xfce4-session x11-wm/xfce4-wm x11/libxfce4windowing
> [140amd64-2025Q1] [2025-01-17_07h00m48s] [committing] Queued: 10 Built: 0 Failed: 1 Skipped: 9 Ignored: 0 Fetched: 0 Tobuild: 0 Time: 00:00:51
> [00:00:52] Logs: /poudriere/data/logs/bulk/140amd64-2025Q1/2025-01-17_07h00m48s
> [00:00:52] Cleaning up
> [00:00:52] Unmounting file systems
> real 52.84
> user 183.75
> sys 123.83
> titan#
>
> I see things like that ALL the time. Daily.
Racy/random failures can happen, but repeatabilty is more typical.
In other words: not unexpected if one tests for repeatability like
the above did.
>>> . . .
>>>
>>> Why do I always see things like this :
>> Just rhetorical? (See more later.)
>>
>
>
> Sort of. Yes. I think the specifics are just pine needles on the little
> trees in a big big forest. Some are going to be dead and fall away. Some
> will need a bit of trimming on some of the trees. That is the nature of
> the forest. However I have been looking at poudriere build failures for
> weeks. Right, it is the holidays and so I do not expect much and really
> should not. I don't.
>
>> I've not seen anything presented that might allow someone to help
>> improve what happens in your context --or even to test if they can
>> specifically reproduce any of your builds' specific failures, not
>> even if they can match which ports blocked the potentially more
>> overall builds.
>> The note reads more as an expression of furstation than anything
>> else, no attempt at getting help. Or, maybe, an attempt, in part,
>> to see how many "me too" responses show up.
>
> I was mostly dipping my toes in the water and waiting for the usual
> flames. I have been doing something or other related to software package
> stuff for a while now. I am not surprised when things go sideways a bit.
> I am, yes, frustrated when building the big three window manager bits
> and some database tools falls over. Daily. Weekly.
>
>
>> The note was sent to freebsd-current. I'm not sure if I should
>> read someting into that or not.
>
> Well, not sure else what/where to do what? I figure a 15.0-CURRENT
> machine is what one would need to build with.
That context was not specified nor obvious. Sometimes explaining
why a specific list was picked as be useful to those reading.
It is true that you can not reasonably use 13.4 to build 14.*+]
or main. Use of main usually works for going backwards.
But I'll note that what is tested by use on the official builders
is only the specific snapshots they install for booting the
machines. They do not update which snapshot for booting all that
often.
> Certainly for every
> reasonable version from 13.4 upwards. Just a guess but I think that
> is what would need.
>
>> Nothing hints at the platform(s) used vs. not or that fail
>> vs. not.
>> Prior to my FreeBSD use stopping on 2025-Jan-7, bulk builds that
>> I had done worked fine. But it is likely to be a rather different
>> workload vs. yours.
>
> Yep likely. I have a package list of stuff and it usually rounds out to
> a grand whopping 1500+ things. Never more than 1600. Not ever.
>
>> Looking at the FreeBD package builders, I do see latest for
>> main armv7 having:
>> x11/kde5 being blocked by libfido2-1.15.0
>> x11-wm/xfce being blocked by vte3-0.70.2_5
>> x11/xfce4-goodies being blocked by vte3-0.70.2_5
>> x11/xfce4-terminal being blocked by vte3-0.70.2_5
>
> Well, that is armv7? I am looking at AMD64 on a Xeon machine.
That was not obvious from the original message. I had to
explore without being able to know how to match your
context.
> More or
> less AMD64 or x86_64 depending on whose church we sit in.
>
>> But I did not see LXDE being blocked.
>> I did not see such blocks for amd64 or arm64.
>> Where I do see all of KDE5, LXDE, and XFCE blocked
>> is by rust-1.81.0 in:
>>
>
> Funny. I have rust built on RISC-V :
>
> enceladus$ uname -apKU
> FreeBSD enceladus 15.0-CURRENT FreeBSD 15.0-CURRENT #2 main-n274434-2e09cef8dc6f: Sun Dec 29 05:26:22 GMT 2024 root@enceladus:/usr/obj/usr/src/riscv.riscv64/sys/SIFIVE-COMPAT11 riscv riscv64 1500029 1500029
> enceladus$
> enceladus$ rustc --version
> rustc 1.83.0 (90b35a623 2024-11-26) (built from a source tarball)
> enceladus$
The armv7 rust issue is a known and fixed issue that the
Quarterly armv7 builds have not yet caught up to.
> However that is a wild red herring. Pay no attention to that.
>
>> But I've no clue if any of this matches your example
>> build failures.
>
>
> I am thankful for the reply. I am curious. Given a hundred users of
> FreeBSD the question would be how many build stuff themselves? The
> numbers may be shockingly low. I have no idea. I am just guessing.
>
There is no place to look up figures for that.
An overall summary:
Trying to keep the likes of 36295 packages (and growing)
building is a non-trivial activity. Most are not
independent of some others. A failure of one can lead to
4000+ other packages to not even be attempted since the
one common prerequisite for those 4000+ happens to end
up not being available to use. (I used the lang/rust
for armv7 example for illustration.)
If you are building packages for yourself, you are going
to see that structural property in various ways at various
times.
On the information I have I cannot be more specific about
your specific build problems. Nor do I have the context to
do test builds of my own at this time.
===
Mark Millard
marklmi at yahoo.com