Re: Possible to automatically pull dependencies from a secondary repo (pkg/poudriere)

I forgot to mention, this is basically happening because there are some options differences somewhere in the dependency tree even if I'm not explicitly customizing them. So I've noticed that if I do a pkg upgrade using a single repo (pointing to my local one specifically), then pkg can detect which package needs to be upgraded (has differences in some way). Once I upgrade that package, I can the run pkg upgrade normally (without -r <repo>), and it will no longer try and install those 3 build dependencies from the official repos. The issue will re-arise the next time I build poudriere updates.

root@leslie:~ # pkg update
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating Leslie repository catalogue...
Leslie repository is up to date.
All repositories are up to date.

root@leslie:~ # pkg upgrade
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating Leslie repository catalogue...
Leslie repository is up to date.
All repositories are up to date.
Checking for upgrades (12 candidates): 100%
Processing candidates (12 candidates): 100%
Checking integrity... done (0 conflicting)
The following 3 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
binutils: 2.40_4,1 [FreeBSD]
gcc12: 12.2.0_6 [FreeBSD]
tcl86: 8.6.13 [FreeBSD]

Number of packages to be installed: 3

The process will require 460 MiB more space.

Proceed with this action? [y/N]: n

root@leslie:~ # pkg upgrade -r Leslie
Updating Leslie repository catalogue...
Leslie repository is up to date.
All repositories are up to date.
Checking for upgrades (2 candidates): 100%
Processing candidates (2 candidates): 100%
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
libgcrypt: 1.9.4_1 -> 1.10.2 [Leslie]

Number of packages to be upgraded: 1

Proceed with this action? [y/N]: y
[1/1] Upgrading libgcrypt from 1.9.4_1 to 1.10.2...
[1/1] Extracting libgcrypt-1.10.2: 100%

root@leslie:~ # pkg update
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating Leslie repository catalogue...
Leslie repository is up to date.
All repositories are up to date.
root@leslie:~ # pkg upgrade
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating Leslie repository catalogue...
Leslie repository is up to date.
All repositories are up to date.
Checking for upgrades (12 candidates): 100%
Processing candidates (12 candidates): 100%
Checking integrity... done (0 conflicting)
Your packages are up to date.
root@leslie:~ # pkg autoremove
Checking integrity... done (0 conflicting)
Nothing to do.
Jonathan Vasquez
PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
Sent with ProtonMail Secure Email

------- Original Message -------
On Monday, June 26th, 2023 at 09:22, Jonathan Vasquez <jon@xyinn.org> wrote:

> Hello all,
>
> I was wondering if there was a way to easily tell pkg to also pull in all dependencies from my primary repo (built with poudriere) and not just the main requested package? I've noticed the following behavior:
>
> root@leslie:~ # pkg update
> Updating FreeBSD repository catalogue...
> FreeBSD repository is up to date.
> Updating Leslie repository catalogue...
> Leslie repository is up to date.
> All repositories are up to date.
> root@leslie:~ # pkg upgrade
> Updating FreeBSD repository catalogue...
> FreeBSD repository is up to date.
> Updating Leslie repository catalogue...
> Leslie repository is up to date.
> All repositories are up to date.
> Checking for upgrades (12 candidates): 100%
> Processing candidates (12 candidates): 100%
> Checking integrity... done (0 conflicting)
> The following 3 package(s) will be affected (of 0 checked):
>
> New packages to be INSTALLED:
> binutils: 2.40_4,1 [FreeBSD]
> gcc12: 12.2.0_6 [FreeBSD]
> tcl86: 8.6.13 [FreeBSD]
>
> ...
>
> root@leslie:~ # pkg autoremove
> Checking integrity... done (0 conflicting)
> Deinstallation has been requested for the following 3 packages:
>
> Installed packages to be REMOVED:
> binutils: 2.40_4,1
> gcc12: 12.2.0_6
> tcl86: 8.6.13
>
> Number of packages to be removed: 3
>
> ...
>
> We can see in the following that if I explicitly ask for gcc12, it will correctly pull it from my main repo:
>
> root@leslie:~ # pkg install gcc12
> Updating FreeBSD repository catalogue...
> FreeBSD repository is up to date.
> Updating Leslie repository catalogue...
> Leslie repository is up to date.
> All repositories are up to date.
> Checking integrity... done (0 conflicting)
> The following 3 package(s) will be affected (of 0 checked):
>
> New packages to be INSTALLED:
> binutils: 2.40_4,1 [FreeBSD]
> gcc12: 12.2.0_6 [Leslie]
> tcl86: 8.6.13 [FreeBSD]
>
> Number of packages to be installed: 3I correctly have priority set for my main repo and using the FreeBSD one as a fallback.
>
> root@leslie:~ # ls -lh /usr/local/etc/pkg/repos/
> total 9
> -rw-r--r-- 1 root wheel 110B Jun 25 21:16 00_Leslie.conf-rw-r--r-- 1 root wheel 174B Jun 26 07:33 01_FreeBSD.conf
>
> root@leslie:~ # cat /usr/local/etc/pkg/repos/00_Leslie.conf
> Leslie: {
> url: "file:///usr/local/poudriere/data/packages/13amd64-main/",
> enabled: yes,
> priority: 100
> }
> root@leslie:~ # cat /usr/local/etc/pkg/repos/01_FreeBSD.conf
> FreeBSD: {
> url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
> mirror_type: "srv",
> signature_type: "fingerprints",
> fingerprints: "/usr/share/keys/pkg",
> enabled: yes
> }
>
> To not get to into the weeds as to why I'm doing this (and I do understand the concerns people have with mixing ports/packages), some particular binaries just take way too long even on my 11th gen intel framework laptop (8 cores, 32 gb of ram, nvme drive) (chromium, electron22, and dont even think about GCC w/ LTO optimizations, I had to explicitly switch to 'STANDARD_BOOTSTRAP' or it basically wouldn't finish). In addition, due to the way the ports tree is built in the official freebsd repos, I've ran into situations where installing modules from the tree are incompatible with the regular freebsd generic kernel (like trying to load vboxdrv​ from the virtualbox-ose-kmod package with a regular 13.2 kernel). I believe that's because freebsd builds for the oldest supported release. Due to this, I basically plucked out the virtualbox-ose (and co) and placed it in my "locally built poudriere repo's packages file". Normally I would either just use the official ports tree, or I would built everything, but since some of my packages just take forever to build, and the official freebsd pkg tree can leave me with modules that don't load, I'm basically forced to handle this situation myself even if I dont need to really have custom ports options. I'm fine doing it, there just needs to be a nice smooth-ish workaround.
>
> CONSERVATIVE_UPGRADE is set to the default which is true (and that's fine since I want it to continue pulling in upgrades for packages installed from my local repo, from the local repo). But I would also like all dependencies for a package installed from a local repo to also pull from that same repo to maintain consistency.
>
> I'm currently running poudriere-3.3.7_1, but I did try I also tried to use poudriere-devel in order to take advantage of the fetch prebuilt packages feature. This would have allowed me to basically pull everything from the official repos, and only build the specific packages that I want, while maintaining dependency integrity. Unfortunately it wasn't really detecting many prebuilt packages and still wanted to build a lot of stuff, even with no customization changes. This is even with me having an up to date ports tree and all of that. So it wasn't a situation where the official repos were too different than my local poudriere ports tree (main branch), and thus it was causing a mismatch.
>
> PACKAGE_FETCH_BRANCH=latest
> PACKAGE_FETCH_WHITELIST="libreoffice* webkit2-gtk3* gcc*" (etc)
>
> Thank you for any advice and feedback,
>
> Jonathan Vasquez
> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
> Sent with ProtonMail Secure Email