new package system proposal

[Mel Flynn]

Sorry to jump in late.

On Saturday 04 April 2009 16:13:22 Chris Whitehouse wrote:

> pkg_add somewhat addresses this but it doesn't work quite as well as
> ports because of possible version mismatches.
>
> The suggestion below is not aimed at servers because they have
> completely different requirements.
>
> My suggestion is to start with a ports tree that is fixed in time. Make
> that ports tree available as part of this package system and compile a
> typical desktop set of ports, particularly choosing ones which are large
> or have many dependencies. When it is all complete release it and start
> again. Surely quite a wide selection of desktops, wm's and apps could be
> compiled in a couple of weeks?
>
> Modify pkg_add so that it can be told to use this 'snapshot' including
> downloading the fixed ports tree that was used.

I've already thought this through and you're going about it the wrong way. 
Version/package mismatches are caused by two things:
1) The buildservers do not repack dependants of an updated port
2) Installed slave ports as dependency cannot be identified by a package that 
has the master port built in. Ex: if installing kdelibs3-nocups through 
pkg_add, anything using it will download kdelibs3, try to install and fail.

The first is easy to solve, by iterating through the built packages and 
repacking them after reinstallation so that +CONTENTS is updated correctly 
with the new dependency. I've written a tool pkgsync which does that, but 
there's still some corner cases I need to fix and I'm not happy with the time 
it takes to read deps from 800+ packages, so might consider using aio(4) or 
fixing up my algorithm.

The second case is more difficult, as it requires "flexible dependencies". 
I've fixed this myself, by hacking port Makefiles to correctly set the 
dependency origin, but this only applies to "one build" (meaning: my 
buildserver only makes packages with a fixed set of slave ports, if I'd want 
to use CUPS, I'd have to create a seperate buildserver or jail).

The most important thing is that you don't need a snapshot ports tree if 1) is 
met. What you also need for binary upgrading is an index format that doesn't 
need a ports tree and is consistent with what is actually available on the 
given build server.

I've solved this like so:
#!/bin/sh
# vim: ts=4 sw=4 noet ai tw=78
# $Coar: pkgtools/pkgindex/pkgindex.sh,v 1.5 2008/04/17 21:43:46 mdev Exp $

AWK=${AWK:="/usr/bin/awk"}
PKGDB=${PKGDB:="/var/db/pkg"}
PACKAGES=${PACKAGES:="/home/packages-6"}
PKGINDEX=${PKGINDEX:="${PACKAGES}/All/INDEX.bz2"}

TMPINDEX=/tmp/INDEX.$$

# clean up a previous halted version that had the same pid
[ -f ${TMPINDEX} ] && rm -f ${TMPINDEX}
[ -f ${TMPINDEX}.bz2 ] && rm -f ${TMPINDEX}.bz2

DONE=0
echo -n "Creating new in index in $PACKAGES/All"
cd ${PACKAGES}/All
for pkg in *.tbz; do
        origin=$(pkgorigin ${pkg})
        hash=$(sha256 -q ${pkg})
        size=$(ls -aLln ${pkg} | ${AWK} '{ print $5}')
        echo "$origin:${pkg}:$hash:$size" >> ${TMPINDEX}
        DONE=$((${DONE} + 1))
        if test $((${DONE} % 10)) -eq 0; then
                if test $((${DONE} % 50)) -eq 0; then
                        echo -n ${DONE}
                else
                        echo -n .
                fi
        fi
done
echo done.
bzip2 ${TMPINDEX}
mv -v ${TMPINDEX}.bz2 ${PKGINDEX}

This gives you origin, pkgname, sha256 hash and filesize for download 
verification. All you need to determine whether a package is eligible for 
binary upgrade.

pkgorigin is a utility I wrote that quickly gets an origin out of a package 
without the need for a staging directory. Attached as shar if it makes it to 
the list.

Btw, I don't think supporting various OPTIONS is a good idea. Port maintainers 
should properly use PACKAGE_BUILDING. There's no such thing as "good 
defaults", so try to satisfy what you think is the majority and redirect 
complaints to the infamous bit bucket. Proper use also includes disabling 
RESTRICTED and IS_INTERACTIVE ports.
-- 
Mel