Problems with packaging /etc and etcupdate(8)

[John Baldwin]

I had assumed that in a pkgbase setup, one wouldn't use etcupdate or mergemaster.
Instead, my understanding is that pkg was going to "learn" about configuration
files and would grow etcupdate-like functionality to merge in changes to configuration
files during upgrades (and would also support 'pkg confdiff', etc. so that
'pkg confdiff -a' would replace 'etcupdate diff' for example, but would also support
packages built from ports, not just base system conf files).  Each pkgbase package
would contain its associated /etc files as configuration files in this scheme.

On Monday, November 13, 2017 05:20:54 PM Glen Barber wrote:
> Adding jhb, in case I'm doing something wrong with etcupdate(8) in this
> approach, but gratuitously pinging the list to solicit ideas.
> 
> Glen
> 
> On Wed, Sep 27, 2017 at 04:30:12PM +0000, Glen Barber wrote:
> > Hi,
> > 
> > Over the past two weeks, I spent quite a bit of time looking into using
> > etcupdate(8) to bootstrap /var/db/etcupdate for use with pkg(8), and
> > here are some of the issues I have run into.  The most current diff
> > tested is attached.
> > 
> > 1) A new 'create-packages-files' target was added, which invokes the
> >    etcupdate(8) 'extract' command using WSTAGEDIR for the DESTDIR.
> >    Despite passing 'TAGS=package=etc' and 'PACKAGE=etc' to etcupdate(8)
> >    '-M' flag, in addition to '-DNO_ROOT', some files still fail to be
> >    tagged in WSTAGEDIR/var/db/etcupdate/current/METALOG.  To work around
> >    this, a hack was added to release/scripts/mtree-to-plist.awk to trap
> >    and tag files when '-v etcupdate=yes' is passed to awk(1) in this
> >    target.
> > 
> > 2) Some files do not automatically get created by etcupdate(8), as noted
> >    in the manual page.  For example, pwd_mkdb(8), cap_mkdb(1),
> >    newaliases(1), tzsetup(8), and services_mkdb(8) are invoked after
> >    merging files with the newly-created tree.  This causes, for example,
> >    files like /etc/passwd to be nonexistent when creating a package.  To
> >    work around this, another hack was added to the mtree-to-plist.awk
> >    script to skip nonexistent files in WSTAGEDIR/var/db/etcupdate.  This
> >    is still non-ideal, I think.
> > 
> > 3) Setting 'prefix = /var/db/etcupdate' in the release/packages/etc.ucl
> >    file appears to be ignored.  Files in the 'etc' package conflict with
> >    files installed by other packages, such as runtime, autofs, and so
> >    on, with conflicting files being things like /etc/autofs/include_ldap
> >    or other similarly-installed files within /etc.  To work around this
> >    when packaging files, a hack was added to the mtree-to-plist.awk
> >    script (again) to prepend '/var/db/etcupdate' to the file path in
> >    order to get the correct files included in the package.  (This is
> >    somewhat in parallel with (2) above.)
> > 
> > 4) When running etcupdate(8) after hacking around some of the issues
> >    above, I discovered that /var/db/etcupdate/current in the root of the
> >    system in which it is being run disappears.  Additionally, I had to
> >    restore files within /etc (such as master.passwd(5), passwd(5), etc.)
> >    in order to make my test system usable again.  In addition to
> >    replacing files that should have been merged, 'pkg check' against the
> >    'FreeBSD-etc' package reports all of the files from the 'current'
> >    directory have vanished.
> > 
> > In addition to the above items, I also did some (albeit minimal) testing
> > with mergemaster(8), as it creates /var/db/mergemaster.mtree, however in
> > my brief testing it seems the mtree(8) file created is not compatible
> > with pkg(8) or the mtree-to-plist.awk script.  (Note: the mergemaster(8)
> > bits from my testing are not included in the diff, as I had to revert
> > a number of changes to ensure the correct diff was included in this
> > email.
> > 
> > I am at the point now where I am out of ideas on how to proceed, and
> > could use some help on figuring this out.  Any help on the above issues,
> > or anything in the Wiki would be greatly appreciated.
> > 
> > https://wiki.freebsd.org/PkgBase
> > 
> > Thanks,
> > 
> > Glen
> > 
> 
> > Index: Makefile
> > ===================================================================
> > --- Makefile	(revision 324062)
> > +++ Makefile	(working copy)
> > @@ -132,6 +132,7 @@
> >  	builddtb xdev xdev-build xdev-install \
> >  	xdev-links native-xtools stageworld stagekernel stage-packages \
> >  	create-packages-world create-packages-kernel create-packages \
> > +	create-files-packages \
> >  	packages installconfig real-packages sign-packages package-pkg \
> >  	print-dir test-system-compiler
> >  
> > Index: Makefile.inc1
> > ===================================================================
> > --- Makefile.inc1	(revision 324063)
> > +++ Makefile.inc1	(working copy)
> > @@ -1555,6 +1555,7 @@
> >  
> >  .ORDER:		stage-packages create-packages
> >  .ORDER:		create-packages create-world-packages
> > +.ORDER:		create-packages create-files-packages
> >  .ORDER:		create-packages create-kernel-packages
> >  .ORDER:		create-packages sign-packages
> >  
> > @@ -1596,6 +1597,12 @@
> >  			DESTDIR=${WSTAGEDIR} \
> >  			PKG_VERSION=${PKG_VERSION} create-world-packages
> >  
> > +create-packages-files:	_pkgbootstrap _repodir .PHONY
> > +	${_+_}@cd ${.CURDIR}; \
> > +		${MAKE} -f Makefile.inc1 \
> > +			DESTDIR=${WSTAGEDIR} \
> > +			PKG_VERSION=${PKG_VERSION} create-files-packages
> > +
> >  create-packages-kernel:	_pkgbootstrap _repodir .PHONY
> >  	${_+_}@cd ${.CURDIR}; \
> >  		${MAKE} -f Makefile.inc1 \
> > @@ -1603,8 +1610,25 @@
> >  			PKG_VERSION=${PKG_VERSION} DISTDIR=kernel \
> >  			create-kernel-packages
> >  
> > -create-packages: .PHONY create-packages-world create-packages-kernel
> > +create-packages: .PHONY create-packages-world create-packages-files create-packages-kernel
> >  
> > +create-files-packages:	_pkgbootstrap .PHONY
> > +	mkdir -p ${WSTAGEDIR}/var/db/etcupdate/current
> > +	cd ${WSTAGEDIR} ; \
> > +	${WSTAGEDIR}/usr/sbin/etcupdate extract -B -M \
> > +		"TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} -DNO_ROOT TAGS=package=etc PACKAGE=etc" \
> > +		-s ${SRCDIR} -d ${WSTAGEDIR}/var/db/etcupdate
> > +	awk -f ${SRCDIR}/release/scripts/mtree-to-plist.awk \
> > +		-v etcupdate=yes -v ws=${WSTAGEDIR} \
> > +		${WSTAGEDIR}/var/db/etcupdate/current/METALOG
> > +	sh ${SRCDIR}/release/packages/generate-ucl.sh -o etc \
> > +		-s ${SRCDIR} -u ${WSTAGEDIR}/etc.ucl
> > +	${PKG_CMD} -o ABI_FILE=${WSTAGEDIR}/bin/sh -o ALLOW_BASE_SHLIBS=yes \
> > +		create -M ${WSTAGEDIR}/etc.ucl \
> > +		-p ${WSTAGEDIR}/etc.plist \
> > +		-r ${WSTAGEDIR} \
> > +		-o ${REPODIR}/$$(${PKG_CMD} -o ABI_FILE=${WSTAGEDIR}/bin/sh config ABI)/${PKG_VERSION}
> > +
> >  create-world-packages:	_pkgbootstrap .PHONY
> >  	@rm -f ${WSTAGEDIR}/*.plist 2>/dev/null || :
> >  	@cd ${WSTAGEDIR} ; \
> > Index: release/packages/etc.ucl
> > ===================================================================
> > --- release/packages/etc.ucl	(nonexistent)
> > +++ release/packages/etc.ucl	(working copy)
> > @@ -0,0 +1,22 @@
> > +#
> > +# $FreeBSD: head/release/packages/runtime.ucl 322327 2017-08-09 19:16:54Z gjb $
> > +#
> > +
> > +name = "FreeBSD-etc"
> > +origin = "base"
> > +version = "%VERSION%"
> > +comment = "etc package"
> > +categories = [ base ]
> > +maintainer = "re at FreeBSD.org"
> > +www = "https://www.FreeBSD.org"
> > +prefix = "/var/db/etcupdate"
> > +licenselogic = "single"
> > +licenses = [ BSD2CLAUSE ]
> > +desc = <<EOD
> > +etc package
> > +EOD
> > +scripts: {
> > +	post-install = <<EOD
> > +	echo "Please use etcupdate(8) or mergemaster(8) to update"
> > +	echo "files under /etc."
> > +EOD
> > Index: release/scripts/mtree-to-plist.awk
> > ===================================================================
> > --- release/scripts/mtree-to-plist.awk	(revision 324062)
> > +++ release/scripts/mtree-to-plist.awk	(working copy)
> > @@ -28,6 +28,13 @@
> >  			tags=tags""_kernconf
> >  		}
> >  	}
> > +	if (etcupdate != "") {
> > +		if (length(gsub(/tags=/, "", tags) == 0))
> > +			tags="package=etc"
> > +		if ((length(type) != 0) && (type == "dir")) {
> > +			next
> > +		}
> > +	}
> >  	if (length(tags) == 0)
> >  		next
> >  	if (tags ~ /package=/) {
> > @@ -76,5 +83,17 @@
> >  		output=pkgname".plist"
> >  	}
> >  
> > +	if (etcupdate != "") {
> > +		pfx="/var/db/etcupdate/current"
> > +		filename=ws""pfx""$1
> > +		if (system("test -e " filename) == 0) {
> > +			print "@"type"("uname","gname","mode","flags") " pfx""$1 > output
> > +			next
> > +		} else {
> > +				next
> > +		}
> > +		next
> > +	}
> > +
> >  	print "@"type"("uname","gname","mode","flags") " $1 > output
> >  }
> 
> 
> 


-- 
John Baldwin