Problems with packaging /etc and etcupdate(8)

[Glen Barber]

Adding jhb, in case I'm doing something wrong with etcupdate(8) in this
approach, but gratuitously pinging the list to solicit ideas.

Glen

On Wed, Sep 27, 2017 at 04:30:12PM +0000, Glen Barber wrote:
> Hi,
> 
> Over the past two weeks, I spent quite a bit of time looking into using
> etcupdate(8) to bootstrap /var/db/etcupdate for use with pkg(8), and
> here are some of the issues I have run into.  The most current diff
> tested is attached.
> 
> 1) A new 'create-packages-files' target was added, which invokes the
>    etcupdate(8) 'extract' command using WSTAGEDIR for the DESTDIR.
>    Despite passing 'TAGS=package=etc' and 'PACKAGE=etc' to etcupdate(8)
>    '-M' flag, in addition to '-DNO_ROOT', some files still fail to be
>    tagged in WSTAGEDIR/var/db/etcupdate/current/METALOG.  To work around
>    this, a hack was added to release/scripts/mtree-to-plist.awk to trap
>    and tag files when '-v etcupdate=yes' is passed to awk(1) in this
>    target.
> 
> 2) Some files do not automatically get created by etcupdate(8), as noted
>    in the manual page.  For example, pwd_mkdb(8), cap_mkdb(1),
>    newaliases(1), tzsetup(8), and services_mkdb(8) are invoked after
>    merging files with the newly-created tree.  This causes, for example,
>    files like /etc/passwd to be nonexistent when creating a package.  To
>    work around this, another hack was added to the mtree-to-plist.awk
>    script to skip nonexistent files in WSTAGEDIR/var/db/etcupdate.  This
>    is still non-ideal, I think.
> 
> 3) Setting 'prefix = /var/db/etcupdate' in the release/packages/etc.ucl
>    file appears to be ignored.  Files in the 'etc' package conflict with
>    files installed by other packages, such as runtime, autofs, and so
>    on, with conflicting files being things like /etc/autofs/include_ldap
>    or other similarly-installed files within /etc.  To work around this
>    when packaging files, a hack was added to the mtree-to-plist.awk
>    script (again) to prepend '/var/db/etcupdate' to the file path in
>    order to get the correct files included in the package.  (This is
>    somewhat in parallel with (2) above.)
> 
> 4) When running etcupdate(8) after hacking around some of the issues
>    above, I discovered that /var/db/etcupdate/current in the root of the
>    system in which it is being run disappears.  Additionally, I had to
>    restore files within /etc (such as master.passwd(5), passwd(5), etc.)
>    in order to make my test system usable again.  In addition to
>    replacing files that should have been merged, 'pkg check' against the
>    'FreeBSD-etc' package reports all of the files from the 'current'
>    directory have vanished.
> 
> In addition to the above items, I also did some (albeit minimal) testing
> with mergemaster(8), as it creates /var/db/mergemaster.mtree, however in
> my brief testing it seems the mtree(8) file created is not compatible
> with pkg(8) or the mtree-to-plist.awk script.  (Note: the mergemaster(8)
> bits from my testing are not included in the diff, as I had to revert
> a number of changes to ensure the correct diff was included in this
> email.
> 
> I am at the point now where I am out of ideas on how to proceed, and
> could use some help on figuring this out.  Any help on the above issues,
> or anything in the Wiki would be greatly appreciated.
> 
> https://wiki.freebsd.org/PkgBase
> 
> Thanks,
> 
> Glen
> 

> Index: Makefile
> ===================================================================
> --- Makefile	(revision 324062)
> +++ Makefile	(working copy)
> @@ -132,6 +132,7 @@
>  	builddtb xdev xdev-build xdev-install \
>  	xdev-links native-xtools stageworld stagekernel stage-packages \
>  	create-packages-world create-packages-kernel create-packages \
> +	create-files-packages \
>  	packages installconfig real-packages sign-packages package-pkg \
>  	print-dir test-system-compiler
>  
> Index: Makefile.inc1
> ===================================================================
> --- Makefile.inc1	(revision 324063)
> +++ Makefile.inc1	(working copy)
> @@ -1555,6 +1555,7 @@
>  
>  .ORDER:		stage-packages create-packages
>  .ORDER:		create-packages create-world-packages
> +.ORDER:		create-packages create-files-packages
>  .ORDER:		create-packages create-kernel-packages
>  .ORDER:		create-packages sign-packages
>  
> @@ -1596,6 +1597,12 @@
>  			DESTDIR=${WSTAGEDIR} \
>  			PKG_VERSION=${PKG_VERSION} create-world-packages
>  
> +create-packages-files:	_pkgbootstrap _repodir .PHONY
> +	${_+_}@cd ${.CURDIR}; \
> +		${MAKE} -f Makefile.inc1 \
> +			DESTDIR=${WSTAGEDIR} \
> +			PKG_VERSION=${PKG_VERSION} create-files-packages
> +
>  create-packages-kernel:	_pkgbootstrap _repodir .PHONY
>  	${_+_}@cd ${.CURDIR}; \
>  		${MAKE} -f Makefile.inc1 \
> @@ -1603,8 +1610,25 @@
>  			PKG_VERSION=${PKG_VERSION} DISTDIR=kernel \
>  			create-kernel-packages
>  
> -create-packages: .PHONY create-packages-world create-packages-kernel
> +create-packages: .PHONY create-packages-world create-packages-files create-packages-kernel
>  
> +create-files-packages:	_pkgbootstrap .PHONY
> +	mkdir -p ${WSTAGEDIR}/var/db/etcupdate/current
> +	cd ${WSTAGEDIR} ; \
> +	${WSTAGEDIR}/usr/sbin/etcupdate extract -B -M \
> +		"TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} -DNO_ROOT TAGS=package=etc PACKAGE=etc" \
> +		-s ${SRCDIR} -d ${WSTAGEDIR}/var/db/etcupdate
> +	awk -f ${SRCDIR}/release/scripts/mtree-to-plist.awk \
> +		-v etcupdate=yes -v ws=${WSTAGEDIR} \
> +		${WSTAGEDIR}/var/db/etcupdate/current/METALOG
> +	sh ${SRCDIR}/release/packages/generate-ucl.sh -o etc \
> +		-s ${SRCDIR} -u ${WSTAGEDIR}/etc.ucl
> +	${PKG_CMD} -o ABI_FILE=${WSTAGEDIR}/bin/sh -o ALLOW_BASE_SHLIBS=yes \
> +		create -M ${WSTAGEDIR}/etc.ucl \
> +		-p ${WSTAGEDIR}/etc.plist \
> +		-r ${WSTAGEDIR} \
> +		-o ${REPODIR}/$$(${PKG_CMD} -o ABI_FILE=${WSTAGEDIR}/bin/sh config ABI)/${PKG_VERSION}
> +
>  create-world-packages:	_pkgbootstrap .PHONY
>  	@rm -f ${WSTAGEDIR}/*.plist 2>/dev/null || :
>  	@cd ${WSTAGEDIR} ; \
> Index: release/packages/etc.ucl
> ===================================================================
> --- release/packages/etc.ucl	(nonexistent)
> +++ release/packages/etc.ucl	(working copy)
> @@ -0,0 +1,22 @@
> +#
> +# $FreeBSD: head/release/packages/runtime.ucl 322327 2017-08-09 19:16:54Z gjb $
> +#
> +
> +name = "FreeBSD-etc"
> +origin = "base"
> +version = "%VERSION%"
> +comment = "etc package"
> +categories = [ base ]
> +maintainer = "re at FreeBSD.org"
> +www = "https://www.FreeBSD.org"
> +prefix = "/var/db/etcupdate"
> +licenselogic = "single"
> +licenses = [ BSD2CLAUSE ]
> +desc = <<EOD
> +etc package
> +EOD
> +scripts: {
> +	post-install = <<EOD
> +	echo "Please use etcupdate(8) or mergemaster(8) to update"
> +	echo "files under /etc."
> +EOD
> Index: release/scripts/mtree-to-plist.awk
> ===================================================================
> --- release/scripts/mtree-to-plist.awk	(revision 324062)
> +++ release/scripts/mtree-to-plist.awk	(working copy)
> @@ -28,6 +28,13 @@
>  			tags=tags""_kernconf
>  		}
>  	}
> +	if (etcupdate != "") {
> +		if (length(gsub(/tags=/, "", tags) == 0))
> +			tags="package=etc"
> +		if ((length(type) != 0) && (type == "dir")) {
> +			next
> +		}
> +	}
>  	if (length(tags) == 0)
>  		next
>  	if (tags ~ /package=/) {
> @@ -76,5 +83,17 @@
>  		output=pkgname".plist"
>  	}
>  
> +	if (etcupdate != "") {
> +		pfx="/var/db/etcupdate/current"
> +		filename=ws""pfx""$1
> +		if (system("test -e " filename) == 0) {
> +			print "@"type"("uname","gname","mode","flags") " pfx""$1 > output
> +			next
> +		} else {
> +				next
> +		}
> +		next
> +	}
> +
>  	print "@"type"("uname","gname","mode","flags") " $1 > output
>  }



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-pkgbase/attachments/20171113/948b936d/attachment.sig>