svn commit: r360833 - head

Thu May 14 21:02:25 UTC 2020

On Tue, May 12, 2020 at 09:01:33PM -0500, Kyle Evans wrote:
> On Tue, May 12, 2020 at 5:16 PM Brooks Davis <brooks at freebsd.org> wrote:
> >
> > On Mon, May 11, 2020 at 01:45:14PM -0500, Kyle Evans wrote:
> > > On Mon, May 11, 2020 at 1:10 PM Brooks Davis <brooks at freebsd.org> wrote:
> > > >
> > > > On Sat, May 09, 2020 at 02:01:29AM +0000, Kyle Evans wrote:
> > > > > Author: kevans
> > > > > Date: Sat May  9 02:01:29 2020
> > > > > New Revision: 360833
> > > > > URL: https://svnweb.freebsd.org/changeset/base/360833
> > > > >
> > > > > Log:
> > > > >   installworld: attempt a certctl rehash at the tail end
> > > > >
> > > > >   This can be run as root or normal user with no problem; if they hadn't
> > > > >   twisted the WITHOUT_CAROOT knob, we'll attempt to use the host certctl to
> > > > >   rehash the DESTDIR. This would allow one to build systems WITHOUT_OPENSSL +
> > > > >   WITH_CAROOT with a populated /etc/ssl that they can then use with an
> > > > >   appropriate *ssl from somewhere else.
> > > > >
> > > > >   Cross-builds are fine because this will always use the host certctl, or just
> > > > >   nag if it's missing and it wasn't a WITHOUT_CAROOT build.
> > > > >
> > > > >   MFC after:  1 week
> > > > >   Differential Revision:      https://reviews.freebsd.org/D24641
> > > > >
> > > > > Modified:
> > > > >   head/Makefile.inc1
> > > > >
> > > > > Modified: head/Makefile.inc1
> > > > > ==============================================================================
> > > > > --- head/Makefile.inc1        Sat May  9 01:48:08 2020        (r360832)
> > > > > +++ head/Makefile.inc1        Sat May  9 02:01:29 2020        (r360833)
> > > > > @@ -1403,6 +1403,16 @@ distributeworld installworld stageworld: _installcheck
> > > > >       ${DESTDIR}/${DISTDIR}/${dist}.debug.meta
> > > > >  .endfor
> > > > >  .endif
> > > > > +.elif make(installworld) && ${MK_CAROOT} != "no"
> > > > > +     # We could make certctl a bootstrap tool, but it requires OpenSSL and
> > > > > +     # friends, which we likely don't want.  We'll rehash on a best-effort
> > > > > +     # basis, otherwise we'll just mention that we're not doing it to raise
> > > > > +     # awareness.
> > > > > +     @if which certctl>/dev/null; then \
> > > > > +             certctl rehash \
> > > >
> > > > Does this update METALOG with the added links?
> > > >
> > > > It seems a little weird to rely on DESTDIR from the environment.
> > > >
> > > > In general I'm not enthusiastic about additions to installworld that do
> > > > anything other than copying files, creating links, etc in simple ways.
> > >
> > > I will happily back this out if I can get some qualified eyes to
> > > review/improve it. It does not update METALOG, and it probably should.
> > > Agreed on DESTDIR. As for point #3, I guess we can continue spreading
> > > `certctl rehash` all over the tree in various points that may need it;
> > > the release(7) scripts will need to be done if we don't do it here at
> > > a minimum, and I haven't put much thought into it beyond that.
> >
> > I'm not in a rush to back this out given that it's solving a real
> > problem, but lets talk about improvements.
> >
> > I kind of feel like this belongs in distribution (which I think would
> > deal with release scripts) or in etcupdate/mergemaster, but I'm not
> > sure either of those are correct.  I'd be happy to review changes to
> > update the METALOG (I guess we'd extend certctl with an option to do
> > that?)  I think that's the most important things because we really
> > should be routinely validating that DESTDIR only contains things in the
> > METALOG.  A quick and dirty fix for the DESTDIR weirdness might be adding
> > "env DESTDIR=${DESTDIR}" so it's explicit.
> >
> 
> Got it- not an egregious enough violation to promptly back out, but we
> will work towards a better solution.
> 
> For the larger picture, distribution is probably the correct spot for
> this; we currently rehash at the following points:
> 
> - mergemaster/etcupdate
> - freebsd-update
> - post-install of the caroot pkg (pkgbase)
> 
> This leaves two primary final blind spots, which this commit was
> attempting to resolve:
> 
> 1.) Install media (including VM images)
> 2.) Initial installs (e.g. from bsdinstall)
> 
> #1 could be solved by directly adding it to the release(7) scripts and
> #2 could be solved by having bsdinstall do it at config time, but
> there seem to be multiple potential points that could instead hit both
> (with a single stone) while also making covering the array of other
> images that re@ produces (e.g. AWS images) and perhaps other points
> that we've not immediately considered.
> 
> To this end, distribution or installworld are probably equally
> sufficient, but I'm incredibly unfamiliar with the former. My
> understanding from glancing at it over the years is that it may be
> executed into a tempdir and merged with mergemaster/etcupdate, or
> directly into the new root if we're looking at a new install that
> doesn't have potential local changes to merge in.

My understanding of distribution is that it's the things that don't make
sense to update automatically (e.g. config files that have to be merged.
For #1 above distribution clearly solves the problem.  For #2 I kind of
think this is the installers job.

> Part of the problem that I see in my head is probably solved by just
> having a var that mergemaster/etcupdate can pass in to distribution to
> not bother with the rehash. We want to keep the mergemaster/etcupdate
> rehash because if they're running then the target's certainly not been
> rehashed. OTOH, we probably don't want to rehash multiple times in
> that pipeline- from a certctl standpoint, it's completely harmless,
> but it's not the most effective use of system resources and probably
> looks a bit bad.

In some ways I wonder if we actually want a three phase split:
installworld (programs, fixed data, etc), distribution (config bits),
and regen (generate files based on the -- possibly altered -- contents
of config files, etc.  If we could split out the regen into (e.g.) an
installed shell script or set of scripts then we could call that from
the installer, etc.  That might provide a generic solution.

So far this isn't a well thought out idea, but maybe there's something
here?  Presumably we'd need to bolt a compatibility shim onto
distribution to start out given how much blowback I got for breaking
some nonsensical orderings of the top-level targets when adding METALOG
support.

-- Brooks
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/svn-src-head/attachments/20200514/d0799a4d/attachment.sig>