[RFC] SA/EN ABI/Lib flagging for package rebuilding

[John Baldwin]

On Thursday, April 09, 2015 12:37:36 PM Bryan Drewery wrote:
> Currently for any SA/EN we force rebuild all packages for
> pkg.freebsd.org. This is a very time-consuming process at 24 hours for
> each set and 10 (non-head) sets to build.
> 
> We really only need to rebuild packages if ABI/KBI is changed (which I
> would think would happen almost never) or if a library is updated. The
> library being updated needs a rebuild in case of packages linking
> against the library statically. There's no quick way to determine what
> package may be infected by a static linkage to a library so we just
> rebuild them all.
> 
> There are 2 somewhat conflicting goals here. We need head packages to be
> usable on all ABI/KBI changes. The simplest here was decided to just
> rebuilding head packages always. This is a compromise I am willing to
> accept. I had wanted to use __FreeBSD_version for this in the past but
> it tends to be missed or bumped gratuitously.
> 
> 
> My idea to cover both cases so that packages will only rebuild if the
> public API or libraries change in the jail or anything else which
> affects the resulting binaries.
> 
> I will take a checksum of the following and if anything differs then do
> a rebuild:
> 
> /lib /usr/lib /libexec (.so included due to symbol versioning and
> detecting added/removed libraries which can change packages)
> /usr/include (all public API for the system including kernel API for
> modules)
> /usr/libdata
> 
> That is all I am aware of for the public API/KPI/ABI/KBI that is
> relevant for packages.
> 
> I also realize I need to include some of the binutils files as they may
> change how binaries are produced. This is all of 'ar as ld nm objcopy
> objdump ranlib readelf strip' in /usr/bin/. Also all of GCC/Clang
> binaries in /usr/bin: 'cc clang clang++ clang-cpp clang-tblgen cpp
> c++filt CC c++ g++ gcc gnu-ar gnu-ranlib'.
> 
> I don't want to consider all of /bin /usr/bin /sbin and /usr/sbin as it
> catches SA that do not really need a rebuild, such as
> FreeBSD-SA-15:07.ntp which modified /usr/sbin/ntpd.
> 
> By having a whitelist of files from *bin* I risk not rebuilding in a
> rare case where /bin/sh or /bin/cp (random example) had a vulnerability
> fixed in them that could change resulting binaries. I consider that to
> be very unlikely though. Perhaps for head packages it is more relevant
> but I still consider it unlikely other changes will require rebuilding
> packages.
> 
> If I forgot other critical pieces from *bin* please list them.
> 
> An alternative would be for there to be a reliable flag in head and
> SA/EN noting whether binaries need to be rebuilt. There had been a
> technical limitation to bumping __FreeBSD_version for SA as it would
> modify all binaries. I think that was fixed. Portmgr fought hard to
> always have __FreeBSD_version bumped but that's a lost cause. As I
> mentioned I don't trust this flag in head either.

I also want __FreeBSD_version always bumped for SA/EN, but secteam@ actively
refuses to do this, even after being explicitly told to do so by core at . :(

Regardless, I think your approach is more robust than hoping developers
remember to bump the version for any API/ABI change.  Looking at your
proposed list I can only think of a few other binaries that matter, in
particular code-generators like lex, yacc, and rpcgen.  I don't claim that
that list is exhaustive, but I do think it is a class of binaries that were
not in your list.

-- 
John Baldwin