Re: Question regarding crunchgen(1) binaries

--------
Warner Losh writes:

> Maybe start there to understand what "LTO" the security thing is doing and
> why it's either wrong or violates an assumption in crunchgen that can be
> fixed.

Crunch binaries were invented 30 years ago, to make FreeBSD 
installation program fit on a single floppy disk.

Note that the goal was saving disk-space rather than RAM.

The "architecture" of crunchgen is to take a lot of programs, rename
their main() and link them all together with a new main() which
dispatches to the right program's main() based on argv[0]

Statistically you save half a disk-allocation unit for each program
which was nothing to sneeze at, but the real disk-space dividend
comes from linking the resulting combi-program static.

Because it is linked static, only those .o files which are referenced
gets pulled in from the libraries, libm::j0.o only gets pulled in
if you Bessel functions, which, countrary to rumours, sysinstall
did not.

(The goal of shared libraries is saving RAM:  Everybody gets the
complete library, but only one copy of it's code ever gets loaded.)

But the real trick is actually not crunchgen, which was originally just
a shell script, but rather crunchide(1).

Crunchide(1) does unnatural acts to an objectfile's symboltabel,
to get around the fact that all the programs have a function called
"main" and that they litter the global symbol namespace with their
private inter-file references.

To make a crunched binary, the .o files for the individual programs
are first "pre-linked" without libraries so that internal interfile
references are resolved.

Then crunchide changes all global symbols, except "main" to be local
symbols, so that they become unavailable for symbol resolution in
the final run of the linker.  The "main" symbol is also renamed
to a per-program name, something like "cp_main" for cp(1) etc.

And then all the prelinked .o files, one per program, gets linked
together with the "dispatch main" and this time with libraries.

I see no reason why crunchgen cannot be done with Link Time
Optimization, but somebody has to write the new crunchide(1), and
I suspect it will have a tougher row to hoe, because pre-linking
cannot be used to take care of the inter-program symbols.

As I understand it LTO can also link with "normal libraries"
so one option might be to only LTO the final linking step of
the crunch process, treating all the programs as "normal libraries",
but still getting LTO advantage internally in the libraries.

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.