Re: git: fb5f03a87cf4 - main - Mk/bsd.lto.mk: add global LTO support for ports

On 21-10-05 22:04:32, Matthias Andree wrote:
> 
> Am 05.10.21 um 18:49 schrieb Piotr Kubaj:
> > On 21-10-05 18:31:52, Mathieu Arnold wrote:
> >> On Tue, Oct 05, 2021 at 06:27:15PM +0200, Piotr Kubaj wrote:
> >>> On 21-10-04 15:30:56, Mathieu Arnold wrote:
> >>>> On Thu, Sep 30, 2021 at 06:34:20PM +0000, Piotr Kubaj wrote:
> >>>>> The branch main has been updated by pkubaj:
> >>>>>
> >>>>> URL: https://cgit.FreeBSD.org/ports/commit/?id=fb5f03a87cf432751fae1f0ae7f29c9d4fc65917
> >>>>>
> >>>>> commit fb5f03a87cf432751fae1f0ae7f29c9d4fc65917
> >>>>> Author:     Piotr Kubaj <pkubaj@FreeBSD.org>
> >>>>> AuthorDate: 2021-09-30 18:27:50 +0000
> >>>>> Commit:     Piotr Kubaj <pkubaj@FreeBSD.org>
> >>>>> CommitDate: 2021-09-30 18:27:50 +0000
> >>>>>
> >>>>>     Mk/bsd.lto.mk: add global LTO support for ports
> >>>>>     
> >>>>>     It's well known that LTO provides both performance and size benefits for
> >>>>>     binaries.
> >>>>>     
> >>>>>     Add preliminary, opt-in support for global LTO enforcement to ports. Ports that
> >>>>>     provide LTO option on their own and the ones that don't work with LTO will need
> >>>>>     to set LTO_UNSAFE in the future.
> >>>>>     
> >>>>>     PR:     258536
> >>>>
> >>>> Not to be picky about approval and all, but this was added to the
> >>>> framework, and the framework is maintained by portmgr.  When you want to
> >>>> add something to it, you must consult with portmgr before anything gets
> >>>> committed.
> >>>>
> >>>> In that case, we would have told you not to do it this way, but to make
> >>>> this a Mk/Uses/lto.mk.
> >>>>
> >>>> So please, turn this into a USES=lto.
> >>>
> >>> I did consult, but no one replied.
> >>
> >> There is absolutely no maintainer timeout for the framework, you cannot
> >> just add code there without explicit approval.
> > 
> > And this is a port of a bigger issue, where portmgr ignores emails until numerously asked for (if one is lucky).
> > As one of users wrote in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=251117, for which portmgr is assigned, "portmgr@ more and more feels to me like a black hole, or /dev/null: Anything sent there seems to disappear without effect."
> > 
> > Since it was a change that doesn't change anything out-of-the-box, I decided to commit it.
> > 
> >>
> >>> IMO adding it to USES is not a good idea, since USES are supposed to be used per port and my idea was to force LTO for all ports, same way that SSP already does.
> >>
> >> All I see in the patch is a USE_LTO knob, and a LTO_UNSAFE one, without
> >> any documentation of what it is for, what it does, what it might do,
> >> what it is about, or anything else.
> > Neither has SSP, I don't see any documentation for it (including commiter handbook which has just one line regarding USES=kmod at https://docs.freebsd.org/en/books/porters-handbook/book/#uses-kmod).
> 
> Piotr,
> 
> While I sympathize with your findings about portmgr@ 'responsiveness'
> having had my shares of ignores and brushes, I would tend to agree that
> we should not add undocumented knobs anywhere, so:
> 
> 1. please add documentation including motivation
OK, just take note that there are several issues with it right now, so it's definitely not ready for use. E.g. libffi, perl and pkgconf don't build.

Other than that, what is the best place for documentation? Do you mean porter's handbook?


> 
> 2. I wonder about a deployment plan.  LTO linking incurs several issues,
> such as memory use, disk-space use with fat objects, and concurrency
> issues that cause a massive prolongation of linking, which I've often
> seen to run a single thread.  I think these items need consideration.
That's what we have package builders for. AFAIK they run Poudriere without ALLOW_MAKE_JOBS, so mostly single-threaded.
If LTO is ever enabled for everyone, we should still keep a knob to disable it globally.

But I agree that building everything with LTO may require memory upgrades for builders. The extreme example is mongodb (has an LTO option enabled by default) which allocates close to 30GB RAM during linking.

> 
> Other than that, I appreciate the initiative to support LTO inside the
> framework, to avoid each and every port implementing it by itself, which
> will cause a maintenance mess down the road.
Thanks.

> 
> Regards,
> Matthias