Moving to synth (was: Removing documentation)

Tue Feb 9 07:18:02 UTC 2016

On 9 February 2016 at 11:58, Michel Talon <talon at lpthe.jussieu.fr> wrote:

> Greg 'groggy' Lehey wrote:
>
> > So how would things improve in this respect if we change to synth?
> > There we need a maintainer who understands Ada.  OK, at the moment
> > that's you.  But what happens when you relinquish maintainership for
> > whatever reason?
>
> > My feeling on the matter is that there's space for more than one tool,
> > as Mathias suggested.  But I think it would help synth to have
> > user-oriented documentation similar to that for portmaster or
> > portupgrade ("to achieve this, do this...").  I could see this as a
> > good addition to the handbook (4.5.3.3).  A(n objective) discussion of
> > the pros and cons of the three alternatives would also be useful.
>
> Having been in the situation of doing such a discussion some 10 years ago,
> when portupgrade and portmaster were the fashionable tools, and pkgng
> did not exist, i was very interested to discover this tool synth that i
> had never heard of.
>
> Of course unable to find it in the ports i finally found it was a
> DragonFly port very recently
> ported to FreeBSD, and then that it was written in Ada. Next task find the
> Ada compiler.
> Needless to say, no port named ada under lang, finally found it was
> gcc5-aux. Downloaded
> the packages for gcc5-aux and ncurses the port for synth from SVN
> repository, and began compiling.
> Somewhere the build broke because of unsupported
> pragma Suppress (Tampering_Check);
> but after removing this line i had finally a brand new synth.
> Frankly one first prerequisite for having this discussion is that the
> *package* synth is ready to be downloaded
> and people don't have to go through this ordeal.
>
> Immediately i wanted to test it on a port which is not too much connected
> to other stuff
> so i did after synth configure
> synth status
> and
> synth build lang/chicken.
> The first one bombed since there was a problem with the options on some
> port. I solved it as required and
> synth status gave some answer.
> Then the second one decided it needed to recompile 6 ports and started
> doing it. The nice curses
> screen appeared, which for sure is beautiful but doesn't say much about
> what is going on.
> Fortunately i discovered beautiful logs under /var/log/synth, and
> apparently the 6 builds went OK.
> Then it asked me to scan the ports tree which took ages, finally to
> discover that my builds failed
> "option check" and do nothing.
> This prompted me to try understanding what the crap synth was doing, and i
> finally found that the
> "repository" was the place under /usr/obj where synth moved all the
> packages it had compiled, that
> there was a ton of mounts under /usr/obj reproducing a clean room freebsd
> system to compile the port.
> That the long operation above consisted in running make -V in a lot of
> ports to discover the value
> of certain variables which is usually done to compute the INDEX in
> /usr/ports, and that, due to the
> above failure the corresponding packages had been removed from the
> repository.  Part of this information
> i obtained by looking at the code which is surprisingly readable for
> someone who doesn't know ada.
> The exact
>
> This little story leads me to a second prerequisite, produce a more
> complete documentation describing exactly
> what synth does, how to solve buggy situations etc. The argument "the soft
> can be directly used by newbies
> without documentation" i don't buy.
>
> Finally lets us target the center. I like very much the idea of using
> these mounts to compile the packages.
> It is fast and simpler than jails. I also like the idea of doing several
> things simultaneously. Is it really necessary
> to use thousands and thousands of lines of code to do that? Concerning the
> objection that portmaster
> has no dependency while synth has dependencies i think this objection has
> no merit since it is a binary
> which will run without problem, its data structures being in memory. This
> is in great contrast with portupgrade
> where if you upgrade ruby when portupgrade runs, you are in a mess (or if
> you upgrade the db that
> portupgrade uses). I have not seen if synth checks for circular
> dependencies, portupgrade certainly
> did it.
>
> In summary, synth seems to be a very nice work. As an exercice in shell
> writing, portmaster is
> certainly a master's work, but it is a poor substitute to a real
> management tool. Portupgrade
> was more ambitious, but had too many problems and was incredibly slow. So
> i think J. Marino
> is right, better bite the bullet as soon as possible, polish whatever
> needs polishing, update the documentation
> and go ahead.  The fact that synth is written in a relatively obscure
> language can be a deterrent,
> but in fact it is very readable by non ada practitioners.
>
>
>
>
>
>
>
>
> --
> Michel Talon
>
> _______________________________________________
> freebsd-ports at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ports
> To unsubscribe, send any mail to "freebsd-ports-unsubscribe at freebsd.org"
>

Thanks for sharing Michel.  Like many I was agast at portmaster being
marked as  deprecated and seriously thought that its time to pull the
FreeBSD plug.  I apprecate that John was asked, and did remove the
deprecation tag, and at roughly the same time introduced the synth tool
into port-mgmt.  The concern around maintenance is acknowledged
unfortunately there seems to be a dirth of expertise in this area. So...

This morning I created a new jail and built synth from scratch using
port-mgmt/synth.

Using synth in the jail, was initially unsuccessful due to it complaining
that it couldn't mount tmpfs.  So I moved synth and devel/ncurses into the
host system (which really irked me).  Our environment is a little
complicated as we build and test packages for i386 and amd64 on the
physical box using multiple jails for building and preliminary testing; so
touching the locked down, highly protected base system wasn't going to be a
viable solution.

The short story - jails aren't required for synth (maybe good/bad).
However we did get it working in a "build" jail by adding to its jail.conf
enforce_statfs=0; allow.mount; allow.mount.nullfs; allow.mount.tmpfs;
allow.mount.devfs;
using synth's default settings, synth created/used an additional 127 mount
points, to the usual 5 of our normal environment.  (What tha?!)

We're really trying to just drop synth into the portmaster role, as easily
as possible, so the first test is to arbitrarily picked the first 32 ports
from our build-list which are cat'ed to portmaster.

A few failures, such as:
 ftp/curl failed due to
"configure: using CFLAGS: -I/usr/include -O2 -pipe  -isystem
/usr/local/include -fno-strict-aliasing
configure: CFLAGS note: CFLAGS should only be used to specify C compiler
flags, not include directories. Use CPPFLAGS for: -I/usr/include"
which wasn't entirely unexpected as synth's jail may be lacking appropriate
configuration, regardless 73/76 ports were successfully built.  Compare to
portmaster coming to a halt.

Unfortunately getting synth to play nicely with our core2-make.conf (core2
is the name of the profile that we created in
/usr/local/etc/synth/synth.ini) proved a challenge as other necessary files
weren't able to be included.  We'll need to look into why...

Synth is almost complete.  It has been released "early" so folks have
another option available to them.  But what does it do really well?
a) easy to get going (too easy, because of the tight integration with pkg,
pkg was built during a "test" run.  RTFM!)
b) documentation is clear (see "man synth", or "synth help")
c) *significantly* reduced disk io. During our normal build with ccache
"gstat -aI1s" is usually around 70% busy, it was closer to 3% during most
of the build.
d) because 14% of our ports use MAKE_JOBS_UNSAFE having multiple builders
makes much better utilisation of the box.  Apart from that the hardward is
better utilised and synth experienced almost no involuntary context
switching during the build process.
e) The informational ncurses interface was excellent, but being batch
oriented we turned it off for the final build.  Synth provided ongoing
status information such as:
00:07:21 => [03]          Shutting down
00:07:27 => [05] 00:00:12 Success devel/jsoncpp
00:07:27 => [05]          Shutting down
00:08:06 => [02] 00:00:58 Success net/ntp
00:08:06 => [02]          Shutting down
00:08:08 => [01] 00:02:50 Success devel/binutils
00:08:11 => [04] 00:03:24 Success security/openssl
00:08:11 => [04]          Shutting down
00:09:39 => [06] 00:06:00 Success security/heimdal
00:09:39 => [06]          Shutting down
00:21:21 => [01] 00:13:12 Success lang/gcc
00:21:21 => [01]          Shutting down

Synth does pretty much what is needed.  Inputs are taken from
- /usr/local/etc/synth/synth.ini (which can be edited via "synth
configure", or a text editor.  Its largely self-explanatory.)
- /usr/local/etc/synth/${profile_name}-make.conf
Outputs placed in:-
- /var/synth/live_packages/All  - built packages
- /var/log/synth  - log files, including all package lots and standard
reports that are numbered for easy access 00_last_results.log
01_success_list.log 04_skipped_list.log

Certainly worth consideration.  For our part, I'm looking forward to
ironing out our issues and adding this to our build system.

Thanks John M.
Regards, Dewayne
PS I'm sure I could rewrite synth in APL or PROLOG if that would help, but
neither have demonstrated large scale application suitability like Ada ;)