Some reproducible builds notes

[Maxim Sobolev]

Hi Ed, I have not got time to look at diffoscope myself yet, but it's
definitely in my short-term TODO list. Quick question for you though. Would
it work given two tar.bz2 packages or does it need two directories? Just in
case it does it as well (although I do not hope for that much), our build
system also produces corresponding mtree files so it would be nice if it
could also parse those and only extract files that have different checksum.
Preferably we would like to diff mtree first, and then run extraction once
as extracting files one by one from GB-size .tar.bz2 is painful CPU-wise.
We have code to do just that (i.e. inspect mtree and extract files that
differ, so if you don't, we can probably contribute that.

P.S. For anyone interested in Ed's work, here is his BSDCan talk here live:
https://youtu.be/z7pDnBO5wSM?t=5h37m6s
-Max

On Jun 16, 2016 7:11 AM, "Ed Maste" <emaste at freebsd.org> wrote:

> I recently presented on "Reproducible Builds in FreeBSD" at BSDCan.
> For anyone unfamiliar with the topic, from
> https://reproducible-builds.org/ "Reproducible builds are a set of
> software development practices which create a verifiable path from
> human readable source code to the binary code used by computers." In
> brief, the idea is that building the same binary, software package,
> document or other binary artifact twice from the same source produces
> identical output. There's good background information, documentation
> on making builds reproducible, and links to test results on the
> reproducible-builds.org site.
>
> Many folks have contributed to the reproducible build effort in
> FreeBSD src and ports over time -- at least a decade. There are many
> practical benefits of reproducible builds (such as bandwidth and
> storage savings). However, there's been a growing interest over the
> last few years in the broad open source and free software community in
> the topic, coming primarily from a software and toolchain integrity
> perspective. Over the last few years some Debian folks have been
> leading a comprehensive and structured reproducible builds effort.
> bapt@ and I attended the first Reproducible Builds Summit in Athens
> last year, and I had a talk accepted at BSDCan on it. The BSDCan
> schedule page for my talk[1] has a link to the slides[2].
>
> I'd like to continue discussing reproducible builds in the FreeBSD
> context, but for now just want to capture some data from my talk so
> that it's available for interested maintainers of individual ports
> who'd like to take a look. I used src r300165 and ports r415464, with
> a few patches as described in the talk.
>
> I've put data from the ports build runs for my talk at [3]. In that
> directory nonrepro.1.txt contains the set of packages that built
> nonreproducibly (with a patch set the timestamps in pkg's output).
> nonrepro.4.txt contains the set of packages that built nonreproducibly
> with the patch above, SOURCE_DATE_EPOCH set in the build environment,
> a Clang patch[4] to honour SOURCE_DATE_EPOCH, and a change to make GNU
> ar default to deterministic archives, since committed as ports
> r416639.
>
> Diffoscope[5] is a tool that attempts to show the differences between
> two binary artifacts in a concise and human-readable form. It's
> available in ports as sysutils/py-diffoscope and in the
> py34-diffoscope package. You can also try it out online[6]. In the
> diffoscope/ subdirectory[7] I've put the output for most of the
> nonreproducible packages. (Some packages[8] are excluded because of
> excessive diffoscope runtime.)
>
>
> [1] http://www.bsdcan.org/2016/schedule/events/714.en.html
> [2]
> http://www.bsdcan.org/2016/schedule/attachments/375_2016-06-11-BSDCan-2016-Reproducible-Builds.pdf
> [3] https://people.freebsd.org/~emaste/reproducible-builds/iteration-1/
> [4] http://reviews.llvm.org/D20791
> [5] https://diffoscope.org/
> [6] https://try.diffoscope.org/
> [7]
> https://people.freebsd.org/~emaste/reproducible-builds/iteration-1/diffoscope/
> [8]
> https://people.freebsd.org/~emaste/reproducible-builds/iteration-1/excessive-diffoscope-runtime.txt
> _______________________________________________
> freebsd-ports at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ports
> To unsubscribe, send any mail to "freebsd-ports-unsubscribe at freebsd.org"
>
>