FYI: WITH_REPRODUCIBLE_BUILD= problem for some files?

Mark Millard marklmi at
Tue May 4 07:10:28 UTC 2021

On 2021-May-3, at 21:27, Mark Millard <marklmi at> wrote:

> On 2021-May-3, at 19:26, Mark Millard <marklmi at> wrote:
>> On 2021-May-3, at 10:51, Mark Millard <marklmi at> wrote:
>>> On 2021-May-3, at 07:47, Ed Maste <emaste at> wrote:
>>>> On Thu, 29 Apr 2021 at 02:50, Mark Millard via freebsd-current
>>>> <freebsd-current at> wrote:
>>>>> Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/sbin/ping and /usr/obj/DESTDIRs/13_0R-CA7-poud/sbin/ping differ
>>>>> Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/sbin/ping6 and /usr/obj/DESTDIRs/13_0R-CA7-poud/sbin/ping6 differ
>>>>> Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/usr/bin/ntpq and /usr/obj/DESTDIRs/13_0R-CA7-poud/usr/bin/ntpq differ...
>>>> This is unexpected. Unfortunately I haven't looked at reproducibility
>>>> in a while, and my work was all on x86. This could be a regression or
>>>> a longstanding issue with arm64.
>>>> If you install the diffoscope package (py37-diffoscope) and run it on
>>>> the two directories / files it should give a more convenient view of
>>>> the differences. (Or, if you can make a tarball of the differing files
>>>> I can take a look.)
>>> I no longer have the same content in those directory
>>> trees: newer rebuild and the same buildworld used to
>>> installworld to both places, instead of 2 different
>>> buildworld's. I'm also unsure how reproducible getting
>>> differences was.
>>> I can eventually do experiments to test multiple separate
>>> buildworld's and installworld's, but the machine is busy
>>> building ports and the llvm builds involved means it
>>> will be some time before I'd switch activities. And the
>>> buildworld's involve llvm builds as well and take notable
>>> time themselves. So my next comparison will not be any
>>> time soon.
>>> I'll let you know if I manage to generate another example,
>>> this time being sure to keep the data. If I try multiple
>>> times without finding any differences, I'll eventually
>>> decide "enough is enough" and let you know.
>> I've still got a long ways to go to do the first
>> actual comparison of builds.
>> But I'll note that I've built and stalled py37-diffoscope
>> (new to me). A basic quick test showed that it reports:
>> W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" module is unavailable.
>> As I'm not familiar with the tool, you might need to send
>> notes about how you want me to use the tool to get the
>> output that you would want. (And, so, I get to learn . . .)
> I've tried another experiment (* in the path matches "28" and "30"):
> # diffoscope /.zfs/snapshot/2021-04-*-01:40:48-0/bin/sh
> $<3/>2021-05-03 21:08:48 W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" module is unavailable.
> $<3/>Traceback (most recent call last):
>  File "/usr/local/lib/python3.7/site-packages/diffoscope/", line 745, in main
>    sys.exit(run_diffoscope(parsed_args))
>  File "/usr/local/lib/python3.7/site-packages/diffoscope/", line 677, in run_diffoscope
>    difference = load_diff_from_path(path1)
>  File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/", line 31, in load_diff_from_path
>    return load_diff(codecs.getreader("utf-8")(fp), path)
>  File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/", line 35, in load_diff
>    return JSONReaderV1().load(fp, path)
>  File "/usr/local/lib/python3.7/site-packages/diffoscope/readers/", line 33, in load
>    raw = json.load(fp)
>  File "/usr/local/lib/python3.7/json/", line 293, in load
>    return loads(,
>  File "/usr/local/lib/python3.7/", line 504, in read
>    newchars, decodedbytes = self.decode(data, self.errors)
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 18: invalid start byte
> The two older snapshots of a Boot Environment have
> bin/sh files that compare equal. But every program I
> tried the above sort of thing against on got the same
> UnicodeDecodeError result from diffoscope, byte value
> and position matching.
> These snapshots have more than an installworld in them
> and so are messy to compare overall. But the
> installworld (and installkernel) content show similar
> differences to what I reported before as far as
> example files with differences go. But this is aarch64,
> not armv7.
> It will still be notable time before I have simple
> installworld tree's to compare.

While waiting for the 2nd buildworld to complete, I used
the 1st buildworld's materials to installworld twice (to
different, empty directory trees) and then did a diff.
The diff reported:

# diff -rq /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/ /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm2/ | more
diff: /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/usr/tests/local: No such file or directory
diff: /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/usr/tests/sys/pjdfstest/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests: Too many levels of symbolic links


# ls -Tld /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/usr/tests/local
lrwxr-xr-x  1 root  wheel  14 May  3 23:18:22 2021 /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/usr/tests/local -> ../local/tests

So this looks like the expected result and suggests that
the problem(s) are at the buildworld stage. (No surprise.)

I should be sleeping by the time the 2nd buildworld for
cortex-a72 (aarch64) is done. (That is what I decided
to test initially.) So it will be more hours until I have
basic comparison results from 2 distinct buildworlds.

Mark Millard
marklmi at
( went
away in early 2018-Mar)

More information about the freebsd-current mailing list