OpenZFS branch tracking policy

Fri Apr 9 22:15:36 UTC 2021

On Fri, Apr 2, 2021 at 6:44 PM Martin Matuska <mm at freebsd.org> wrote:

> I have prepared an example merged branch here:
> https://github.com/mmatuska/freebsd-src/tree/openzfs_master_merged
>
> The magical command was:
> git merge -s subtree -Xsubtree="sys/contrib/openzfs" 891568c99
> --allow-unrelated-histories
>
> Luckily, our current diff is manageable.
>

So I did this for bzip2 using approximately:

git add remove bzip2 <url>
git fetch bzip2
git merge -s subtree -Xsubtree=contrib/bzip2 bzip2/master
--allow-unrelated-histories --squash

[1] At this point I resolved conflicts, where were the entire files since I
guess I didn't bootstrap right to the last merge. There were 4 files in
conflict.

Then I did a git add of all the files in conflict and a git commit.

This produced a good commit. since it was a squash commit, there were no
issues.

However, it turns out I botched the commit at point [1] above. So I ran
this again and got a conflict for the whole file that I'd removed a blank
line from.

So, this looks like it could be workable, but does lead me to a few
questions:

(1) How do we do this so that the conflicts aren't add/add conflicts? Is
there some way to bootstrap this?
(2) Do we need to keep track of the last merge point and use that in
merging the next one in?
(3) I assume we keep track of FreeBSD diffs in a branch off <url> and we
merge that instead of master.
(4) What do we do about adjustments to the build that are needed?
(5) Do we need to host a FreeBSD-specific repo with this stuff, maybe with
tags we don't want widely pushed to ease the next merge? Eg, make this the
first case of a 'vendor repo' that we then pull squash commits from so that
the vendor repo can track upstream, but not otherwise be pushed to all our
users....

Finally, how did you deal with [1] producing so many full-file add/add
conflicts? Oh, and what kind of commit message when things merge do you
suggest? I rather like your 'bring in hash XXXX branch blah, here's the
important highlights' emails and think that would be a good first cut at
advice on what to put in these.

This suggests the current answer is 'seems doable, but we need to document
it and come up with recommendations for how to do it'.

Warner

On 3. 4. 2021 1:37, Martin Matuska wrote:
> > Hi Warner and Ed,
> >
> > 2.1-release has already been branched. The stable branch policy in
> > OpenZFS is somewhat strange, they make a staging branch for each
> > patchlevel release, but the commits are continuous.
> >
> > To have some idea how big the repo history is:
> >
> > $ git rev-list master --count
> > 6662
> >
> > $ git rev-list zfs-2.1-release --count
> > 6650
> >
> > master and zfs-2.1-release have 6650 common commits at the  moment
> >
> > $ git log master | wc -l
> > 129868
> >
> > (linecount - 4 * revcount) / revcount = linecount / revcount - 4 =
> > 15,4938 comment lines per commit on average
> >
> > Initial commit was made in Feb 26, 2008.
> >
> > Yearly commit counts:
> >
> > $ git log master | grep -c -E '^Date:.* 2020 -[0-9]+$'
> > 666
> >
> > $ git log master | grep -c -E '^Date:.* 2019 -[0-9]+$'
> > 535
> >
> > $git log master | grep -c -E '^Date:.* 2018 -[0-9]+$'
> > 428
> >
> > Martin
> >
> > On 2. 4. 2021 20:15, Warner Losh wrote:
> >>
> >>
> >> On Fri, Apr 2, 2021 at 11:56 AM Ed Maste <emaste at freebsd.org
> >> <mailto:emaste at freebsd.org>> wrote:
> >>
> >>     On Fri, 2 Apr 2021 at 11:50, Warner Losh <imp at bsdimp.com
> >>     <mailto:imp at bsdimp.com>> wrote:
> >>     >
> >>     > We'd always hoped that we'd be able to do subtree merges from
> >>     upstreams
> >>     > that use git into FreeBSD. The big worry, though, was that this
> >>     would
> >>     > needless bloat the repo with a lot of history. We don't want,
> >>     for example,
> >>     > all of LLVM's history in the tree. We'd always anticipated that
> >>     there'd be
> >>     > some things we'd just accept the history for, since it is
> >> similar in
> >>     > character to the vendor branches (though of course a bit more).
> >>
> >>     Note that if we do want to avoid bringing in the full history `git
> >>     subtree merge` supports a `--squash` option. This brings in the
> >> set of
> >>     upstream changes as a single commit, without bringing along the
> >>     associated history. We will need to do more experimentation to
> >> confirm
> >>     that the full process, including bootstrapping, will work as we
> >> want.
> >>     Assuming this all works it should allow us to forgo the use of a
> >>     FreeBSD-specific vendor branch in src.
> >>
> >>     We've discussed mirroring any such 3rd-party source in some
> >>     FreeBSD-controlled repository. This would allow the project to
> >> retain
> >>     a full copy of the history, but avoid bloating src with it.
> >>
> >>     I agree with Warner that we may want a different policy (full
> >> history
> >>     or snapshots) for different contrib sources.
> >>
> >>
> >> Good points Ed. I'd forgotten about --squash.
> >>
> >> Martin, what's your timeline for wanting to implement these things?
> >> I'm unfamiliar with the OpenZFS schedules.
> >>
> >> Warner
> > _______________________________________________
> > freebsd-git at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-git
> > To unsubscribe, send any mail to "freebsd-git-unsubscribe at freebsd.org"
>