From nobody Wed May 19 19:04:57 2021 X-Original-To: freebsd-git@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 9984F624A9A for ; Wed, 19 May 2021 19:05:11 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Flj461VrBz3qNH; Wed, 19 May 2021 19:05:09 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by mail-wm1-f51.google.com with SMTP id u133so7839320wmg.1; Wed, 19 May 2021 12:05:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=VcVilJajRw8sX2OLpKHpfbJnJ+ag1cTNZyN4U/mSyP4=; b=h93wn2cnePUSYt1uvLMa5dOf3dvEwspitpksACy+tHQWgOn+WK/NTaiJqLijG5WTJA hjmVHDdZuLOHdPuF0OrcvbS2r8/UXliZ/kIb77DWsqqVcI72kRBTq/BB9X+hgMmlKQGW kRA3DQCGed4ulfpSv/CTxnDQ2iXSt3xpPGfpP/MVvwRm8CEVMDHtRtJ6h/7Z5GWQgp2c lrzC0VIza/1OTuz5FxnR4/4ISlBxDeapl9ZrFyQOpe6qHJ3eVewIn2Au74GUaBdzsPux mO+w4YIUrMWQuIBqITF/LnVG5crBtoXZQblidU7TDoBhcqsOLeRTA5C3LXDgjfL1hvf5 RB/Q== X-Gm-Message-State: AOAM531lhwg8A/TP6KcUZLuiR+yahSZQ3iBGkuZITOWfBN3mccF+BLfk CuHX8egLRpNmWUrInQY2O/GfRSozS5zMDJvs5B3z43JA X-Google-Smtp-Source: ABdhPJxyd3Z5NhhCyyRAIJFaAY1eGeFwLIToBp4yifkFqsHVdlQawq+sA+pNDfFo/ZpyyYh51eI0HBGRPLq0cgr57nY= X-Received: by 2002:a1c:98c6:: with SMTP id a189mr655660wme.178.1621451108260; Wed, 19 May 2021 12:05:08 -0700 (PDT) List-Id: Discussion of git use in the FreeBSD project List-Archive: https://lists.freebsd.org/archives/freebsd-git List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-git@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?Q?Ulrich_Sp=C3=B6rlein?= Date: Wed, 19 May 2021 21:04:57 +0200 Message-ID: Subject: Re: vendor/illumos merges To: Mark Johnston Cc: freebsd-git@freebsd.org Content-Type: multipart/alternative; boundary="00000000000054214305c2b38120" X-Rspamd-Queue-Id: 4Flj461VrBz3qNH X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of uspoerlein@gmail.com designates 209.85.128.51 as permitted sender) smtp.mailfrom=uspoerlein@gmail.com X-Spamd-Result: default: False [-1.47 / 15.00]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; R_MIXED_CHARSET(0.83)[subject]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[freebsd.org]; RBL_DBL_DONT_QUERY_IPS(0.00)[209.85.128.51:from]; TO_DN_SOME(0.00)[]; SPAMHAUS_ZRD(0.00)[209.85.128.51:from:127.0.2.255]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-0.30)[-0.303]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[209.85.128.51:from]; NEURAL_HAM_SHORT(-1.00)[-0.999]; FORGED_SENDER(0.30)[uqs@freebsd.org,uspoerlein@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.128.51:from]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[uqs@freebsd.org,uspoerlein@gmail.com]; MAILMAN_DEST(0.00)[freebsd-git]; RCVD_COUNT_TWO(0.00)[2] X-Spam: Yes --00000000000054214305c2b38120 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, May 18, 2021 at 7:49 PM Mark Johnston wrote: > On Sun, Apr 25, 2021 at 04:02:22PM +0200, Ulrich Sp=C3=B6rlein wrote: > > On Sat, 2021-04-24 at 11:08:58 -0400, Mark Johnston wrote: > > >On Sat, Apr 24, 2021 at 12:44:40PM +0200, Ulrich Sp=C3=B6rlein wrote: > > >> On Fri, 2021-04-23 at 17:26:33 -0400, Mark Johnston wrote: > > >> >Hi, > > >> > > > >> >Now that FreeBSD uses OpenZFS as the upstream for ZFS, > vendor/illumos is > > >> >mostly unused. However, we still use illumos as an upstream for CT= F > > >> >tools and DTrace, though there haven't been any imports in a while. > > >> > > > >> >illumos has put a lot of work into their CTF toolchain, and I'd lik= e > to > > >> >import that. There are a couple of snags that I'd appreciate some > > >> >guidance on. > > >> > > > >> >First, I believe I should delete now-unused ZFS code from the vendo= r > > >> >branch and merge the result to main. I did this locally and got an > > >> >empty merge, which is what I'd expect. Is there any problem with > this? > > >> > > >> Why would you record this empty merge? If you clean up vendor/foo, > just > > >> do that but don't merge a no-op back into main (nothing changed, aft= er > > >> all). > > > > > >Ok, I guess there is no reason to merge that change separately. It > > >will end up being merged with subsequent imports though. > > > > > >> >Second, with Subversion we had both vendor/illumos and > > >> >vendor-sys/illumos, and now we just have the former, seemingly with > > >> >sys/* bits imported from vendor-sys. Some of the upstream commits > touch > > >> >both userspace and kernel bits, but the merge targets for these in > > >> >FreeBSD are different: cddl/contrib/opensolaris vs. > > >> >sys/cddl/contrib/opensolaris. How should I merge into main in this > > >> >case? I don't really see any options other than to split each > offending > > >> >upstream commit into two parts, one for userspace and one for the > > >> >kernel, and merge them separately. > > >> > > > >> >If it helps to look at the branch where I staged the upstream > commits, > > >> >I've pushed it to vendor/illumos2 in > https://github.com/markjdb/freebsd > > >> >. > > >> > > >> Can you clarify why the merging of the two might be an issue? Note > that > > >> unlike subversion, in git there's no "merge a certain subtree" > handling, > > >> all that is recorded is a tree of some form and then a set of parent= s > or > > >> ancestor commits. (git is a content tracker, not really a VCS :) > > >> > > >> I was under the impression that userland and kernel imports/merges > need > > >> to happen at the same time anyway, so I assume you would import all > the > > >> bits under vendor/foo in 1 commit and then merge them in 1 commit in= to > > >> main. Is that not how it goes? > > > > > >How can I do that with git subtree merge? Suppose an illumos commit > > >modifies cmd/dtrace/foo.c (userspace) and uts/common/dtrace/foo.c > > >(kernel). That maps to cddl/contrib/opensolaris/cmd/dtrace/foo.c and > > >sys/cddl/contrib/opensolaris/uts/common/dtrace/foo.c in FreeBSD, > > >respectively. So to do a subtree merge, I need to use distinct prefix= es > > >depending on whether I'm importing userspace or kernel changes. When > > >they are mixed together, it's not clear to me how I can merge at all. > > > > > >I see that for OpenZFS we keep all code, including userspace code, und= er > > >sys/contrib/openzfs, so it doesn't have this problem. > > > > I don't think you want a subtree merge, especially as things are > > scattered all over the place. Also note that none of this subtree magic > > is in any way recorded in the git data, all it does is help you with th= e > > 3-way merges (or whatever). > > > > So I would do: > > - import whatever you need into contrib/foo, commit normally. > > - munge /usr/src to have every kernel and userland stuff (not sure what > > other merge tools exist, just make sure to copy over file deletions as > > well :). You could rsync --del two times with the right source/dest > > pairs, or export a diff/patch from step 1 and apply it under the right > > prefixes. test, test, test. > > - write out this tree to git using: git write-tree > > - then commit this using: git commit-tree -m "my message" -p HEAD -p > > origin/vendor/illumos > > - bump main to point to that hash using git update-ref > > - git log --graph and inspect the hell out of this > > - git push, then curse that we disallow merge commits and you need to > > `git pull --rebase` to advance to the latest published head and that > > might mess up your merge commit pretty bad :( > > > > > > Maybe 2x git subtree merge + then rewriting and squashing them into 1 > > would work. But I fear it will record 3 parents, not 2 parents. > > > > Whatever you do, maybe please push to your private Github clone or our > > dev repo first and tell us where to look, so we can inspect whether it > > looks ok. > > I followed your suggestions and have what looks like a clean result. > Basically I did two subtree merges and committed the result. > > I pushed it here: > https://github.com/markjdb/freebsd/tree/ff/illumos-merge > The merge is the second-last commit on that branch. > > Rebasing the merge is painful, partly because there are some old ZFS > commits in the illumos vendor branch which git tries to merge. Rename > detection leads to some really weird conflicts as well. I'm not sure > how to handle this: there's a lot of room to make mistakes when > rebasing, so I would want to be careful and do extra testing before > pushing the final merge, but during that window it's likely that I will > end up having to rebase again. > > I should perhaps remove all ZFS bits from the vendor branch, and merge > it into main first before importing other stuff. > I see only 1 merge commit in there, is that the expected outcome? The output of git show -p --first-parent 23a19903267ee799cbddc35eb3e6f978ac1b4f38 looks mostly sane though, does it look like the changes that you'd like to bring into main? Warner, could you please also have a look? Cheers Uli --00000000000054214305c2b38120 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Tue, May 18, 2021 at 7:49 PM Mark = Johnston <markj@freebsd.org>= wrote:
On Sun, = Apr 25, 2021 at 04:02:22PM +0200, Ulrich Sp=C3=B6rlein wrote:
> On Sat, 2021-04-24 at 11:08:58 -0400, Mark Johnston wrote:
> >On Sat, Apr 24, 2021 at 12:44:40PM +0200, Ulrich Sp=C3=B6rlein wro= te:
> >> On Fri, 2021-04-23 at 17:26:33 -0400, Mark Johnston wrote: > >> >Hi,
> >> >
> >> >Now that FreeBSD uses OpenZFS as the upstream for ZFS, ve= ndor/illumos is
> >> >mostly unused.=C2=A0 However, we still use illumos as an = upstream for CTF
> >> >tools and DTrace, though there haven't been any impor= ts in a while.
> >> >
> >> >illumos has put a lot of work into their CTF toolchain, a= nd I'd like to
> >> >import that.=C2=A0 There are a couple of snags that I'= ;d appreciate some
> >> >guidance on.
> >> >
> >> >First, I believe I should delete now-unused ZFS code from= the vendor
> >> >branch and merge the result to main.=C2=A0 I did this loc= ally and got an
> >> >empty merge, which is what I'd expect.=C2=A0 Is there= any problem with this?
> >>
> >> Why would you record this empty merge? If you clean up vendor= /foo, just
> >> do that but don't merge a no-op back into main (nothing c= hanged, after
> >> all).
> >
> >Ok, I guess there is no reason to merge that change separately.=C2= =A0 It
> >will end up being merged with subsequent imports though.
> >
> >> >Second, with Subversion we had both vendor/illumos and > >> >vendor-sys/illumos, and now we just have the former, seem= ingly with
> >> >sys/* bits imported from vendor-sys.=C2=A0 Some of the up= stream commits touch
> >> >both userspace and kernel bits, but the merge targets for= these in
> >> >FreeBSD are different: cddl/contrib/opensolaris vs.
> >> >sys/cddl/contrib/opensolaris.=C2=A0 How should I merge in= to main in this
> >> >case?=C2=A0 I don't really see any options other than= to split each offending
> >> >upstream commit into two parts, one for userspace and one= for the
> >> >kernel, and merge them separately.
> >> >
> >> >If it helps to look at the branch where I staged the upst= ream commits,
> >> >I've pushed it to vendor/illumos2 in https:/= /github.com/markjdb/freebsd
> >> >.
> >>
> >> Can you clarify why the merging of the two might be an issue?= Note that
> >> unlike subversion, in git there's no "merge a certai= n subtree" handling,
> >> all that is recorded is a tree of some form and then a set of= parents or
> >> ancestor commits. (git is a content tracker, not really a VCS= :)
> >>
> >> I was under the impression that userland and kernel imports/m= erges need
> >> to happen at the same time anyway, so I assume you would impo= rt all the
> >> bits under vendor/foo in 1 commit and then merge them in 1 co= mmit into
> >> main. Is that not how it goes?
> >
> >How can I do that with git subtree merge?=C2=A0 Suppose an illumos= commit
> >modifies cmd/dtrace/foo.c (userspace) and uts/common/dtrace/foo.c<= br> > >(kernel).=C2=A0 That maps to cddl/contrib/opensolaris/cmd/dtrace/f= oo.c and
> >sys/cddl/contrib/opensolaris/uts/common/dtrace/foo.c in FreeBSD, > >respectively.=C2=A0 So to do a subtree merge, I need to use distin= ct prefixes
> >depending on whether I'm importing userspace or kernel changes= .=C2=A0 When
> >they are mixed together, it's not clear to me how I can merge = at all.
> >
> >I see that for OpenZFS we keep all code, including userspace code,= under
> >sys/contrib/openzfs, so it doesn't have this problem.
>
> I don't think you want a subtree merge, especially as things are <= br> > scattered all over the place. Also note that none of this subtree magi= c
> is in any way recorded in the git data, all it does is help you with t= he
> 3-way merges (or whatever).
>
> So I would do:
> - import whatever you need into contrib/foo, commit normally.
> - munge /usr/src to have every kernel and userland stuff (not sure wha= t
> other merge tools exist, just make sure to copy over file deletions as=
> well :). You could rsync --del two times with the right source/dest > pairs, or export a diff/patch from step 1 and apply it under the right=
> prefixes. test, test, test.
> - write out this tree to git using: git write-tree
> - then commit this using: git commit-tree -m "my message" -p= HEAD -p
> origin/vendor/illumos <tree hash from previous command>
> - bump main to point to that hash using git update-ref
> - git log --graph and inspect the hell out of this
> - git push, then curse that we disallow merge commits and you need to =
> `git pull --rebase` to advance to the latest published head and that <= br> > might mess up your merge commit pretty bad :(
>
>
> Maybe 2x git subtree merge + then rewriting and squashing them into 1 =
> would work. But I fear it will record 3 parents, not 2 parents.
>
> Whatever you do, maybe please push to your private Github clone or our=
> dev repo first and tell us where to look, so we can inspect whether it=
> looks ok.

I followed your suggestions and have what looks like a clean result.
Basically I did two subtree merges and committed the result.

I pushed it here:
https://github.com/markjdb/freebsd/tree/ff/i= llumos-merge
The merge is the second-last commit on that branch.

Rebasing the merge is painful, partly because there are some old ZFS
commits in the illumos vendor branch which git tries to merge.=C2=A0 Rename=
detection leads to some really weird conflicts as well.=C2=A0 I'm not s= ure
how to handle this: there's a lot of room to make mistakes when
rebasing, so I would want to be careful and do extra testing before
pushing the final merge, but during that window it's likely that I will=
end up having to rebase again.

I should perhaps remove all ZFS bits from the vendor branch, and merge
it into main first before importing other stuff.

<= /div>
I see only 1 merge commit in there, is that the expected outcome?=

The output of
git show -p --first-paren= t 23a19903267ee799cbddc35eb3e6f978ac1b4f38
looks mostly sane = though, does it look like the changes that you'd like to bring into mai= n?

Warner, could you please also have a look?

Cheers
Uli
--00000000000054214305c2b38120--