From nobody Wed May 19 19:08:49 2021 X-Original-To: freebsd-git@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5497F8B1364 for ; Wed, 19 May 2021 19:08:51 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qv1-xf31.google.com (mail-qv1-xf31.google.com [IPv6:2607:f8b0:4864:20::f31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Flj8M1YFzz3s5m; Wed, 19 May 2021 19:08:51 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qv1-xf31.google.com with SMTP id q6so7372730qvb.2; Wed, 19 May 2021 12:08:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=x389GeFtfw8lj1/IPscnikVYSYj5+b2yIfdXOGCmp0Y=; b=BlxfLbiVJPTkCon+Be7ouE4Ix4+eLm4swE2NDtVIusazUZlSclZbroh1E30ecogW9P dx+Sxg0wUh8RxFAjur64M8fIqfOvWyIFlTbAb75YcCrvIP7fNoo4efRimGaty02/ph2C 53J9Zimn60nwSOuehk1Y8MP7qfY39dL8OT5pRwHFYS2zMiJ4dce+EaC6UHERFgMI2tSY HP+biV9rlIgHk3JIUJU83xsbE3Id20XGVCtIUIbHbn9w46ctB1CNPWrgVLg3AqmhmO7I o7jw1/sHmnaOK06x70VscKfBzCgKFUNCVgT6znN7//iqJT1VuOEfHu9neTOLT+Lru/IB KoLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition :content-transfer-encoding:in-reply-to; bh=x389GeFtfw8lj1/IPscnikVYSYj5+b2yIfdXOGCmp0Y=; b=pcHi+NcmoclLOB/rDHBwZI8ri/ywwxAYNRPDuOCFGvJfDhCcOTUTeR237wW86K0U46 p7eXX1zYNMFRakQ3jDmDaw4Wj20uiKyHoj4wqdnSrWqvUUWbHXTe/emMWIOq6kQQfWpG WD8w3Olhxd46RjCyhem1/Rzi5koubVCkQPTA0aneZJV8XaQR03ri7NC9mno/Axsze83g OdvAB1HLpJpIw6efw0fPrzUWb5KXeB5/LKx6bP7Mssxhvjte8+XkVXvISMnIiZfLaq68 X3mqBmds/faX7Myj/hD6lSCmarVJAaPJynUG5n0fk2Ixo35K6z+uQm3/MGyB4IgoJpY9 vnkw== X-Gm-Message-State: AOAM530mZcZRnOVDNd+5BTZlPFLVmFs1UsAdHBh2zggBo2Qs7LbAp9yA +pSyllyFmupUnE11MkSk/i2DcVOOtSg= X-Google-Smtp-Source: ABdhPJxhTNV45JxL6YQuNcJGjhkyY83Cv+sjQX0VmF4Qjf/HmE4Y1eu3dis9vMQQdjnNyncKSlxOoA== X-Received: by 2002:ad4:5310:: with SMTP id y16mr1049134qvr.39.1621451329912; Wed, 19 May 2021 12:08:49 -0700 (PDT) Received: from nuc ([142.126.156.186]) by smtp.gmail.com with ESMTPSA id d200sm446155qkc.44.2021.05.19.12.08.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 May 2021 12:08:48 -0700 (PDT) Sender: Mark Johnston Date: Wed, 19 May 2021 15:08:49 -0400 From: Mark Johnston To: Ulrich =?iso-8859-1?Q?Sp=F6rlein?= Cc: freebsd-git@freebsd.org Subject: Re: vendor/illumos merges Message-ID: References: List-Id: Discussion of git use in the FreeBSD project List-Archive: https://lists.freebsd.org/archives/freebsd-git List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-git@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: 4Flj8M1YFzz3s5m X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] On Wed, May 19, 2021 at 09:04:57PM +0200, Ulrich Spörlein wrote: > On Tue, May 18, 2021 at 7:49 PM Mark Johnston wrote: > > > On Sun, Apr 25, 2021 at 04:02:22PM +0200, Ulrich Spörlein wrote: > > > On Sat, 2021-04-24 at 11:08:58 -0400, Mark Johnston wrote: > > > >On Sat, Apr 24, 2021 at 12:44:40PM +0200, Ulrich Spörlein wrote: > > > >> On Fri, 2021-04-23 at 17:26:33 -0400, Mark Johnston wrote: > > > >> >Hi, > > > >> > > > > >> >Now that FreeBSD uses OpenZFS as the upstream for ZFS, > > vendor/illumos is > > > >> >mostly unused. However, we still use illumos as an upstream for CTF > > > >> >tools and DTrace, though there haven't been any imports in a while. > > > >> > > > > >> >illumos has put a lot of work into their CTF toolchain, and I'd like > > to > > > >> >import that. There are a couple of snags that I'd appreciate some > > > >> >guidance on. > > > >> > > > > >> >First, I believe I should delete now-unused ZFS code from the vendor > > > >> >branch and merge the result to main. I did this locally and got an > > > >> >empty merge, which is what I'd expect. Is there any problem with > > this? > > > >> > > > >> Why would you record this empty merge? If you clean up vendor/foo, > > just > > > >> do that but don't merge a no-op back into main (nothing changed, after > > > >> all). > > > > > > > >Ok, I guess there is no reason to merge that change separately. It > > > >will end up being merged with subsequent imports though. > > > > > > > >> >Second, with Subversion we had both vendor/illumos and > > > >> >vendor-sys/illumos, and now we just have the former, seemingly with > > > >> >sys/* bits imported from vendor-sys. Some of the upstream commits > > touch > > > >> >both userspace and kernel bits, but the merge targets for these in > > > >> >FreeBSD are different: cddl/contrib/opensolaris vs. > > > >> >sys/cddl/contrib/opensolaris. How should I merge into main in this > > > >> >case? I don't really see any options other than to split each > > offending > > > >> >upstream commit into two parts, one for userspace and one for the > > > >> >kernel, and merge them separately. > > > >> > > > > >> >If it helps to look at the branch where I staged the upstream > > commits, > > > >> >I've pushed it to vendor/illumos2 in > > https://github.com/markjdb/freebsd > > > >> >. > > > >> > > > >> Can you clarify why the merging of the two might be an issue? Note > > that > > > >> unlike subversion, in git there's no "merge a certain subtree" > > handling, > > > >> all that is recorded is a tree of some form and then a set of parents > > or > > > >> ancestor commits. (git is a content tracker, not really a VCS :) > > > >> > > > >> I was under the impression that userland and kernel imports/merges > > need > > > >> to happen at the same time anyway, so I assume you would import all > > the > > > >> bits under vendor/foo in 1 commit and then merge them in 1 commit into > > > >> main. Is that not how it goes? > > > > > > > >How can I do that with git subtree merge? Suppose an illumos commit > > > >modifies cmd/dtrace/foo.c (userspace) and uts/common/dtrace/foo.c > > > >(kernel). That maps to cddl/contrib/opensolaris/cmd/dtrace/foo.c and > > > >sys/cddl/contrib/opensolaris/uts/common/dtrace/foo.c in FreeBSD, > > > >respectively. So to do a subtree merge, I need to use distinct prefixes > > > >depending on whether I'm importing userspace or kernel changes. When > > > >they are mixed together, it's not clear to me how I can merge at all. > > > > > > > >I see that for OpenZFS we keep all code, including userspace code, under > > > >sys/contrib/openzfs, so it doesn't have this problem. > > > > > > I don't think you want a subtree merge, especially as things are > > > scattered all over the place. Also note that none of this subtree magic > > > is in any way recorded in the git data, all it does is help you with the > > > 3-way merges (or whatever). > > > > > > So I would do: > > > - import whatever you need into contrib/foo, commit normally. > > > - munge /usr/src to have every kernel and userland stuff (not sure what > > > other merge tools exist, just make sure to copy over file deletions as > > > well :). You could rsync --del two times with the right source/dest > > > pairs, or export a diff/patch from step 1 and apply it under the right > > > prefixes. test, test, test. > > > - write out this tree to git using: git write-tree > > > - then commit this using: git commit-tree -m "my message" -p HEAD -p > > > origin/vendor/illumos > > > - bump main to point to that hash using git update-ref > > > - git log --graph and inspect the hell out of this > > > - git push, then curse that we disallow merge commits and you need to > > > `git pull --rebase` to advance to the latest published head and that > > > might mess up your merge commit pretty bad :( > > > > > > > > > Maybe 2x git subtree merge + then rewriting and squashing them into 1 > > > would work. But I fear it will record 3 parents, not 2 parents. > > > > > > Whatever you do, maybe please push to your private Github clone or our > > > dev repo first and tell us where to look, so we can inspect whether it > > > looks ok. > > > > I followed your suggestions and have what looks like a clean result. > > Basically I did two subtree merges and committed the result. > > > > I pushed it here: > > https://github.com/markjdb/freebsd/tree/ff/illumos-merge > > The merge is the second-last commit on that branch. > > > > Rebasing the merge is painful, partly because there are some old ZFS > > commits in the illumos vendor branch which git tries to merge. Rename > > detection leads to some really weird conflicts as well. I'm not sure > > how to handle this: there's a lot of room to make mistakes when > > rebasing, so I would want to be careful and do extra testing before > > pushing the final merge, but during that window it's likely that I will > > end up having to rebase again. > > > > I should perhaps remove all ZFS bits from the vendor branch, and merge > > it into main first before importing other stuff. > > > > I see only 1 merge commit in there, is that the expected outcome? Yes. I should probably do this as two separate merge commits as I suggested above, but I wanted to know if I had done anything incorrectly in this attempt before spending more time on it. The branch isn't ready to merge yet in any case, there are some upstream bugs to squash. > The output of > git show -p --first-parent 23a19903267ee799cbddc35eb3e6f978ac1b4f38 > looks mostly sane though, does it look like the changes that you'd like to > bring into main? Yes, that's correct. > Warner, could you please also have a look? > > Cheers > Uli