distributed scm+freebsd svn?
Alfred Perlstein
alfred at freebsd.org
Tue Aug 4 11:38:30 UTC 2009
This is amazing! Have you considered an article for publication based
on your experiences? Perhaps a handbook article?
This will be very helpful going forward.
(Although, I do kind of wish it was a "instructions fits on
the back of a napkin" kind of operation.) :)
thanks again!
-Alfred
* Giorgos Keramidas <keramida at freebsd.org> [090802 18:32] wrote:
> On Sun, 26 Jul 2009 16:15:34 -0700, Alfred Perlstein <alfred at freebsd.org> wrote:
> > Hello hackers,
> >
> > Does anyone here use one of the distributed SCMs to manage
> > contributions to FreeBSD in an easy manner?
>
> Hi Alfred,
> Yes, I do that.
>
> > Any pointers to a setup you have?
> >
> > I thought "git" was supposed to make this easy, but going over the
> > docs leaves me with a lot of questions.
>
> Git is a wonderful system but it's "UI" and documentation often make me
> want to scream bad things. My own suggestion is to go with Mercurial,
> because it's command set looks a *lot* like CVS or Subversion, it's often
> as fast or even faster than Git, and it doesn't seem as 'confusing' as Git.
>
> More details below...
>
> > I'm hoping to be able to basically:
> > sync into my "distributed repo".
> > allow a third party access to it.
> > easily commit upstream back into svn from a branch
> > in my distributed scm.
>
> I use a local Mercurial repository for my own patches. It seems to
> support most of the things I want to do, i.e.:
>
> * Keep a clean `/hg/bsd/head' workspace and pull full changesets into
> that from our svn repository
>
> * Support incremental updates of `/hg/bsd/head'.
>
> * Easily clone my `/hg/bsd/head' to one or more `feature' branches.
>
> * Allow others to pull from `head' as a read-only source over http or
> ssh.
>
> The /head branch has a huge history that I don't really want to keep
> around in every clone. So I started my conversion from 2007-12-31 and I
> keep updating it with the `hg convert ...' command wrapped in a small
> shell script:
>
> $ cat -n /hg/bsd/pull-head.sh
> 1 #!/bin/sh
> 2
> 3 set -e
> 4 hg convert \
> 5 --config convert.svn.startrev='175021' \
> 6 --config convert.svn.trunk='head' \
> 7 --config convert.svn.branches='' \
> 8 --config convert.svn.tags='' \
> 9 file:///home/svn/base/ \
> 10 /hg/bsd/head
>
> You can use the webdav http://svn.freebsd.org/base/ or an SSH tunneled
> URI to access to Subversion repository, but I keep a local mirror of the
> Subversion repository too, so I prefer that.
>
>
> Typical Mercurial-based Workflow
> ================================
>
> 1. Pull subversion commits into the 'head' workspace.
>
> 2. Pull these changes from 'head' to my working tree.
>
> 3. Merge the changes with the local patches of the working tree.
>
> 4. Extract one or more patches for committing to Subversion
>
> 5. Rinse, leather, repeat...
>
> Pulling the latest commits from Subversion
> ------------------------------------------
>
> The first step is the easiest bit. I just run `/hg/bsd/pull-head.sh'.
>
> This requires an installed copy of the Python bindings of Subversion
> [devel/py-subversion] and the `convert' extension enabled in my ~/.hgrc
> file with:
>
> [extensions]
> convert =
>
> A sample run of `pull-head.sh' looks like this:
>
> keramida at kobe:/hg/bsd$ time ./pull-head.sh
> scanning source...
> sorting...
> converting...
> 1 Many network stack subsystems use a single global data structure to hold
> 0 Add padding to struct inpcb, missed during our padding sweep earlier in
> 3.306 real 1.809 user 0.619 sys
> keramida at kobe:/hg/bsd$
>
> This is reasonably fast, but it does come with an important caveat.
> It's not terribly important for my own work, but it *may* be for yours:
>
> The Python bindings of Subversion do not support svn:keywords, so
> all our manually configured '$FreeBSD$' stuff is unexpanded in the
> converted tree. Mergemaster may cause various levels of "fun" and
> "amusement" if you mix, match and alternate between svn-based and
> mercurial-based workspaces often!
>
> At this point, after "pull-head.sh" has finished running, the most
> recent commit in the head/.hg/ workspace state is the last commit by
> rwatson:
>
> keramida at kobe:/hg/bsd/head$ hg log --limit 1
> changeset: 12589:8ce7c7a0b804
> branch: head
> tag: tip
> user: rwatson
> date: Sun Aug 02 22:47:08 2009 +0000
> summary: Add padding to struct inpcb, missed during our padding sweep earlier in
>
> keramida at kobe:/hg/bsd/head$
>
> This clone/workspace is my 'clean' slate, and it only contains an `.hg'
> data store. No checkout or other workspace contents:
>
> keramida at kobe:/hg/bsd/head$ ls -la
> total 6
> drwxr-xr-x 3 keramida users - 512 Nov 10 2008 .
> drwxr-xr-x 8 keramida users - 512 Aug 3 02:36 ..
> drwxr-xr-x 3 keramida users - 512 Aug 3 02:36 .hg
> keramida at kobe:/hg/bsd/head$ du -sh .
> 243M .
> keramida at kobe:/hg/bsd/head$
>
> It does however contains separate changesets for each subversion commit,
> so I can browse the (local only now) history with a fair amount of speed.
>
>
> Pulling these changes in my personal workspace
> ----------------------------------------------
>
> The second step is to pull the latest versions in my personal workspace
> at `/hg/bsd/src':
>
> keramida at kobe:/home/keramida$ cd /hg/bsd/src
> 1) keramida at kobe:/hg/bsd/src$ hg incoming --style compact ../head
> comparing with /hg/bsd/head
> searching for changes
> 12588 6b04ed36e454 2009-08-02 19:43 +0000 rwatson
> Many network stack subsystems use a single global data structure to
> hold
>
> 12589[tip] 8ce7c7a0b804 2009-08-02 22:47 +0000 rwatson
> Add padding to struct inpcb, missed during our padding sweep earlier
> in
>
> 2) keramida at kobe:/hg/bsd/src$ hg pull ../head
> pulling from /hg/bsd/head
> searching for changes
> adding changesets
> adding manifests
> adding file changes
> added 2 changesets with 16 changes to 16 files (+1 heads)
> (run 'hg heads' to see heads, 'hg merge' to merge)
> keramida at kobe:/hg/bsd/src$
>
> Comments:
> ---------
>
> (1) Just look at what would be pulled from 'head' but don't actually
> make any changes to the local workspace or branch/data store.
>
> (2) Pull the changes, creating a 'fork' in the history of the local
> workspace at the place where the changes were grafted on top of
> the previous svn-based changeset.
>
> At this point the history of my personal workspace has "forked". The
> subversion changesets have been grafted on top of the previous svn
> commit. This is easier to show with a picture/graph of the local
> history, so here it is (from the 'graphlog' extension of Mercurial):
>
> keramida at kobe:/hg/bsd/src$ hg glog --limit 5
> o 12785[tip] 8ce7c7a0b804 2009-08-02 22:47 +0000 rwatson
> | Add padding to struct inpcb, missed during our padding sweep earlier in
> |
> o 12784:12782 6b04ed36e454 2009-08-02 19:43 +0000 rwatson
> | Many network stack subsystems use a single global data structure to hold
> |
> | @ 12783:12762,12782 c50060fec1db 2009-08-02 21:06 +0300 keramida
> |/| Merge from head
> | |
> o | 12782 8e4fc85e5aa3 2009-08-02 16:59 +0000 julian
> | | Stop uuidgen(2) from crashing in vimage kerenels.
> | |
> o | 12781 bf9c3383d680 2009-08-02 14:28 +0000 attilio
> | | Make the newbus subsystem Giant free by adding the new newbus sxlock.
> | |
> keramida at kobe:/hg/bsd/src$
>
> The last time I pulled from Subversion was after 16:59 UTC (this is the
> buildworld I am still running in the background). My current workspace
> contains a checkout of revision 12783/c50060fec1db (hence the '@' in the
> relevant node of the history graph).
>
> Merging the New Changesets
> --------------------------
>
> To prepare for my next `merge with head', I first look at the two
> 'heads' of the history. My last successful merge and the new 'head'
> created by importing from Subversion:
>
> keramida at kobe:/hg/bsd/src$ hg heads --style compact
> 12785[tip] 8ce7c7a0b804 2009-08-02 22:47 +0000 rwatson
> Add padding to struct inpcb, missed during our padding sweep earlier in
>
> 12783:12762,12782 c50060fec1db 2009-08-02 21:06 +0300 keramida
> Merge from head
>
> keramida at kobe:/hg/bsd/src$
>
> Then I verify that I do *not* have local uncommitted stuff:
>
> keramida at kobe:/hg/bsd/src$ hg status
> keramida at kobe:/hg/bsd/src$
>
> This takes a few seconds, but I don't want to throw away any changes I
> may have been working on without noticing.
>
> Since this is a clean checkout of revision 12783:c50060fec1db, I don't
> necessarily need the next step, but it's safe to do it. I check out a
> clean copy of the local merge head:
>
> keramida at kobe:/hg/bsd/src$ time hg update --clean c50060fec1db
> 0 files updated, 0 files merged, 0 files removed, 0 files unresolved
> 4.684 real 1.691 user 2.558 sys
>
> Being a clean checkout already and having the filesystem cache 'hot'
> from the last pull I just did, this is often fast enough.
>
> Finally a merge of the 'remote' Subversion based commits I just pulled:
>
> keramida at kobe:/hg/bsd/src$ hg merge
> merging sys/netinet/in_gif.c
> merging sys/netinet/ip_var.h
> 14 files updated, 2 files merged, 0 files removed, 0 files unresolved
> (branch merge, don't forget to commit)
> You have new mail in /home/keramida/Mailbox
> keramida at kobe:/hg/bsd/src$ hg commit -m 'Merge from head'
> keramida at kobe:/hg/bsd/src$
>
> That's all. Now I have my own local changes 'merged' with the latest
> changeset from Subversion. There are no conflicts to resolve manually,
> because the subversion changesets I pulled do not affect any of the
> files with local-only changes, so this merge was relatively pain-free,
> quite fast and most importantly required no manual input at all :)
>
> Looking at the history with graphlog again, shows something like this:
>
> keramida at kobe:/hg/bsd/src$ hg glog --limit 5
> @ 12786[tip]:12783,12785 368efb2b98b9 2009-08-03 03:00 +0300 keramida
> |\ Merge from head
> | |
> | o 12785 8ce7c7a0b804 2009-08-02 22:47 +0000 rwatson
> | | Add padding to struct inpcb, missed during our padding sweep earlier in
> | |
> | o 12784:12782 6b04ed36e454 2009-08-02 19:43 +0000 rwatson
> | | Many network stack subsystems use a single global data structure to hold
> | |
> o | 12783:12762,12782 c50060fec1db 2009-08-02 21:06 +0300 keramida
> |\| Merge from head
> | |
> | o 12782 8e4fc85e5aa3 2009-08-02 16:59 +0000 julian
> | | Stop uuidgen(2) from crashing in vimage kerenels.
> | |
> keramida at kobe:/hg/bsd/src$
>
> Merging with 'head' often means that you can then publish this workspace
> so others can pull from it, and publish their own patches for the same
> workspace. How often you merge with each other is up to you. In the
> Greek documentation project we often merge after many days or several
> weeks. With projects that have a higher local change rate merging every
> day might be nicer (and result in far fewer conflicts).
>
>
> Extracting Patches from the Local Workspace
> ===========================================
>
> Keeping local changes means you may eventually want to push those
> changes towards the /head of Subversion. You'll have to extract patches
> for one or more file then. Since you have the tools to look at the
> local history, you can use "hg diff" with the last subversion commit,
> i.e. with the history shown above:
>
> keramida at kobe:/hg/bsd/src$ hg glog --limit 5
> @ 12786[tip]:12783,12785 368efb2b98b9 2009-08-03 03:00 +0300 keramida
> |\ Merge from head
> | |
> | o 12785 8ce7c7a0b804 2009-08-02 22:47 +0000 rwatson
> | | Add padding to struct inpcb, missed during our padding sweep earlier in
> | |
> | o 12784:12782 6b04ed36e454 2009-08-02 19:43 +0000 rwatson
> | | Many network stack subsystems use a single global data structure to hold
> | |
> o | 12783:12762,12782 c50060fec1db 2009-08-02 21:06 +0300 keramida
> |\| Merge from head
> : :
>
> You can run "hg diff" between changesets 12785 and 12786 (tip):
>
> keramida at kobe:/hg/bsd/src$ hg diff -r 12785:tip | diffstat -p1
> contrib/mg/Makefile | 29 +
> contrib/mg/README | 74 ++
> contrib/mg/autoexec.c | 111 ++++
> ... snip ...
> usr.bin/yacc/reader.c | 2
> usr.sbin/chown/chgrp.1 | 10
> usr.sbin/chown/chown.8 | 12
> usr.sbin/chown/chown.c | 17
> 76 files changed, 18355 insertions(+), 44 deletions(-)
> keramida at kobe:/hg/bsd/src$
>
> My own local workspace includes an import of OpenBSD's mg(1) editor.
> This shows as a single patch in the diff command I used above, along
> with *every* other file that differs in my local 'branch' of FreeBSD
> head. Let's assume, for example's sake, that you don't really care
> about mg(1) patches, and you want to look at everything else. The
> --include and --exclude options of "hg diff" help a lot there (short
> names -I and -X):
>
> keramida at kobe:/hg/bsd/src$ hg diff -r 12785:tip -X contrib/mg -X usr.bin/mg | diffstat -p1
> contrib/top/top.X | 5 -
> contrib/top/top.c | 2
> etc/Makefile | 6 +
> etc/mtree/BSD.games.dist | 16 +++
> etc/mtree/BSD.usr.dist | 2
> etc/mtree/BSD.var.dist | 2
> libexec/rtld-elf/rtld.c | 2
> share/mk/bsd.own.mk | 2
> sys/amd64/conf/KOBE | 185 ++++++++++++++++++++++++++++++++++++++
> sys/boot/common/interp.c | 2
> sys/boot/common/interp_forth.c | 2
> sys/conf/newvers.sh | 26 +++++
> sys/i386/conf/KOBE | 195 +++++++++++++++++++++++++++++++++++++++++
> usr.bin/Makefile | 5 +
> usr.bin/truss/amd64-fbsd.c | 4
> usr.bin/truss/amd64-fbsd32.c | 3
> usr.bin/truss/amd64-linux32.c | 4
> usr.bin/truss/i386-fbsd.c | 4
> usr.bin/truss/i386-linux.c | 4
> usr.bin/truss/ia64-fbsd.c | 4
> usr.bin/truss/powerpc-fbsd.c | 4
> usr.bin/truss/sparc64-fbsd.c | 4
> usr.bin/yacc/reader.c | 2
> usr.sbin/chown/chgrp.1 | 10 +-
> usr.sbin/chown/chown.8 | 12 +-
> usr.sbin/chown/chown.c | 17 ++-
> 26 files changed, 480 insertions(+), 44 deletions(-)
> keramida at kobe:/hg/bsd/src$
>
> If you want to commit *all* the local changes as a single patch to
> Subversion, you can keep refining the --exclude/--include patterns until
> the final patch looks and smells "right".
>
> If you know the directories that you want to diff, i.e. you want to
> commit all the local `usr.sbin/chown' changes in one subversion
> changeset, you can use a directory with "hg diff" too (which works quite
> unsurprisingly like "svn diff"):
>
> keramida at kobe:/hg/bsd/src$ hg diff -r 12785:tip usr.sbin/chown | diffstat -p1
> usr.sbin/chown/chgrp.1 | 10 +++++++---
> usr.sbin/chown/chown.8 | 12 ++++++++----
> usr.sbin/chown/chown.c | 17 +++++++++++------
> 3 files changed, 26 insertions(+), 13 deletions(-)
> keramida at kobe:/hg/bsd/src$
>
> Extracting Local Changesets as Patches
> ======================================
>
> Another option is to extract only the very specific local commit that
> affected a file, i.e. my local `usr.sbin/chown/chown.c' changes. First
> you'd have to look at the local history of the file:
>
> keramida at kobe:/hg/bsd/src$ hg log --style compact usr.sbin/chown/chown.c
> 12010:11998,12009 7f4fa839afed 2009-06-20 00:50 +0300 keramida
> Merge from head
>
> 12002 96e04082ef3f 2009-06-19 15:58 +0000 brooks
> In preparation for raising NGROUPS and NGROUPS_MAX, change base
>
> 7782 141cd5ffef80 2009-01-22 07:18 +0200 keramida
> Add a new -x option to chown and chgrp, to inhibit file system
>
> 0 dd5ed0412a8b 2007-12-31 22:03 +0000 jhb
> Actually declare the kern.features sysctl node.
>
> keramida at kobe:/hg/bsd/src$
>
> >From this output it's obvious my local changes were made on 2009-01-22
> 07:18 +0200, and they are available as changeset 7782:141cd5ffef80. You
> can extract this changeset only with "hg export":
>
> keramida at kobe:/hg/bsd/src$ hg export 141cd5ffef80
> # HG changeset patch
> # User Giorgos Keramidas <keramida at ceid.upatras.gr>
> # Date 1232601528 -7200
> # Branch head
> # Node ID 141cd5ffef80ff979627d8898500c92984287426
> # Parent e8506b2ac7aefbfb875f0def0de8dd6441885a40
> Add a new -x option to chown and chgrp, to inhibit file system
> mount point traversal. The -x option is documented, like -v, as
> a non-standard option in the COMPATIBILITY manpage sections.
>
> diff -r e8506b2ac7ae -r 141cd5ffef80 usr.sbin/chown/chgrp.1
> --- a/usr.sbin/chown/chgrp.1 Wed Jan 21 21:31:33 2009 +0200
> +++ b/usr.sbin/chown/chgrp.1 Thu Jan 22 07:18:48 2009 +0200
> @@ -31,7 +31,7 @@
> .\" @(#)chgrp.1 8.3 (Berkeley) 3/31/94
> .\" $FreeBSD$
> .\"
> -.Dd April 25, 2003
> +.Dd January 22, 2009
> .Dt CHGRP 1
> .Os
> .Sh NAME
> @@ -39,7 +39,7 @@
> .Nd change group
> .Sh SYNOPSIS
> .Nm
> -.Op Fl fhv
> +.Op Fl fhvx
> .Oo
> .Fl R
> ... snip ...
>
> This patch is _directly_ usable with `patch -p1' in a checkout of /head
> from Subversion, but it *may* require a bit of `svn merge' work if you
> have fast-forwarded chown/chgrp to its latest 'head' version. It is not
> a diff of the *latest* chown/chgrp from /head but a _precise_ copy of
> the past changeset, as it was committed.
>
> More...
> =======
>
> * You can browse the 'clean' head/ copy I am using at
>
> http://hg.hellug.gr/freebsd/head/
>
> Note: this only has the head history since 2007-12-31. For older head
> commits, you will have to use a new "convert" run and it will change
> all the commit/changeset hashes even if their patch data is identical.
>
> * You can find a compressed 'bundle' file called 'head.hg' in my home
> directory at freefall. This can be used to 'seed' the initial copy
> of your local 'head', eg. by pulling directly from the bundle:
>
> % cd /var/tmp
> % scp freefall.freebsd.org:'~keramida/head.hg' .
>
> % cd ~
> % mkdir -p ~/work/freebsd/head
> % hg init ~/work/freebsd/head
> % cd ~/work/freebsd/head
> % hg pull /var/tmp/head.hg
>
> * If you plan to use the incremental conversion script I described
> earlier in this message, you will also need a "SHA map" file that
> maps Subversion changesets to Mercurial changeset hashes. This is
> also available at freefall as `~keramida/head.shamap' and you have
> to copy it to your `head/.hg/shamap' file. Then either run the "hg
> convert" extension manually or use a small shell wrapper like the
> one I pasted here.
>
> * For more information about some of the extensions I've mentioned it
> is always a good idea to check the Mercurial Wiki at:
>
> http://www.selenic.com/mercurial/
>
> I hope all this helps a bit...
>
> If you need more help with publishing local workspaces over http and/or
> extracting patches (these are often two of the points where help is
> necessary and welcome), please feel free to ask. I've been using this
> sort of workflow for local changesets quite some time and I know enough
> about Mercurial to help where needed.
--
- Alfred Perlstein
.- AMA, VMOA #5191, 03 vmax, 92 gs500, 85 ch250
.- FreeBSD committer
More information about the freebsd-hackers
mailing list