Monitoring commits on all branches

[Dan Langille]

> On Nov 19, 2020, at 11:16 AM, Marc Branchaud <marcnarc at gmail.com> wrote:
> 
> On 2020-11-18 8:49 p.m., Dan Langille wrote:
>> How can a repo be monitored for commits on all branches?
>> I know how to ask a given branch: do you have any commits after foo_hash?
>> How do I:
>> * get a list of all commits since foo_hash
> 
> A quick a note about Warner's reply:
> 
>> git log $hash..HEAD
> 
> "HEAD" is just a git nickname for "whatever you have currently checked-out" (which can be a branch, a tag, or "detached" commit SHA ID).

Mathieu mentioned: git log $foo_hash...branch_name

That was the first time I've seen that used. All previous suggestions were HEAD, not branch_name. But they are all the same in this context?

>> * know which branch each of those commits was on (e.g. master, branches/2020Q4)
> 
> Unfortunately you'll find most normal git advice to be a bit frustrating with the FreeBSD repos, because FreeBSD doesn't work the way most people use git.  Specifically, the FreeBSD project does not ever merge branches (in the git sense of the word "merge").  Things would be very, very much easier if the FreeBSD project were to use git-style merging.  I believe there are discussions underway about adjusting the whole MFC process for the git world.  I admit that part of my motivation in writing this message is to provide grist for that mill.
> 
> Fortunately even without git-merged branches, there are still git tools that help, though they're not as precise as one would like.
> 
> Let's look at a concrete example with the beta ports git repo (which I just cloned), and compare the 2020Q4 and main branches.  I'll start with some overall exploration, then address your specific question.
> 
> There are 298 commits in the 2020Q4 branch.  I know this because
>    git merge-base origin/main origin/branches/2020Q4
> tells me where 2020Q4 branched off of main: commit 5dbe4e5f775ea2.  And
>    git rev-list 5dbe4e5f775ea2..origin/branches/2020Q4 | wc -l
> says "299".  (The "rev-list" command is a bare-bones version of "log" that only lists commit SHA IDs.)

[examples snipped]

I followed that.

I took the merge information as background and good-to-know, because FreshPorts won't be doing any merging. It just needs a good "git checkout" working copy.

Sorry for such a long reply.


* How FreshPorts extracts data

FreshPorts is only interested in a snapshot of the repo with respect to a given commit.  It works on the 'repo as a whole' to extract values from the ports which were affected by that commit. Case in point: a commit to a parent port might affect any or all of the child ports.  All the child ports need to be refreshed.

I am quickly concluding that FreshPorts must decide in advance what git branches it will pay attention to. At present, it follows all branches.


* FreshPorts (without git) uses email to create XML

When moving FreshPorts from subversion to git, one of the goals was to avoid relying on email to know that a commit has occurred. That is how FreshPorts has always worked. The email (from the CVS commit) was parsed and XML created. This code was updated for SVN. The XML is then used to load the commit into the FreshPorts database which then drives the website contents.

When I started the GIT conversion, there was no commit email. "git log $foo_hash...HEAD" is how FreshPorts knows what commits to process.

One positive aspect of email approach: it identified the branch. So far, I can't see how I can process the repo as a whole and see every commit and know what branch it was on.


* Polling git

It is beginning to sound like the FreshPorts git code for detecting incoming commits will be:

Every N minutes, do this:

for each repo in REPOS
  for branch in BRANCHES
    cd to the directory for that repo
    git checkout branch
    git log $branch_last_hash...HEAD
    for each of those commits
       process the commit
    end for
  end for
end for

At present, the REPOS and BRANCHES are:

* freebsd BRANCHES="master branches/2020Q4 branches/2020Q3 branches/2020Q2 ...etc"
* freebsd-ports BRANCHES="master stable/12 stable/11"
* freebsd-doc BRANCHES="master"

Some might ask:

* Why not just master and latest-quarterly for freebsd-ports?
* Because commits to older branches sometimes occur (or at least I thought I saw one once)

Commit hooks might also help, but I'm not sure if that will make things easier or complicate everything


* When new branches arrive

It is vital that FreshPorts remain automated as much as it can be. At present, under SVN, I might have to fix things perhaps 5 or 6 times a year, usually because a commit did not get processed.

Keeping that in mind, I do not yet know how to handle the following situations:

A new branch comes out.

* Automation might be possible for ports quarterly branches
* FreshPorts has to know there is a new branch
* BRANCHES needs to be updated
* I don't see that it can be automated for stable/*
* need to handle 'git checkout branch' when branch does not exist?
* Once branch exists, how do you find out about the commits when you have no starting point for 'git log'?

Right now, a new quarterly branch is noticed when the first commit email comes through.  FreshPorts then does an 'svn co' for that branch.



I'm hoping someone has good ideas for my edge cases.

Thank you.

— 
Dan Langille
http://langille.org/