Re: When are the git servers available to obtain the ports tree?

On 2022-05-03 00:48, Dave Cottlehuber wrote:
> On Tue, 3 May 2022, at 04:04, Philip Paeps wrote:
>>>>> I'm a maintainer for well over 100 ports. But more often than
>>>>> not, I am not permitted to obtain the ports tree from any of
>>>>> the FreeBSD git servers:
>>>>> 
>>>>> # git clone -o freebsd --config
>>>>> remote.freebsd.fetch='+refs/notes/*:refs/notes/*'
>>>>> https://git.freebsd.org/ports.git PORTS-20220502
> 
> The initial full clone from github, of ports and/or src,
> often fails with gateway errors. I've bugged them in the
> past and they have amended their "limits" once, and given
> up the next time.
> 
> I'm curious about the "shallow-info" appearing in your logs
> though.
Not sure what that meant either. The language is also not telling.
> 
> https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables
> 
> There's extensive debugging for git, and for HTTP, you
> could try `GIT_CURL_VERBOSE=1` and see if that helps
> identify problems.
clusteradm was (is?) kind enough to try and pin it down. So
we exchanged some info last night (mid day for them). I'll
give your suggestion a try. Thanks.
> 
> I've previously seen gateway errors when the remote git
> server takes too long to prepare whatever packfile it
> needs to send me. I'm murky on git internals so I don't
> really understand what it's failing on.
Make that 2 of us.
> 
> Generally, I only cloned a single branch, and fetch main,
> which seems to have removed issues for me, but I do this
> at least 1x/week. My freebsd remote is called upstream.
> 
> # fetch a single branch instead of all branches
> 
> git fetch upstream main
> 
> # my snippet from /usr/ports/.git/config
> 
> [remote "upstream"]
> 	url = https://git.freebsd.org/ports.git
> 	fetch = +refs/heads/main:refs/remotes/upstream/main
> 	fetch = +refs/notes/*:refs/notes/*
> 	pushurl = ssh://git@gitrepo.freebsd.org/ports.git
> 
> [branch "upstream"]
> 	remote = upstream
> 	merge = refs/heads/main
> 
> If, for some reason, you really need to do a from-scratch
> clone every time, it might be simpler to have a personal
> mirror of just the /main/ branch on github or similar, and
> avoid pulling the whole lot from the mirrors each time.
While I *do* fetch the entire history. On average, I only ever
do it at most, once a month. So I don't think it's unreasonable,
or that I'm overloading the FreeBSD git services. ;-)

I like you're personal mirror idea. In light of all this, I was
also thinking I might just create a mirror to pull from on either
my GitLab, or Codeberg account. But then that just felt like putting
Duct Tape on the problem. Rather than understanding, and fixing it. :-)
> 
> You can also put `git fetch <remote> <branch>` into a
> crontab and let it catch up daily and with less to fetch.
> 
Really great suggestions, Dave. I'll probably at least adopt your
.gitconfig and your GIT_CURL_VERBOSE=1 idea.
Thanks for taking the time to reply!
> A+
> Dave
l8r,
Chris