Git mirroring halted for freebsd-base

[Shawn Webb]

On Sat, Dec 03, 2016 at 06:45:32PM +0100, Ulrich Sp??rlein wrote:
> 2016-12-03 18:29 GMT+01:00 Shawn Webb <shawn.webb at hardenedbsd.org>:
> > On Sat, Dec 03, 2016 at 06:25:29PM +0100, Ulrich Sp??rlein wrote:
> >> 2016-12-03 18:21 GMT+01:00 Shawn Webb <shawn.webb at hardenedbsd.org>:
> >> > On Sat, Dec 03, 2016 at 12:42:56PM +0100, Ulrich Sp??rlein wrote:
> >> >> 2016-12-02 11:07 GMT+01:00 Ulrich Sp??rlein <uqs at freebsd.org>:
> >> >> > The conversion process started chewing up 100% cpu without making much progress, first attempts to rectify this have failed.
> >> >> >
> >> >> > The svn2git conversion and pushes to github have been halted. Pushes to bitbucket also have been halted (we're reaching the 2GB limit imposed by bitbucket).
> >> >> >
> >> >> > I'll update this thread in about 24h.
> >> >> > Uli
> >> >>
> >> >> Service is fully restored now for the github mirror, sorry for the downtime.
> >> >> Bitbucket will stop working soon because of the size limitations.
> >> >> Speak up if you require this mirror to be kept up-to-date.
> >> >>
> >> >> Cheers,
> >> >> Uli
> >> >
> >> > Looks like it might be easier for some downstream projects to fully
> >> > recreate their ports repositories from scratch than to try to merge from
> >> > upstream.
> >>
> >> What are you referring to here?
> >
> > The ports repo at https://github.com/freebsd/freebsd-ports was force
> > pushed. Now attempts at merging in upstream's ports tree into
> > hardenedbsd's causes merge conflicts for hundreds of files, including
> > files we didn't change.
> >
> > So I'm forced to either inspect hundreds of files, manually merging in
> > the changes or recreate our ports tree from scratch, re-importing our
> > changes in a single atomic commit. The second option sounds more
> > appealing, though we'd lose the entire history of our changes.
> >
> > Additionally, anyone downstream from HardenedBSD might have to do the
> > same. Domino affect.
> 
> I see. This shouldn't have happend, but as svnsync is
> non-transactional, we picked up some bad SVN metadata that made it
> into ports and base repos about a year ago. The SVN corruption was
> promptly fixed (I didn't ask for this), but that now leaves us with no
> way to actually re-do the conversion from scratch, as you'd need a
> corrupted SVN repo to produce the same results.
> 
> You should be able to simply merge whatever "official" commit you last
> merged to with whatever the new "official" commit is now. This only
> affected metadata, so you'll get a clean merge (no conflicts) but you
> end up depending on 2x the history for about a year or so. Shouldn't
> be that much of a problem. Ask your local git wizard on how to do this
> best.
> 
> >> > What caused the issue? What is going to be done to prevent it from
> >> > happening again?
> >>
> >> I have no root cause, other than bitbucket changing permissions and
> >> somehow git ending up using 100% CPU for most of the operations.
> >
> > So no guarantees this massive screw-up won't happen again?
> 
> I said this before, and I'll say it again. This is a best-effort
> conversion and we're at the mercy of whatever SVN fucks up next. I
> provided clear instructions as to how to do the conversion in-house,
> and guess how many people actually wrote to me that they end up with
> different SHA hashes on github than they can produce in-house for both
> src and ports?
> 
> What would be your guess?
> 
> Exactly, 0 people have done the in-house conversion and have compared
> this to github. I could have put all kinds of backdoors in FreeBSD on
> github and not a single soul would've noticed.
> 
> So if you depend on it, I would very much appreciate if you could do
> the same conversion in-house and report any drift as soon as possible,
> because it's a mess otherwise, as you can see.
> 
> Any thoughts on how to fix this for src would also be appreciated, all
> I can think of is either pushing 2 heads and telling people to
> migrate, or doing the switchover on a flag day.

Hey Uli,

Sorry for the harsher tone earlier. I'm a bit stressed and it was unfair
of me to use that tone.

I'm grateful for your efforts. I understand that supporting git isn't an
official service provided by FreeBSD. One item on my Christmas wishlst
would be to have official support for a read-only git mirror of the
various FreeBSD projects (mainly src and ports).

Thanks,

-- 
Shawn Webb
Cofounder and Security Engineer
HardenedBSD

GPG Key ID:          0x6A84658F52456EEE
GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89  3D9E 6A84 658F 5245 6EEE
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-git/attachments/20161203/112003ce/attachment.sig>