Git mirroring halted for freebsd-base

[Ulrich Spörlein]

2016-12-03 19:12 GMT+01:00 Shawn Webb <shawn.webb at hardenedbsd.org>:
> On Sat, Dec 03, 2016 at 06:45:32PM +0100, Ulrich Sp??rlein wrote:
>> 2016-12-03 18:29 GMT+01:00 Shawn Webb <shawn.webb at hardenedbsd.org>:
>> > On Sat, Dec 03, 2016 at 06:25:29PM +0100, Ulrich Sp??rlein wrote:
>> >> 2016-12-03 18:21 GMT+01:00 Shawn Webb <shawn.webb at hardenedbsd.org>:
>> >> > On Sat, Dec 03, 2016 at 12:42:56PM +0100, Ulrich Sp??rlein wrote:
>> >> >> 2016-12-02 11:07 GMT+01:00 Ulrich Sp??rlein <uqs at freebsd.org>:
>> >> >> > The conversion process started chewing up 100% cpu without making much progress, first attempts to rectify this have failed.
>> >> >> >
>> >> >> > The svn2git conversion and pushes to github have been halted. Pushes to bitbucket also have been halted (we're reaching the 2GB limit imposed by bitbucket).
>> >> >> >
>> >> >> > I'll update this thread in about 24h.
>> >> >> > Uli
>> >> >>
>> >> >> Service is fully restored now for the github mirror, sorry for the downtime.
>> >> >> Bitbucket will stop working soon because of the size limitations.
>> >> >> Speak up if you require this mirror to be kept up-to-date.
>> >> >>
>> >> >> Cheers,
>> >> >> Uli
>> >> >
>> >> > Looks like it might be easier for some downstream projects to fully
>> >> > recreate their ports repositories from scratch than to try to merge from
>> >> > upstream.
>> >>
>> >> What are you referring to here?
>> >
>> > The ports repo at https://github.com/freebsd/freebsd-ports was force
>> > pushed. Now attempts at merging in upstream's ports tree into
>> > hardenedbsd's causes merge conflicts for hundreds of files, including
>> > files we didn't change.
>> >
>> > So I'm forced to either inspect hundreds of files, manually merging in
>> > the changes or recreate our ports tree from scratch, re-importing our
>> > changes in a single atomic commit. The second option sounds more
>> > appealing, though we'd lose the entire history of our changes.
>> >
>> > Additionally, anyone downstream from HardenedBSD might have to do the
>> > same. Domino affect.
>>
>> I see. This shouldn't have happend, but as svnsync is
>> non-transactional, we picked up some bad SVN metadata that made it
>> into ports and base repos about a year ago. The SVN corruption was
>> promptly fixed (I didn't ask for this), but that now leaves us with no
>> way to actually re-do the conversion from scratch, as you'd need a
>> corrupted SVN repo to produce the same results.
>>
>> You should be able to simply merge whatever "official" commit you last
>> merged to with whatever the new "official" commit is now. This only
>> affected metadata, so you'll get a clean merge (no conflicts) but you
>> end up depending on 2x the history for about a year or so. Shouldn't
>> be that much of a problem. Ask your local git wizard on how to do this
>> best.
>>
>> >> > What caused the issue? What is going to be done to prevent it from
>> >> > happening again?
>> >>
>> >> I have no root cause, other than bitbucket changing permissions and
>> >> somehow git ending up using 100% CPU for most of the operations.
>> >
>> > So no guarantees this massive screw-up won't happen again?
>>
>> I said this before, and I'll say it again. This is a best-effort
>> conversion and we're at the mercy of whatever SVN fucks up next. I
>> provided clear instructions as to how to do the conversion in-house,
>> and guess how many people actually wrote to me that they end up with
>> different SHA hashes on github than they can produce in-house for both
>> src and ports?
>>
>> What would be your guess?
>>
>> Exactly, 0 people have done the in-house conversion and have compared
>> this to github. I could have put all kinds of backdoors in FreeBSD on
>> github and not a single soul would've noticed.
>>
>> So if you depend on it, I would very much appreciate if you could do
>> the same conversion in-house and report any drift as soon as possible,
>> because it's a mess otherwise, as you can see.
>>
>> Any thoughts on how to fix this for src would also be appreciated, all
>> I can think of is either pushing 2 heads and telling people to
>> migrate, or doing the switchover on a flag day.
>
> Hey Uli,
>
> Sorry for the harsher tone earlier. I'm a bit stressed and it was unfair
> of me to use that tone.
>
> I'm grateful for your efforts. I understand that supporting git isn't an
> official service provided by FreeBSD. One item on my Christmas wishlst
> would be to have official support for a read-only git mirror of the
> various FreeBSD projects (mainly src and ports).
>

No worries, I feel your pain. My Friday evening was ruined too :(

If you have the resources, *please* try a conversion run of all three
repos too and report your findings. We need to make sure that the
conversion is now more robust and we need to detect drift earlier :/

Cheers,
Uli