Git mirroring halted for freebsd-base

[Ulrich Spörlein]

2016-12-03 18:29 GMT+01:00 Shawn Webb <shawn.webb at hardenedbsd.org>:
> On Sat, Dec 03, 2016 at 06:25:29PM +0100, Ulrich Sp??rlein wrote:
>> 2016-12-03 18:21 GMT+01:00 Shawn Webb <shawn.webb at hardenedbsd.org>:
>> > On Sat, Dec 03, 2016 at 12:42:56PM +0100, Ulrich Sp??rlein wrote:
>> >> 2016-12-02 11:07 GMT+01:00 Ulrich Sp??rlein <uqs at freebsd.org>:
>> >> > The conversion process started chewing up 100% cpu without making much progress, first attempts to rectify this have failed.
>> >> >
>> >> > The svn2git conversion and pushes to github have been halted. Pushes to bitbucket also have been halted (we're reaching the 2GB limit imposed by bitbucket).
>> >> >
>> >> > I'll update this thread in about 24h.
>> >> > Uli
>> >>
>> >> Service is fully restored now for the github mirror, sorry for the downtime.
>> >> Bitbucket will stop working soon because of the size limitations.
>> >> Speak up if you require this mirror to be kept up-to-date.
>> >>
>> >> Cheers,
>> >> Uli
>> >
>> > Looks like it might be easier for some downstream projects to fully
>> > recreate their ports repositories from scratch than to try to merge from
>> > upstream.
>>
>> What are you referring to here?
>
> The ports repo at https://github.com/freebsd/freebsd-ports was force
> pushed. Now attempts at merging in upstream's ports tree into
> hardenedbsd's causes merge conflicts for hundreds of files, including
> files we didn't change.
>
> So I'm forced to either inspect hundreds of files, manually merging in
> the changes or recreate our ports tree from scratch, re-importing our
> changes in a single atomic commit. The second option sounds more
> appealing, though we'd lose the entire history of our changes.
>
> Additionally, anyone downstream from HardenedBSD might have to do the
> same. Domino affect.

I see. This shouldn't have happend, but as svnsync is
non-transactional, we picked up some bad SVN metadata that made it
into ports and base repos about a year ago. The SVN corruption was
promptly fixed (I didn't ask for this), but that now leaves us with no
way to actually re-do the conversion from scratch, as you'd need a
corrupted SVN repo to produce the same results.

You should be able to simply merge whatever "official" commit you last
merged to with whatever the new "official" commit is now. This only
affected metadata, so you'll get a clean merge (no conflicts) but you
end up depending on 2x the history for about a year or so. Shouldn't
be that much of a problem. Ask your local git wizard on how to do this
best.

>> > What caused the issue? What is going to be done to prevent it from
>> > happening again?
>>
>> I have no root cause, other than bitbucket changing permissions and
>> somehow git ending up using 100% CPU for most of the operations.
>
> So no guarantees this massive screw-up won't happen again?

I said this before, and I'll say it again. This is a best-effort
conversion and we're at the mercy of whatever SVN fucks up next. I
provided clear instructions as to how to do the conversion in-house,
and guess how many people actually wrote to me that they end up with
different SHA hashes on github than they can produce in-house for both
src and ports?

What would be your guess?

Exactly, 0 people have done the in-house conversion and have compared
this to github. I could have put all kinds of backdoors in FreeBSD on
github and not a single soul would've noticed.

So if you depend on it, I would very much appreciate if you could do
the same conversion in-house and report any drift as soon as possible,
because it's a mess otherwise, as you can see.

Any thoughts on how to fix this for src would also be appreciated, all
I can think of is either pushing 2 heads and telling people to
migrate, or doing the switchover on a flag day.

Cheers,
Uli