FreeBSD CURRENT stabilization cycle

  Hi FreeBSD CURRENT users,

back in November I came up with a proposal of providing some stabilization
cadence to development of the main branch, also known as FreeBSD CURRENT.  Here
is a video with the initial proposal and following discussion at VendorBSD
Conference (18 minutes):

https://www.youtube.com/live/k-AzShVdAHo?si=hPAhCd_-RuoTRqcW&t=2511

And here goes an up to date version of the plan!

In the last decade quality of FreeBSD CURRENT improved so much, that not only
brave developers run it on their laptops, but also large companies use it. Time
to bring it to a new level.  Every individual or a business that use CURRENT
has their own protocol of how to stay up date and avoid disasters.  An
individual will first update their desktop and only after that will update
server(s).  A company would run their internal regression test suite or some
other validation protocol.  Right now we all do that independently from each
other having little coordination and providing little help each other.  We also
do not broadcast to the world that FreeBSD CURRENT is usable. I've seen a lot
of people who stay away from CURRENT based on their 20-year old experience with
it.

Here is how we are going to improve:

* Last week of a month is declared a stabilization week

Src committers are encouraged to avoid pushing risky changes to FreeBSD/main
during this week.  This is an advice, not a policy!  If a committer breaks
something during the week they got 3x public shame, but no administrative
penalties or fines.  Committers are encouraged to push bug fixes, improve unit
tests, clean up comments and improve documentation.  It is a also a good time
to do merging of past work to stable branches. Developers of course will
continue their work on bigger projects in their private branches.

Sidenote: there is no agreement in the world what is "the last week of a
month".  For our purposes we will use the week that contains the last Friday of
the month.  Because we want the monthly snapshot to be called by the name of
the month (not next month) and thus we want the last day of the stabilization
cycle always to be in that month.

* Monday of the stabweek is the day to update your CURRENT and test it

Monday 8:00 GMT a tag is created and published.  Right now it is published
at my personal https://github.com/glebius/FreeBSD/tags.  Note that the tag
points at a hash in the official repo, so there is no trust involved here.

At Netflix I will be working on merging the tagged revision into our tree and I
will hand off the resulting branch to our excellent testing team (dhw@ +
olivier@) usually by the end of Monday (PST time).  Other companies and parties
are encouraged to start testing the tagged revision.  Peter Holm may switch his
stress2 to run that revision.  You are encouraged to update your desktop or
laptop that of course runs FreeBSD CURRENT.

* A short lived stabilization branch may be created

In case we discover regressions compared to the previous month stabweek, bug
fixes to them will be committed to a short lived branch.  This branch may
contain direct cherry-picks from main, as well as work-in-progress bugfixes
that had not yet been committed to main, reverts of commits and even stop gaps
that disable certain functionality for the sake of stability.  This branch may
be rebased and force pushed if a temporary bugfix appears different to a final
one in main.  The branch may observe commits immediately Monday morning in case
we already know about a certain regression.  The branch will not observe
commits to a long standing bugs that were fixed in main during the stabweek,
unless somebody explicitly asks to include one.  And finally, the branch may
not even be created in case testing confirms everything is alright with the
Monday tag.

The branch will be published at https://github.com/glebius/FreeBSD.  There is
certain level of trust required to use it. That may change to a more official
publishing point in the future.

* The stabweek quiet period ends no later than Friday 18:00 GMT

No matter if we were able to identify and fix any or all bugs the quiet period
ends.  The public shame level for src committers breaking FreeBSD CURRENT goes
back to normal level.  In a case we were not able to address all issues by end
of Friday the stabweek branch will be active past the end of the stabweek, as
we want to collect all regression fixes in the branch.  But this is the worst
case scenario!

A more appreciated scenario is that the stabilization period ends earlier in
the week. If all testing parties report their satisfaction with state of main
as is or of the stabweek branch and if I don't see any fresh bug reports in
bugzilla or submissions via other channels, there is no reason to withheld
committers with pushing their stuff.

At the end of the stabilization period be it Friday or earlier I will write
email to current@ reporting the results:

- were there any regression identified with the Monday tag
- what has been accumulated in the stabweek branch
- known stable point(s) of FreeBSD/main during the period, recommended for use

The free riders who did not participate in the testing are now welcome to
update their machines to published stable points :) More seriously speaking, I
actually hope that in some future snapshots.FreeBSD.org will start using these
points for snapshot generation.

P.S. The February 2024 stabilization week was run internally, without sharing
publicly and in a few minutes I will post its results in a separate email.

-- 
Gleb Smirnoff