DRAFT - Release Guide

[Ken Smith]

Here is a draft of a Release Guide which would take most of the
hassles out of Release Day I *think*.  I want to say up front that if
you read this it sounds like it's a list of demands being made by the
Mirror Folks to the Release folks but that is NOT its intention.  I
wrote it in the context of it being exactly what I called it - a
Release Guide to be read/followed while doing a new release.  But the
assumption is this will initially be presented to all parties for
*discussion*, and the end result is concensus amongst everyone
involved.  I also want to say we all know there is way more stuff
involved in a Release than just this stuff.

I hope this is at least in general terms what people were expecting.
If you have minor edits (my spelin wuz rong ur me grammer wuznt quit
rite) could you just send it straight to me please?  If you have any
thoughts that you think should be discussed, or if there are major
omissions, or if you think this whole thing is just one big mistake
feel free to send to the list but again please don't just include
this entire message in the reply - edit out anything that is
irrelevant please.

I'll wear asbestos undies for the next couple of days, flame away.
If in the end we hash this thing out and decide that it's worth doing
something with I'll take it to re@ and portmgr@ since it effects how
both of them operate, stressing at first that it's for *discussion*.

----------------------------------------


			Release Guide Rev. 0.0
			======================

Introduction
------------

In the early years FreeBSD Distributions were, compared to now,
relatively simple to handle.  The data got copied to ftp.freebsd.org
and everyone got it from there.  Due to the poplarity of FreeBSD that
simple a distribution channel is no longer practical.  The mirror
system that now supports the distribution of FreeBSD has become a
multi-tiered worldwide set of machines, and even in this state some of
the more central distribution machines are under fairly heavy stress
during Release Time.

Because the Mirror System has needed to grow this large and because of
certain aspects of the data in the FTP site there are limitations on
what can be expected of the Mirror System.  In particular there is now
a significant amount of time between when data gets placed on the main
distribution server (ftp-master.freebsd.org) and when it can be
expected to be available from the majority of the mirror servers.
Since it is highly desirable for the release files to be on the
majority of the mirror servers at the point the Release is announced
this effects the schedule for when the data should be placed on
ftp-master.

Terminology
-----------

ftp-master: the master distribution server, data is initially loaded
here

Tier-1 server: official FreeBSD mirror site that pulls data from
ftp-master

Tier-2 server: official FreeBSD mirror site that pulls data from a
Tier-1 server.  Not all Tier-1 servers feed Tier-2 servers (we would
like to set up more that do).

sync: what happens when a mirror server pulls new data from its
up-stream site. Different mechanisms can be used to do this (cvsup,
rsync, mirror, omi, etc).  Only newly updated files will be
transferred through the network BUT typically a large number of files
need to be checked to see if they should be transferred. Depending
on how much of the FreeBSD FTP site is being checked (only checking
sub-sections is possible) this could cause a significant load on both
the client and server involved.

staging the release: placing new files onto ftp-master but the
permissions on the files are set such that anonymous users can not
download the files.  This would be done so that there is time for the
files to be transferred to other mirror sites before a Release Date
but users can't get the files until the permissions get changed.
Under normal circumstances files are simply "posted" to ftp-master and
downloadable by anyone once they get to the Mirror Servers.

Mirror System Operations
------------------------


Most though not all of the mirror sites operate the same way.  Some
time in the wee hours of the morning a cron job starts up what syncs
the FTP site.  FreeBSD may be the only thing this mirror site carries,
or it may carry other distributions (OpenBSD, Linux's, etc).  The sync
may be set up to check through the entire FreeBSD site but it may just
check through some of it, doing other parts some other time (in
particular the ports section is getting a bit large and may be handled
as a separate thing - perhaps only a few days a week).

Because of this it is up to a day before a Tier-1 system will begin
transferring new data from ftp-master.  If there is a lot of new data,
and in particular a new set of packages, the sync may take as long as
six to eight hours between systems that are relatively unloaded and
have a decent network connection between them.  So taking worst case
scenarios into account it can be three days before large amounts of
data can propagate from ftp-master to a Tier-2 site.  This situation
gets worse if there is something causing a larger than normal network
load at the time.

Handling Betas/Release Candidates
---------------------------------

Users' expectations and general anxieties are smaller for the Betas
and Release Candidates so timing of these is a little less critical.
They seem to "naturally" appear a week or more between Beta's and RC's
which is reasonable as far as propagation goes.  The important thing
for the Release Engineering Team to keep in mind is the propagation
delay of two to three days for large amounts of data to make it
through to the majority of Tier-2 sites.  What is critical for the
Mirror Sites is that there be a day or two worth of advanced notice
(weekdays) that the Beta is coming AND a reasonable estimate of how
much disk space will be needed.  The Mirror sites that carry many
distributions tend to keep the partition(s) dedicated to mirrored data
fairly close to being full (maximizing how much potentially useful
stuff is available to their users).  If large amounts of new stuff is
about to arrive some of the older stuff needs to be removed.  Other
sites can't afford the disk space to keep the entire FreeBSD FTP site
and may need to adjust the sync scripts so they will not pull the
Betas.  The size estimates should be given for both the -RELEASE tree
and the new package trees.

It is most economical to the Mirror Sites if Beta-1 can be removed at
the point Beta-2 is uploaded to ftp-master, and likewise for the
RC(s).  If that is possible size estimates for the follow-up Betas and
RC's is not as critical because the change in disk space usage is
close enough to zero - the size estimates are mostly needed to make
sure the mirror sites' disk partitions don't get full.  Similarly at
the point the Release is being staged if possible remove the RC.  If
the Release has been finalized and is ready to be staged in the long
run it is a dis-service for users to continue having access to the
non-Release files (odds seem small someone would download an RC at
this point and find/report a bug that would stop the release).  If
Beta-1 needs to co-exist with Beta-2 and so on then size estimates
will again be needed a day or two before Beta-2 gets posted on
ftp-master. 

Staging the Release
-------------------

It would be best if the Packages could be loaded first, on the order
of a week before the target Release Date.  Sync-ing the ports
directory is the worst part of the FTP site to deal with.  It can at
times wind up involving several sync attempts to get it all.  The
permissions on the files should not need any special attention - there
seems to be no point to stopping users from being able to download
them.  Until Release Day they are basically useless so nobody is
likely to try and get them.  It is actually best if the -RELEASE stuff
NOT appear until later because some end-users of Mirror sites will
start to pester Mirror Operators as soon as something like an official
Release directory structure shows up.

The -RELEASE stuff should be posted to ftp-master three days before
Release Day, with the permissions set so that access by "other" is
turned off.  Mirror sites are encouraged to run the sync as a
different user than "ftp" so this will load the files onto the Mirror
sites but they will not be downloadable until permissions are changed.

Based on past history not all the packages for the different
architectures are available at the same time, not to mention the
-RELEASE trees.  It is important that at least the i386 packages be
available very early because of the relative popularity of that
architecture.  If packages for a given architecture are not ready by
three days before Release Day it may be best to not post it to
ftp-master until after Release Day.  That way it will not compete for
bandwidth with the -RELEASE trees being staged and it is probably most
important to get as much of all the -RELEASE trees out as possible.

Release Day
-----------

Mail should be sent to hubs at freebsd.org a few days before Release Day
giving the date and time that the Release Announcement will be sent
out.  At this time there is no "push" mechanism in the Mirror System
so the opening up of the permissions needs to be either manual or it
will happen at the time of the next automated sync.  The tree on
ftp-master should have its permissions changed and hopefully the
Tier-1 sites will take care of the permissions at or close to the
announcement time so down-stream Tier-2 sites can pick up the
permission change fast.

Consequences
------------

What happens if the guidelines are not followed?  If the suggestion
about not announcing disk space requirements when there is a large
change some sites' disk partitions will fill, at which point the
FreeBSD sync fails and any others that were supposed to happen
afterwards also fail.  This delays the propagation of the data.  If
the packages are not available on the mirror sites because of delays
in getting them posted FTP-based installs users attempt before the
packages become fully available may not complete correctly.  Any data
that gets posted to ftp-master after the Release Date will take a
dramatically longer time to propagate to the Tier-2 servers.  During
Release times ftp-master becomes much more heavily loaded (slower).
In addition to that the users are clogging the network pipes of the
downstream sites so the syncs from these sites need to compete with
all that network traffic.

----------------------------------------
-- 
						Ken Smith
- From there to here, from here to      |       kensmith at cse.buffalo.edu
  there, funny things are everywhere.   |
                      - Theodore Geisel |