Release engineering confusion

[Ted Mittelstaedt]

>-----Original Message-----
>From: owner-freebsd-questions at freebsd.org
>[mailto:owner-freebsd-questions at freebsd.org]On Behalf Of Steve Bertrand
>Sent: Wednesday, November 16, 2005 4:52 PM
>To: 'David Kirchner'
>Cc: 'FreeBSD Questions'
>Subject: RE: Release engineering confusion
>
>
>
>
>> -----Original Message-----
>> From: dpkirchner at gmail.com [mailto:dpkirchner at gmail.com] On
>> Behalf Of David Kirchner
>> Sent: Wednesday, November 16, 2005 7:45 PM
>> To: Steve Bertrand
>> Cc: RW; FreeBSD Questions
>> Subject: Re: Release engineering confusion
>>
>> On 11/16/05, Steve Bertrand <iaccounts at ibctech.ca> wrote:
>> > Thank you. However, that entire page out of the handbook
>> pretty much
>> > clarifies that a production environment should *not* track either
>> > STABLE or CURRENT.
>> >
>> > So I'm assuming I'm best off with RELENG_6_0 etc, etc? Does anyone
>> > here actually run STABLE or CURRENT in a production
>> environment? I've
>> > personally had the most luck with RELENG_4 which is still
>> my main box,
>> > but now my curiosity has got the best of me.
>> >
>> > Steve
>>
>> Ultimately it depends on how much downtime and difficulty
>> you're willing to endure, just in case the -STABLE branch
>> ends up not working for your servers for some particular
>> reason. We use -RELEASE almost exclusively (we have one
>> -STABLE machine, because we needed a newer version of a
>> kernel driver) as we manage hundreds of servers, and there's
>> no one -STABLE release (to properly describe the -STABLE
>> version you're using you have to have the date and time of
>> the cvsup, as opposed to -RELEASE versions being like
>> 5.4-RELEASE-p9). It's easier, and thus more reliable, for us
>> to have stable(heh) version strings.
>
>I can appreciate the fact it's easier to follow one string for so many
>servers.
>
>> If you're just working with a handful of servers, -STABLE
>> would probably be fine, as long as you have backups and know
>> how to revert to previous versions when it becomes necessary.
>
>I do only have a handful of servers, however thousands of users, and
>indeed, I do have backups.

Hi Steve,

  We have the same thing - I NEVER upgrade a live server.  I ALWAYS
build a brand new install on a spare or a new box then copy over the
data in one fell swoop and swap IP addresses between the new
server and the old one.

  With the cost of server hardware today it's very compelling to buy
new anyhow.  I have a customer I'm building a mailserver today for
who just bought a brand new rack mounted clone, Intel brand motherboard,
with
daul 200GB mirrored SATA drives and 1 gig of ECC ram for about
$1200 bucks.  I'm running FBSD 6.0 on it and I've run a passle of MySQL
test suites on it and it kicks the shit out of my 1 year old servers that
cost 3 times that.  And he's not even running 10,000 rpm drives on it.

>The problem arises in a criticality that >20
>minutes of downtime would lead to a severe problem....which brings up
>another good question...how do YOU revert back to a previous release? If
>you manage so many servers, I'd love to know what type of routine you'd
>use to revert back (and so would many others I'd think ;)
>

Very scary stuff.

By doing the server swap stuff I do I am able to run extensive tests on
the new server, new OS and new applications, BEFORE bringing it
online.  The old server it replaces goes into quarentine and isn't
touched
until a month has elapsed, just in case we need to revert back to it
in a hurry.

  I never move to a new release until after extensively
testing it AND the applications I've built on it.

  I know a lot of
work seems to have gone into making FBSD upgradable -on-the-fly-
but the applications we run on it are the important thing and the
developers of those apps are often rather lackadasical about
updates.  Particularly in the case of Perl modules - many of our
apps have lots of dependencies on many modules and I'd say
3/4 of the Perl modules in the ports trees build with dozens of
compiler complaints about miscasting, pointers into ints, and
that sort of thing.  There seems to be a lot of those programmers
that either ignore portability issues, or make assumptions that
it's going to be run on x86 stuff, or just plain don't know how
to code.

Ted