patching a production system

[Ryan Thompson]

Chaos Golubitsky wrote to freebsd-questions at freebsd.org:

> Hi all,

Hello,

> [...]
> real experience with FreeBSD.)  I want to be able to keep the box
> reasonably current on security patches to the os, so it seems to me
> that i should be tracking freebsd-RELEASE.

Specifically, you probably want to track one of the RELENG_4_x
branches. If you want to move to 4.8, use RELENG_4_8 as your CVS tag.

> My question is in two parts:
>
> (a) (I think the answer is no, but would love to hear otherwise):
>     Do i have an alternative to maintaining a source tree on this
>     machine?  The release engineering notes:
>     http://www.freebsd.org/doc/en_US.ISO8859-1/articles/releng/article.html
>     mention binary patchkits for the release branch, but i don't
>     think these actually exist.

Assuming you're running on i386 hardware, and staying current, binary
patches are released for most security advisories. For more
information, look at the advisories themselves, which will direct you
to excellent information on how they may be applied. This usually
includes binary updating instructions. Binary upgrades are fine if you
don't use custom compile flags.

Past advisories can be found here: http://www.freebsd.org/security/

>     Does anyone know?  Conversely, how
>     easy is it to do updates using /stand/sysinstall without changing
>     my system configuration more than needed?  The buildworld ->
>     installworld -> mergemaster routine seems convenient and stable,
>     but i don't like doing source compiles on a production machine,
>     and we don't have budget for a spare with similar architecture.

For a production machine, I'd recommend the buildworld process, with
copious backups, scheduled maintenance window, and a test run on a
different machine. (Even if it's *not* a similar architecture. 99% of
the battle is getting things right with mergemaster... and you can
test *that* on an old Pentium :-)

> (b) Specifically, the machine is currently running 4.6-RELEASE, and
>     i thought i would upgrade it to 4.8-RELEASE and track that,
>     since FreeBSD will test its security patches for longer (right?),

Right.

http://www.freebsd.org/security/#adv


>     so i won't have to upgrade again for awhile.  The machine was
>     originally installed using /stand/sysinstall, and not by me.
>     I have tested out the sysinstall -> cvs upgrade -> build ->
>     install process on a spare machine of my own, and haven't run
>     into any difficult problems.

Good! Document your progress. Try it again by mirroring your
production machine's configuration (and OS build, which you can still
download and install, if necessary), and make sure you can do that
smoothly. An extra hour or two to do this is *well* worth the effort,
if it reduces the risk of serious downtime and pissed customers or
lost productivity.

>     Can i expect this upgrade to go smoothly?

It depends how skilled you are. :-) Honestly, many people on this list
have done this many times, without much grief. Going between minor
point releases (4.6 -> 4.7) is a bit tricker, because new features get
added to the system between releases that you'll have to contend with.
(And, typically, mergemaster comes up with many more diffs).

In your case, since your testing facilities are limited, I'd probably
do 4.6 -> 4.7 first, and test that profusely. Do the upgrade to 4.8 at
a later date. This isn't *necessary*, but you may find it helpful.

Oh, and, as the instructions say, read /usr/src/UPDATING . It will
hopefully contain most of the "gotchas" that you will be expected to
know about while doing the upgrade.


>     The machine is running
>     a lot of third-party software, which i am not going to touch.

Beware statically linked binaries in your 3rd party software. Our
binary format hasn't changed, but you will run into snags with updates
to system libraries if they've been statically linked.

>     Are there any particular red flags i should look for in terms
>     of either (1) going from a sysinstall install to a source
>     install, or (2) going from 4.6-RELEASE to 4.8-RELEASE?  Basically,
>     i'm looking for things i can do to make it more likely that the
>     install will just work (tm).

A source install is still probably the way to go, as it allows you the
greatest range of customization and control while you upgrade your
system. In terms of production machines, you're making a relatively
large move, so you want to proceed carefully. (Nix that. You *always*
want to proceed carefully :-). Once you're tracking a more current
release (RELENG_4_8), the updates will be comparatively small. In many
cases, you can apply the updates in binary form, or recompile only
specific bits of the OS (e.g., sendmail), without even rebooting. It
is not usually necessary to rebuild the entire OS.

Pearls of wisdom (not all about FreeBSD):

Give yourself lots of time. cvsup/buildworld anytime. Schedule a
maintenance window for mergemaster and installworld in the wee hours,
if possible. Schedule 4X the amount of time you'll need, and then
happily report you did it in a fraction of the time. Do not respond
(have someone else respond) to customer/internal reports during the
outage. Don't rush; chances are you'll just bugger something up and
waste *more* time. Document everything.

> Sorry this question is so long

:-)

Hope this helps,

- Ryan

-- 
  Ryan Thompson <ryan at sasknow.com>

  SaskNow Technologies - http://www.sasknow.com
  901-1st Avenue North - Saskatoon, SK - S7K 1Y4

        Tel: 306-664-3600   Fax: 306-244-7037   Saskatoon
  Toll-Free: 877-727-5669     (877-SASKNOW)     North America