patching a production system

Bill Moran wmoran at
Sun Apr 20 10:40:51 PDT 2003

Ryan Thompson said most of what I wanted to say, so I'm just going to
add a few things here and there.

Chaos Golubitsky wrote:
> Hi all,
> This is an advice question, so i hope this list is the right place
> to ask: i am tasked with maintaining a FreeBSD box which is a server
> for a very small company.  (I am a sysadmin, but this is my first
> real experience with FreeBSD.)  I want to be able to keep the box
> reasonably current on security patches to the os, so it seems to
> me that i should be tracking freebsd-RELEASE.

There's no -RELEASE to track.  RELEASE is a snapshot, and it never
changes once it's been released.  What you want to track are the
patches for the particular -RELEASE you're using.

> My question is in two parts:
> (a) (I think the answer is no, but would love to hear otherwise):
>     Do i have an alternative to maintaining a source tree on this
>     machine?  The release engineering notes:
>     mention binary patchkits for the release branch, but i don't
>     think these actually exist.  Does anyone know?  Conversely, how
>     easy is it to do updates using /stand/sysinstall without changing
>     my system configuration more than needed?  The buildworld ->
>     installworld -> mergemaster routine seems convenient and stable,
>     but i don't like doing source compiles on a production machine,
>     and we don't have budget for a spare with similar architecture.

I recommend the build->install->mergemaster process.  I've been using
it for years on production machines and never gotten bitten.  A few
hints on "not getting bitten"
1) Subscribe to -STABLE and lurk.  You don't have to read every message,
    but watch for HEADSUPs and subjects that appear to pertain to you.
2) Always read /usr/src/UPDATING before starting anything
3) Always do a 'cp -Rp /etc /etc.old' before starting mergemaster.
    Although mergemaster works very well, it's easy to make a mistake
    (especially when you're new to it).
4) Follow the handbook to the letter
5) If, at step 21.4.8 in the handbook (where you've got a new kernel
    installed, but not a new world yet) you can't boot the machine
    because the new kernel has problems, you can boot kernel.old
    and then copy kernel.old back to kernel and /modules.old back to
    /modules and you've effectively reverted to the pre-installkernel
    state of the machine.  (this failsafe should really be in the
    handbook ... I should really generate some diffs ...)

After the first or second time you've done the procedure, it gets
pretty routine.

I understand your concern about not wanting to build on production
machines, but there are a few things to know:
1) If you're tracking the patch tree (i.e. RELENG_4_8 for example)
    there is very little chance of anything running afoul.
2) I've run builworld/buildkernel on production systems while they
    were in use without the users even knowing I was doing it, then
    run the installworld/installkernel/mergemaster after hours (only
    takes 15-30 minutes in the evening that way)  I don't know if I'd
    recommend running buildworld/buildkernel during business hours
    (unless it's a very overpowered machine) but the point is: you
    can get away with it.
3) I've also done upgrades remotely via ssh (off hours) with no
    problems.  This isn't recommended either, but you can get away
    with it 99% of the time.

> (b) Specifically, the machine is currently running 4.6-RELEASE, and
>     i thought i would upgrade it to 4.8-RELEASE and track that,
>     since FreeBSD will test its security patches for longer (right?),
>     so i won't have to upgrade again for awhile.  The machine was
>     originally installed using /stand/sysinstall, and not by me.
>     I have tested out the sysinstall -> cvs upgrade -> build ->
>     install process on a spare machine of my own, and haven't run
>     into any difficult problems.

Depending on the environment:
1) If the machine is unused on weekends, you can easily schedule upgrades
    as needed.  I recommend scheduling an entire weekend (probably won't
    take that long) to get it up to date with the latest RELENG_4_8 and
    update that as patches are released.  At some point you're going to
    want to move to RELENG_5, but that's a bit in the future and will
    present new and different problems.
2) If the machine is used 24/7, you need to find somewhere to get some
    downtime and keep it relatively current.  Plan for sometime in the
    as-near-as-possible future to upgrade to the latest RELENG_4_6 and
    plan on staying there until you can schedule a little more time to
    bring it up to RELENG_4_8.

>     Can i expect this upgrade to go smoothly?


>  The machine is running
>     a lot of third-party software, which i am not going to touch.

I think that's a BIG mistake.  You need to make a list of third-party
apps that the machine uses and put a plan in motion to get them updated
as well.  A fully-updated FreeBSD machine is little help if you have
an app with a known security hole or memory leak that keeps bringing it

Updating third-part apps is generally much harder than updating FreeBSD
itself as there are more things that can go wrong (statically linked
programs, dependencies ... etc)
Here's my suggestion.
1) Always install from ports
2) Use 'make package' to have a package available for everything you
    install.  That way, if an upgrade fails, you can reinstall the older,
    working version until you figure out what went wrong.
3) Backup /usr/local/etc and any other directories that your apps might
    use before upgrading them!
4) Subscribe to bugtraq and any other relevent security mailing lists
    and make note when things need updated.  Schedule and plan as problems

Unfortunately, you don't have the liberty of doing 1 and 2 on software
that somebody else already installed, so this first upgrade may be tricky.
Do lots of research, and testing on another machine to keep it from going
to hell.

>     Are there any particular red flags i should look for in terms
>     of either (1) going from a sysinstall install to a source
>     install,

Not at all.

> or (2) going from 4.6-RELEASE to 4.8-RELEASE?

They'll be in /usr/src/UPDATING.

>  Basically,
>     i'm looking for things i can do to make it more likely that the
>     install will just work (tm).

Up front research always helps!

I think you'll be fine!  Hope I've helped.

Bill Moran
Potential Technologies

More information about the freebsd-questions mailing list