patching a production system
wmoran at potentialtech.com
Sun Apr 20 10:40:51 PDT 2003
Ryan Thompson said most of what I wanted to say, so I'm just going to
add a few things here and there.
Chaos Golubitsky wrote:
> Hi all,
> This is an advice question, so i hope this list is the right place
> to ask: i am tasked with maintaining a FreeBSD box which is a server
> for a very small company. (I am a sysadmin, but this is my first
> real experience with FreeBSD.) I want to be able to keep the box
> reasonably current on security patches to the os, so it seems to
> me that i should be tracking freebsd-RELEASE.
There's no -RELEASE to track. RELEASE is a snapshot, and it never
changes once it's been released. What you want to track are the
patches for the particular -RELEASE you're using.
> My question is in two parts:
> (a) (I think the answer is no, but would love to hear otherwise):
> Do i have an alternative to maintaining a source tree on this
> machine? The release engineering notes:
> mention binary patchkits for the release branch, but i don't
> think these actually exist. Does anyone know? Conversely, how
> easy is it to do updates using /stand/sysinstall without changing
> my system configuration more than needed? The buildworld ->
> installworld -> mergemaster routine seems convenient and stable,
> but i don't like doing source compiles on a production machine,
> and we don't have budget for a spare with similar architecture.
I recommend the build->install->mergemaster process. I've been using
it for years on production machines and never gotten bitten. A few
hints on "not getting bitten"
1) Subscribe to -STABLE and lurk. You don't have to read every message,
but watch for HEADSUPs and subjects that appear to pertain to you.
2) Always read /usr/src/UPDATING before starting anything
3) Always do a 'cp -Rp /etc /etc.old' before starting mergemaster.
Although mergemaster works very well, it's easy to make a mistake
(especially when you're new to it).
4) Follow the handbook to the letter
5) If, at step 21.4.8 in the handbook (where you've got a new kernel
installed, but not a new world yet) you can't boot the machine
because the new kernel has problems, you can boot kernel.old
and then copy kernel.old back to kernel and /modules.old back to
/modules and you've effectively reverted to the pre-installkernel
state of the machine. (this failsafe should really be in the
handbook ... I should really generate some diffs ...)
After the first or second time you've done the procedure, it gets
I understand your concern about not wanting to build on production
machines, but there are a few things to know:
1) If you're tracking the patch tree (i.e. RELENG_4_8 for example)
there is very little chance of anything running afoul.
2) I've run builworld/buildkernel on production systems while they
were in use without the users even knowing I was doing it, then
run the installworld/installkernel/mergemaster after hours (only
takes 15-30 minutes in the evening that way) I don't know if I'd
recommend running buildworld/buildkernel during business hours
(unless it's a very overpowered machine) but the point is: you
can get away with it.
3) I've also done upgrades remotely via ssh (off hours) with no
problems. This isn't recommended either, but you can get away
with it 99% of the time.
> (b) Specifically, the machine is currently running 4.6-RELEASE, and
> i thought i would upgrade it to 4.8-RELEASE and track that,
> since FreeBSD will test its security patches for longer (right?),
> so i won't have to upgrade again for awhile. The machine was
> originally installed using /stand/sysinstall, and not by me.
> I have tested out the sysinstall -> cvs upgrade -> build ->
> install process on a spare machine of my own, and haven't run
> into any difficult problems.
Depending on the environment:
1) If the machine is unused on weekends, you can easily schedule upgrades
as needed. I recommend scheduling an entire weekend (probably won't
take that long) to get it up to date with the latest RELENG_4_8 and
update that as patches are released. At some point you're going to
want to move to RELENG_5, but that's a bit in the future and will
present new and different problems.
2) If the machine is used 24/7, you need to find somewhere to get some
downtime and keep it relatively current. Plan for sometime in the
as-near-as-possible future to upgrade to the latest RELENG_4_6 and
plan on staying there until you can schedule a little more time to
bring it up to RELENG_4_8.
> Can i expect this upgrade to go smoothly?
> The machine is running
> a lot of third-party software, which i am not going to touch.
I think that's a BIG mistake. You need to make a list of third-party
apps that the machine uses and put a plan in motion to get them updated
as well. A fully-updated FreeBSD machine is little help if you have
an app with a known security hole or memory leak that keeps bringing it
Updating third-part apps is generally much harder than updating FreeBSD
itself as there are more things that can go wrong (statically linked
programs, dependencies ... etc)
Here's my suggestion.
1) Always install from ports
2) Use 'make package' to have a package available for everything you
install. That way, if an upgrade fails, you can reinstall the older,
working version until you figure out what went wrong.
3) Backup /usr/local/etc and any other directories that your apps might
use before upgrading them!
4) Subscribe to bugtraq and any other relevent security mailing lists
and make note when things need updated. Schedule and plan as problems
Unfortunately, you don't have the liberty of doing 1 and 2 on software
that somebody else already installed, so this first upgrade may be tricky.
Do lots of research, and testing on another machine to keep it from going
> Are there any particular red flags i should look for in terms
> of either (1) going from a sysinstall install to a source
Not at all.
> or (2) going from 4.6-RELEASE to 4.8-RELEASE?
They'll be in /usr/src/UPDATING.
> i'm looking for things i can do to make it more likely that the
> install will just work (tm).
Up front research always helps!
I think you'll be fine! Hope I've helped.
More information about the freebsd-questions