Avoiding source code on production servers (Neil Neely)

[Steve Scally]

>
> Tonix (Antonio Nati) wrote:
>> I'm in the phase of planning my new generation of FreeBSD servers,  
>> and
>> I would love to make them more easy to upgrade.
>> Main problem I have currently is I do not want any source code on
>> production server, so freebsd-update is welcome, but... what about
>> packages?
>> I would use packages, but they are not easy to upgrade, while ports
>> can be easy to upgrade, but need to have sources an servers.
> The weakness of FreeBSD here is very unfortunate and IMO goes far  
> beyond
> just source vs binary distribution.  Working in a mixed environment
> where we have begun heavily using CentOS and utilizing yum it's  
> obvious
> how far behind FreeBSD has fallen in this space.  Ports lack any  
> kind of
> concept of  "Long Term Stable", so if you are running anything in  
> ports
> (like say perl...) then when a security issue comes out you end up
> having to install new versions - the default is not to patch the older
> versions.  For non-production environments that is likely fine, but  
> for
> major production services it is a painful scenario.  So you aren't  
> just
> fixing security you are mixing in the concept of adjusting  
> functionality
> as well.

>
> (A recent perl "security" upgrade moved perl to a new version which
> broke compatibility with the Crypt::CBC module requiring a reinstall -
> the new version of that from ports forced salting when it hadn't
> previously and now applications were needing to be recoded to get it  
> all
> working again.)

This seems more like a flaw in the upgrade procedure you have setup  
and not testing before upgrading.
I would assume, being this is the ISP list,  you have multiple  
redundant type boxes of which
one can be taken out of production and upgraded and if it fails will  
not disrupt service.  One
setup, which I can think of, that can't really be tested is if your  
hosting sites for customers in that
case it would be a bit hard to setup a mirrored test environment for  
but nothing is impossible.
>
> At the end of the day FreeBSD of course lets you have all the power to
> just apply the patches yourself to the source and you would be  
> fine.  At
> the cost that you need to be doing all of this work yourself and can't
> rely on nice management tools to help you.  Every problem I've ever
> encountered with FreeBSD can be easily handled by a FreeBSD expert -  
> but
> when I bring in a new green admin they have a heck of a time making  
> any
> sense of it and I'm drug back into the trenches of managing all this.

I am coming from the opposite end, I used to do CentOS work and now  
work on FreeBSD and
after getting over the initial learning curve and methodology of  
FreeBSD I find it much
better than CentOS.  It is much easier to screw up a system with a yum  
upgrade, wrong priority, or a
package from the centos-plus repo.  Also linux in general doesn't have  
a centralized place to read pr
notices or an automated portaudit system.  For me I had to follow a  
security focus list or CERT list to see
which packages were vulnerable then go to the maintainers and get  
information.  Then I had to wait for
CentOS to push out an update and that to me is a maintenance nightmare.

The only way to get your new admin up to speed is to let them  
experience the problem and solve it.
Internal documentation also helps immensely or a wiki.  Without the  
network specific information available
it doesn't matter what OS you use there will be a huge hurtle.  In  
fact if you don't already have this setup
why not let your junior guys work on it.  In our wiki we have all our  
procedures for backups, nightly processes,
system flows, mail paths, port options, etc.  We also have pages  
listing all issues or steps needed for upgrades
of all our specific server types, mail, dns, management and so on.
>
>
> Why the contrast is extra frustrating is that it takes considerable
> skill and understanding of the details of an environment to safely
> update a production FreeBSD server.

Internal documentation would help here as well as a repo for all your  
one-off configs then multiple changes can
be tracked.

> Contrast this with CentOS where an
> extremely green admin can easily manage it with minimal instruction.
> Unlike with the FreeBSD process this has no risk that it will cause
> cascading complex issues that require application modification to
> restore them to operation.
>

Yum can be be just as finicky and destructive if not setup or executed  
correctly.

Just my thoughts.

> I've been using FreeBSD since the 2.x days in '96 or so, and I love  
> it -
> my tone is critical because I'm sad to see the state of things and
> doubly sad that I don't have the time to volunteer with the project to
> help do something about it.  In most ways I consider FreeBSD  
> superior to
> any linux, however this core issue of maintenance over time has been
> driving our shift to using CentOS over the last few years.  If a "Long
> Term Stable Port Tree" concept were to come along I think that would
> plug the hole here.  While I lack the time to lead such a charge, I
> would be happy to assist if such an effort were to get launched.
>
> -- 
> Neil Neely
> http://neil-neely.blogspot.com/