FreeBSD has serious problems with focus, longevity, and lifecycle

Wed Jan 18 00:18:08 UTC 2012

> -----Original Message-----
> From: owner-freebsd-hackers at freebsd.org [mailto:owner-freebsd-
> hackers at freebsd.org] On Behalf Of Julian Elischer
> Sent: Tuesday, January 17, 2012 10:56 AM
> To: Mark Felder
> Cc: freebsd-hackers at freebsd.org
> Subject: Re: FreeBSD has serious problems with focus, longevity, and lifecycle
> 
> On 1/17/12 7:39 AM, Mark Felder wrote:
> > Why is everyone so afraid of running -STABLE? Plenty of stuff gets
> > MFC'd. Yeah, I agree -- running -RELEASE is difficult. Hell, it's
> > frustrating to us that VMWare only supports -RELEASE and it took until
> > ESX 5 to officially support 8.2!
> >
> > More releases / snapshots of -STABLE helps people on physical servers,
> > but anyone who runs VMs on Xen or VMWare won't get any support for
> > those versions because they didn't go through the QA process yet.
> > FreeBSD is increasingly becoming a third world citizen thanks to
> > virtualization efforts being focused on Linux, so I feel that more
> > frequent releases won't help as many people as you think.
> 
> I'm going to go both ways on this one.
> 
> Where I used to work (Devin Teske is now there)  we used to use the 'stable'
> branch and rolll our own releases.

We still do this today, but have only further enhanced the procedure.

Within FreeBSD announcing a new release, we can be ready to ship said-release within 3-6 weeks.

However, that's not to say that our customers can take said-new-release every 3-6 weeks. Our largest customer is not-surprisingly the fastest at turn-around but at-best could only do maybe 2 releases at-most in a single year. Our smaller customers are taking OS upgrades much slower (every-other year if we're lucky; more like once every 3 years).

For both our large and small customers, we actively back-port device support in-accordance with hardware manufacturing windows.

This is to explain that our hardware suppliers notify us ahead-of time when a particular piece of hardware is about to become unavailable. At that time we are usually given a choice of hardware to evaluate as replacements for the existing EOM production-line.

We're usually given at least 3-9 months prior-notice before our current production-line goes EOL.

For each candidate replacement we try each FreeBSD release that we're currently using in production. If it doesn't work, we try another candidate hardware. If we can't seem to find a winning combo that works with our existing -RELEASE, we then start trying newer releases until we find drivers working for each/every required piece of hardware (network cards, RAID cards, serial ports/cards, Adaptec SCSI cards, Fibre HBAs, etc.). When we find a release that contains the necessary drivers, we back-port.

At this time, it's worth noting that we use ONLY monolithic kernels and we deliver them via packages. When we back-port new device support, we just roll a new kernel, ship it via package, pkg_add, reboot.

Similarly, if the patch is in userland, we package up the replaced binaries (produced by using the release engineering procedures documented in build(7) and [more importantly] release(7)).

The net effect is that we run a -RELEASE with patches from either the same -STABLE, a higher -RELEASE, higher -STABLE, or (as a last-resort) -CURRENT or HEAD. We've been known to roll FreeBSD-4.11 kernels with bits back-ported from RELENG_8.

So, I guess what I’m trying to say is that if you're going to take your production environment extremely seriously (as though 1.5-Trillion globally-economic-dollars depended on it) you-too would take a serious look at release(7) and the Release Engineer's handbook.

It really is worth maintaining your own release, taking required fixes from -STABLE (preferred) or higher (as necessary) to satisfy your customers (which are nearly almost always going to have a different release schedule than that-of any OS, be-it FreeBSD, Linux, or other distribution).

> the criticality of those systems was hard to over-emphasize. In 2005 we worked
> out we processed 1.5 trillion dollars of transactions on those systems.
> 

I'm going to have to sit down with DaveR, JPL, and others to get an updated metric for 2011. I'd be willing to bet that we've increased transactional load over the years (with respect to FreeBSD, we brought on one new sizable customer since then and expanded the scope of existing FreeBSD customers to new overseas installations as well as several new sites in the States).

> The other side of the coin is that we had the resources to have someone (me)
> tracking the branch.

That person is now (me) plus I have Dave Robison (DaveR) to lean on occasionally (and you're always a great help, DaveR).

I'd argue that it's not the man-power... it's whether management sees the importance in allowing one (or two) persons to spin his/her wheels, developing a laudable solution to the problem at-hand (precisely what we've done; mitigating extra/busy work).

However, sometimes you just have to take initiative. The company didn't really officially "approve" any project that involved re-architecting our internal release engineering ... I had to take it upon myself over the last 3.5 years to do said monster-undertaking (in my ``spare'' time).

> I only spun a release when I thought it was a good time to do
> so, but I always had a year or so advance warning of when a new release was
> likely to be needed so I could select a good moment from over a wide range.

Likewise!

When Julian was here, the company had the same top-executives (such as Cayford), and likewise, I too have the same guidance.

If/When we ever find ourselves in the boat of our deployed release not supporting some hardware we ask ourselves three things:

1. Can we supplant missing support by making an out-of-band purchase of older hardware? (e.g. an onboard network card doesn't work, so we'll just throw an Intel PRO/100 PCI card into one of the extra slots)

2. Can we back-port missing driver support?

NOTE: I then go do research to answer yes/no.

3. How hard is it to back-port missing support if above-answer is YES?

NOTE: I then look at how hard it is which takes many things into consideration.

4. Is answer to above "days", "weeks", or "fat-chance"?

5. If answer to above is "days", I'm often instructed to do the back-port.

6. If answer is instead "weeks", we way our other options, including...

7. Can we "pre-ship" a newer OS having tested that it "plays-well with others" (in a heterogeneous environment)(example: shipping 8.1 workstations but still using a 4.11 server stack because workstations require better Vid-card support but servers do not).

8. Can we find another supplier that has the ability to source older hardware?

9. Failing any/all "easy" solutions, we then make a blanket statement that "it's time to move our customers to the next release" in which we then start enroads to regression test our product on said newer release as all other tricks to stay on the current release won't work.

It is through the above procedure that we accommodate customers that may or may-not have the annual budget to either:

a. do a "tech refresh" and/or

b. update the OS

in concordance to abate both EOM and EOL timelines dictated by hardware manufacturers (which we can only hope to influence even with *our* sizeable purchasing power; NOTE: We simply can't influence manufacturers like Google, Microsoft, Apple, etc. can).

> Also ran a layer on the top of the sources where I could  add cherry-picked back-
> ports and changes as part of our release.
> 

Yup, we've maintained that ability to this day. It's the only thing that's saved us. In transitioning to 8.1 we've already cherry-picked 8 patches in-wholesale from RELENG_8.

> If it came to that maybe all the people who are currently saying they need better
> support of the 8.x branch could get together and together, support someone to
> do that job for them..would 1/5th of  a person be too expensive for them?
> 
> if not, what is a reasonable cost?  Is it worth 1/20 th of a person?
> 

An interesting thought. Open to discussion on this one (meaning: I'm willing to volunteer).
-- 
Devin

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.