Quality of FreeBSD

[Robert Watson]

On Thu, 21 Jul 2005, Marc Olzheim wrote:

> Indeed. That's why my company started taking FreeBSD 5.3 in use for 
> production servers when it was out. Since then numerous bugs were fixed, 
> some of which reported by us. Now that we're X bug fixes later in time 
> and started to get a good feeling about the number of open problems, it 
> is extremely annoying to hear the "This will (probably) not be fixed in 
> 5.x" statements. That conflicts with 'gradually get resolved'. What do 
> you recommend larger consumers to do ? Keep using FreeBSD 4 and start 
> testing FreeBSD 6.x, dropping 5.x all together ?
>
> I know FreeBSD 5 was a strange exception in the relase scheduling and 
> that a lot has been learned from it for the future and I'm certainly not 
> unthankful for all the work that's done, but I'd like a clear answer on 
> what to do now in regard to taking FreeBSD 5 into 'real' production...

Marc,

I should start out by saying I appreciate your clear and concise bug 
reports, and the list of your company's show-stopper 5.x bugs has made the 
rounds among FreeBSD developers.  I'm happy that at least one of the 
issues on the list was fixed by me. :-)  As you probably saw yesterday, 
I've started bugging Poul-Henning to look at the pty problem you're 
experiencing, and will get that on our 6.0 release show-stopper list.  I 
haven't yet had a chance to reproduce it locally, but it sounds like that 
should be straight forward.

FreeBSD 5 has been an exception -- "normally", in as much as major 
releases have a "normal", the set of new features is a lot less agressive, 
and it has been our goal with 6.x to restore the expectation of a more 
rapid release cycle with a less agressive feature set.  This should reduce 
the number of problems by virtue of reducing the level of change.  It 
should also make it easier for users to pick what version to run on, as 
the amount of adaptation they have to do to slide forward a version will 
be greatly reduced.  I.e., right now it's relatively easy to move back and 
forward between 5.x and 6.x.

With respect to 5.x vs 6.x upgrades: I've seen companies take two 
different strategies.  Most of them have been at least experimenting with 
deploying 5.x, and are very interested in its feature set.  Support for 
large file systems, 64-bit support on newer AMD and Intel hardware, 
improved PAM support, etc.  Some of my customers are specifically 
interested in the support for mandatory access control, but that's 
obviously a less common feature request :-).  The biggest determining 
factor for companies today comes from their own product schedule, since 
most big consumers of FreeBSD treat it as a component in a "product" they 
deliver for others.

For example, my understanding is that Yahoo is now deploying 6.0 betas 
across their server environment with great success, but was actually 
unable to seriously deploy 5.x because their goal was to support full 
32-bit compatibility on 64-bit amd/intel hardware, which has only recently 
reached the level of maturity they require.  In fact, you'll notice if you 
follow FreeBSD commit logs that much of that support has come from Yahoo!. 
Since 6.x is maturing in pretty good synch with their deployment timeline 
for 5.x, they are actually deploying 6.x.  Of course, Yahoo! has a team of 
in-house OS developers who adapt FreeBSD for their needs, and is quite 
capable of debugging a kernel or two if they run into problems.

The ATA driver issue is a sticky one for many users -- we hope to get the 
6.x ATA code back into 5.x in the next 5.x release.  However, hard-earned 
experience tells us that ATA driver code is notoriously difficult to get 
right across the broad range of available hardware.  Soren has been 
lobbying to get it merged to 5.x, but given the level of testing performed 
so far, we can't yet justify the merge.  My hope is that with 6.0 out the 
door and a lot of testing of that code, we can get it merged back to 5.x 
before 5.5.  Many other fixes have gone into 5.x, correcting many of the 
most significant issues.  If you compare 5.4 with 5.3, you'll find that in 
most cases, it's both faster and more stable.

The tty issue is a sticky one also.  The tty code in 6.x has been 
substantially rewritten to better support the SMPng environment.  Because 
the tty code "plugs in" to a number of device drivers, T1 adapter drivers, 
etc, changing the tty interfaces is a fairly big event, and will affect 
third party vendors like Cronyx.  This code has also not yet seen as wide 
deployment as I'd like, so it's also something that really isn't 
appropriate for an MFC immediately.  However, once it has seen significant 
6.0 deployment, it may well be.  A question then will be whether it's 
better to simply say "you're better off making the jump to 6.x, which is 
minor" than backporting, and it's something we can't really answer until 
we're comfortable that it's seen sufficient deployment.  My hope is that 
we can identify a workaround for 5.x that will avoid the code upheaval a 
full backport would require.  It's not as ideal as having the "right" fix, 
but it would stop the panics.  I need to ping phk and some of the other 
tty-centric folk to look at this some more.

In terms of advice:

If you have a "product" due out more than 3 months from now, I think 6.x 
is the obvious way to go: you want to be ahead of the curve so that you 
can have the foundation for your product in sync with the FreeBSD 
production release cycle, and avoid jumping major releases early in the 
product life cycle.  6.x has significant performance and stability 
improvements -- performance especially in the area of file system 
performance on SMP, preemption, network stack, and memory management, and 
stability especially in the area of tty support.  By "product", I mean a 
range of things: the OS foundation of an embedded product such as a 
firewall or storage appliance, or deployment of an internal product, such 
as a virtual server product at an ISP.

On the other hand, if you're deploying today, I think that unless you're 
prepared to deal with the 6.0 bug fix cycle (both the BETA/RC cycle, and 
the inevitable post-release fixes for a .0 release), 5.4+patches or 
5-STABLE is the right place to sit.  At least two of the critical bugs on 
your list were fixed in 5-STABLE after the release of 5.4, so for some, 
5-STABLE is the best place to be.  We've opted not to do a patch/errata 
update for 5.4 for the socket error you were receiving on the basis that 
it doesn't affect a wide audience and doesn't correct a "Critical" failure 
-- i.e., a crash or the like, unlike some of the NFS server fixes, for 
which we did do an errata fix.

From the perspective of the FreeBSD developers, if you can tolerate the 
6.x release process, we encourage you to jump on that bandwagon.  It will 
help us release a better 6.0, and that's where the future lies.  Our goal 
is to make 6.x a pretty seemless upgrade from 5.x, as it has a less 
agressive feature set, and far fewer user-visible changes (i.e., no 
conversion to OpenPAM, devfs, UFS2, large compiler version upgrade, ... as 
in 5.x).  When I upgraded my personal web/shell server to 6.x from 5.x 
last week, I didn't have to change any configuration in /etc at all, other 
than a painless pass through mergemaster to merge the _dhcp user and 
group.  As always, we look to the freebsd-stable users to help us test new 
features ahead of the release.

Thanks,

Robert N M Watson