FreeBSD on Amazon AWS EC2 long standing performance problems

Sat Feb 6 00:23:02 UTC 2021

G'Day Gunther,

I can at least share some experiences from the other side, and how
they can apply to FreeBSD.

On Sat, Feb 6, 2021 at 1:53 AM Gunther Schadow <raj at gusw.net> wrote:
>
> Hi, I've been with FreeBSD since 386BSD 0.0new. Always tried to run
> everything on it. I saw us lose the epic race against Linux over the
> stupid BSDI lawsuit. But now I'm afraid I am witnessing the complete
> fading of FreeBSD from relevance in the marketplace as the performance
> of FreeBSD on AWS EC2 (and as I see in the chatter from other "cloud"
> platforms) falls far behind that of Linux. Not by a few % points, but
> by factors if not an order of magnitude!
>
> The motto "the power to serve" meant that FreeBSD was the most solid
> and consistently performing system for heavy multi-tasking network
> and disk operation. A single thread was allowed to do better on another
> OS without us feeling shame, but overall you could rely on FreeBSD
> being your best choice to overall server performance.
>
> The world has changed. We used to run servers on bare metal in a cage
> in physical data center. I did that. A year or two of instability with
> the FreeBSD drivers for new beefy hardware didn't scare me off.
>
> Now the cost and flexibility calculations today changed the market
> away from bare metal to those "cloud" service providers, Amazon AWS
> (>38% market share), Azure (19% market share), and many others. I
> still remember searching for "hosting" providers who would
> offer FreeBSD (or any BSD) as an option and it was hard to find. On
> Amazon AWS we have the FreeBSD image ready to launch, that is good.
>
> But the problem is, it's disk (and network?) performance is bad (to
> horrible) and it's really sad and embarrassing. Leaving FreeBSD beaten
> far behind and for realistic operations, it's impossible to use, despite
> being so much better organized than Linux. I have put significant
> investment into a flexible scalable FreeBSD image only to find now that I
> just cannot justify using FreeBSD when Linux out of the box is several
> times faster.
>
> There have been few problem reports about this over many years, and
> they all end the same way: either no response, or defensive response
> ("your measures are invalid"), with the person reporting the problem
> eventually walking away with no solution. Disinterest. I can link to
> those instances. Examples:
>
> https://lists.freebsd.org/pipermail/freebsd-performance/2009-February/003677.html
> https://forums.freebsd.org/threads/aws-disk-i-o-performance-xbd-vs-nvd.74751/
> https://forums.freebsd.org/threads/aws-ec2-ena-poor-network-performance-low-pps.77093/#post-492744
> https://forums.freebsd.org/threads/poor-php-and-python-performances.72427/
> https://forums.freebsd.org/threads/freebsd-was-once-the-power-to-server-but-in-an-aws-world-we-have-fallen-way-waaay-behind-and-there-seems-no-interest-to-fix-it.78738/page-2
>
> My intention is not to rant, vent, proselytize to Linux (I hate Linux)
> but to see what is wrong with FreeBSD? And how it can be fixed? Why does
> it seem nobody is interested in getting the dismal AWS EC2 performance
> resolved? This looks to me like a vicious cycle: FreeBSD on AWS is
> bad so nobody will use it for any real work, and because nobody uses it
> there is no interest in making it work well. In addition there is no interest
> on the side of FreeBSD people to make it better. It's got to be the lack
> of interest, not of anyone not having access to the AWS EC2 hardware.

I think the better question is how many full time staff work on EC2
FreeBSD performance, and how to create more roles.

Large companies have performance engineering teams to reduce cost, as
do latency-sensitive companies of any size. I'd estimate there are
well over 100 staff with the title "performance engineer" who work on
Linux: most of whom work on Linux as part of another product.

There is a mentality with these large companies, whether it makes
sense or not, to run their own datacenters. So most of the performance
engineers on Linux are looking at bare metal performance and not EC2.

Fortunately for Linux, there are a few of us who do work on EC2. I
work at Netflix on the streaming side, and myself and a colleague
(Amer) are performance engineers who work on Linux EC2 a lot (as well
as other things). Other teams work on Linux EC2 performance from time
to time (e.g., BaseOS and Titus). If you were to add up all our
collective time, it probably works out to 2 full time engineers
working on Linux EC2 performance. Our focus is LTS releases, so we
aren't finding issues every day (we likely would if we were looking at
mainline), but we do find some and get them fixed.

So who is the Netflix of EC2 FreeBSD? What large company, with a
performance team (or is large enough to create one) runs (or may
consider running) FreeBSD on EC2? I'd identify the company and
management, and help them with a proposal and job description for a
performance engineer. My Systems Performance book (2nd Ed) documents
methodologies that are applicable to BSD (and includes various
mentions of BSD), and could help a new engineer get started.

There are some who do work on FreeBSD EC2 sometimes, e.g.:

https://twitter.com/cperciva/status/1211125881264934917

Work like this is great, but performance work is endless and needs
full-time attention. So I wouldn't say the problem is a lack of
interest, but a lack of full-time roles. Who can create them?

(Yes, Netflix does create full-time roles for FreeBSD bare metal
performance on the OCA team. But their focus is bare metal.)

As for the low-level performance issues: EC2 has been switching to the
Nitro hypervisor, and I imagine there may be driver work to make sure
it's using it correctly. I summarized the switch as:

http://www.brendangregg.com/blog/2017-11-29/aws-ec2-virtualization-2017.html

Brendan

>
> What can be done? I am trying to run a company, so I cannot justify playing
> with this for much longer shooting in the dark. If I wasn't the boss myself,
> my boss would have long told me to quit this nonsense and use Linux.
> If I saw interest, I could justify holding out just a little longer. But
> I don't see any encouraging feedback. Is there anyone at all in the FreeBSD
> dev or FreeBSD.org as an organization interested in actually being competitive
> in the AWS EC2 space (and other virtualization "clouds")? If so, how many?
> How can this be fixed? How can I help? I cannot justify spending too much
> more of my own time on it, but I could help making resources available
> or paying for someone who has both a sense of great urgency to redeem
> FreeBSD and the know-how to make it happen.
>
> regards,
> -Gunther
>
>
> _______________________________________________
> freebsd-performance at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "freebsd-performance-unsubscribe at freebsd.org"