Netflix's New Peering Appliance Uses FreeBSD

[Scott Long]

On Jun 5, 2012, at 9:56 AM, Benjamin Francom wrote:

> I just saw this, and thought I'd share:
> 
> Open Connect Appliance Software
> 
> Netflix delivers streaming content using a combination of intelligent
> clients, a central control system, and a network of Open Connect appliances.
> 
> When designing the Open Connect Appliance Software, we focused on these
> fundamental design goals:
> 
>   - Use of Open Source software
>   - Ability to efficiently read from disk and write to network sockets
>   - High-performance HTTP delivery
>   - Ability to gather routing information via BGP
> 
> Operating System
> 
> For the operating system, we use FreeBSD <http://www.freebsd.org/> version
> 9.0. This was selected for its balance of stability and features, a strong
> development community and staff expertise. We will contribute changes we
> make as part of our project to the community through the FreeBSD committers
> on our team.
> Web server
> 

Yes, we are indeed using FreeBSD at Netflix!  For those who are interested, I
recently moved from Yahoo to Netflix to help support FreeBSD for them, and
I'm definitely impressed with what is going on there.  Other than a few small
changes, we're using stock FreeBSD 9, tracking the 9-stable branch on a
regular basis.  Our chassis is a semi-custom 4U 19" form factor with thirty six
3TB SATA disks and 2 SSDs.  Each disk has its own UFS+J filesystem, except for
the SSDs that are mirrored together with gmirror.  The SSDs hold the OS image
and cache some of the busiest content.  The other disks hold nothing but the
audio and video files for our content streams.  We connect to the outside world
via a twin-port Intel 10GBe optical NIC (only one port is active at the moment),
and we use LSI MPT2 controllers for 32 of the 36 disks.  The other 4 disks
connect to the onboard AHCI SATA controller.  All of the disks are
direct-attach with no SAS backplanes or expanders.  Out-of-band management
happens via IPMI on an on-board 1Gb NIC.  The entire system consumes
around 500W of power, making it a very efficient appliance for its functionality.

Netflix is also at the front of the internet pack with IPv6 roll-out, and FreeBSD
plays an essential part of that.  We've been working hard on stabilizing the
FreeBSD IPv6 stack for production-level traffic, and I recommend that all users
of IPv6 update to the latest patches in 9-stable and 8-stable.  Contact me
directly if you have questions about this.  That said, we're excited about World
IPv6 Day, and we're ready with AAAA DNS records and content service from both
Amazon and the traditional CDNs as well as our OpenConnect network.

>From an advocacy standpoint, Netflix represents 30% of all North American
internet traffic during peak hours, and FreeBSD is becoming an integral part
of that metric as we shift traffic off of the traditional CDNs.  We're expanding
quickly, which means that FreeBSD is once again a core part of the internet
infrastructure.  As we find and fix stability and performance issues, we're
aggressively pushing those changes into FreeBSD so that everyone can
benefit from them, just as we benefit from the contributions of the rest of the
FreeBSD ecosystem.  We're proud to be a part of the community, and look
forward to a long-term relationship with FreeBSD.

If you have any questions, let me know or follow the information links on the
OpenConnect web site.

Scott