UDP performance.

[Robert Watson]

On Wed, 28 Feb 2007, Peter Losher wrote:

> Ivan Voras wrote:
>
>> I agree in general, but MySQL performance is very exposed as an advocacy 
>> issue - it has traditionally been the source of statements like "FreeBSD's 
>> threading implementation is weak/bad/broken".
>
> And these days ISC can't consciously recommend FreeBSD for use on 
> high-traffic DNS servers because UDP performance has (frankly) gone downhill 
> since 5.x.
>
> We recently put a stock Fedora Core 6 and a stock FreeBSD 6.2 on the same HW 
> (HP ProLiant DL320 G5 Dual Core Xeons w/ 16GB RAM) and running BIND 9.4.0 
> and a well known ccTLD zone that we slammed a query stream to.  On a single 
> threaded BIND, there was a 20% advantage to Linux, on a multi threaded 
> build, Linux trounced FreeBSD (39k to 89k queries/sec)
>
> There's also been other analysis done by Marcelo Amarai @ Registro.br that 
> was posted to freebsd-net back last September.
>
> http://lists.freebsd.org/pipermail/freebsd-net/2006-September/011748.html
>
> I know there have been some discussion between some of the FreeBSD folks and 
> my colleague Mark Andrews about improving BIND's performance on FreeBSD. 
> Is there anything coming down the pipeline that will help stem this tide in 
> 7.x?

Yes -- we have a specific optimization in 7.x to improve the performance of 
highly contended multi-threaded send on a single UNIX domain socket as a 
direct result of those conversations with ISC folk.  It's in the MFC pipeline, 
but not something I felt comfortable merging for 6.2 without a bit more 
experience with it in the field.  If you're interested in testing (and 
specifically, measuring) this patch in a high performance DNS test 
environment, that would be wonderful -- in the original e-mail thread I wa 
unable to successfully solicit testing of the patch.  Likewise, when I 
requested access to test query streams from ISC, I was told those couldn't be 
shared.  I've had a recent request for an MFC, and hope to have a chance to 
update and test the RELENG_6 sosend_dgram() patch in the next few days and 
will post it to this mailing list when it's ready.  Unfortunately, this month 
is scary-travel-month so my oppoprtunities to do back-port testing myself in 
the next two weeks are limited.

What would be extremely helpful is to have someone willing to own the "BIND9 
benchmarking" item as Kris has done for MySQL, who could provide regular 
support in testing updated FreeBSD versions and patches against a specific 
reproduceable workload, and participate in a continuing dialog on measuring 
and improving performance.  It's those sorts of dialogs that are invaluable in 
improving performance, as frequently FreeBSD developers have limited access to 
real-world workloads other than the ones in their own environments, and have 
less familiarity with the application architecture.  Kris's long-term work in 
measuring MySQL performance and working with developers to test and improve 
patches is why FreeBSD 7.x (with patches) now performs four times better than 
Linux under load.

The first step here would be to perform a performance baseline measurement 
with head-of-tree 7.x (debugging features in user-space and kernel disabled 
first, threading library switched to libthr) and compare it with 6.2-RELEASE. 
This would let us know if the current generation of optimization work is 
effective in improving BIND9 traffic with your configuration and workload. 
We can then apply a series of the current out-of-tree optimizing patches for 
threading and scheduling and see if those help.

(The above applies to any workload people are concerned about, BTW: what we 
need right now is people willing to take ownership of benchmarking and engage 
with developers in measuring change over time).

Robert N M Watson
Computer Laboratory
University of Cambridge