Benchmark: NetBSD 2.0 beats FreeBSD 5.3 in server performance

[stheg olloydson]

it was said:

Gregory McGarry asked me to host and advertize this paper for him:

Abstract: ``With the recent releases of NetBSD 2.0 and FreeBSD 5.3
operating 
system, many new and exciting features have been implemented. Both
criticism 
and commendation on performance, reliability and scalability have been
directed 
towards these releases.

This paper presents a suite of benchmarks and results for comparing the

performance of these operating systems. The benchmarks target core
operating 
system functionality, server scalability and thread implementation.
<snip>

URL: http://www.feyrer.de/NetBSD/gmcgarry/


---------------

Hello,

I read the "paper" with which you trolled a couple of FreeBSD lists.
Here, quoting the "paper" is what stands out to me:


Abstract:
"These benchmarks are useful server-based criteria for demanding
applications such as loaded webservers, databases, and voice-over-IP
(VoIP) media relays. The results indicate that NetBSD has surpassed
FreeBSD in performance on nearly every benchmark and is poised to grab
the title of the best operating system for the server environment."

System-call overhead
"The results in Table 1 shows that NetBSD 2.0 marginally out-performs
FreeBSD 5.3."

Context-switch time
"The results in Table 1 shows that NetBSD 2.0 marginally outperforms
FreeBSD 5.3."

Process lifecycle
"The results in Table 1 indicate that NetBSD 2.0 marginally outperforms
FreeBSD 5.3."

Forking new processes
"The FreeBSD kernel has an access time which scales linearly with the
number of system processes. There are also many occasions when the
access-time is very fast, resembling a constant access time."

Memory-mapped files
"The total of both benchmarks indicate that for a single mapping and
subsequent access, FreeBSD shows a 38% performance improvement over
NetBSD."

Socket creation scalability
"Neither NetBSD nor FreeBSD shows scalability problems."

Binding addresses to sockets
"This result indicates the FreeBSD scales better for binding addresses
to sockets."

Thread creation benchmark
"For less than 250 threads, the time to create a thread is better in
NetBSD than FreeBSD. For more than 250 threads, the thread creation
time increases as the number of threads increases. Of particular
concern, the relationship is not linear for the number of threads.
Although one thousand threads is ample for most multi-threaded
applications, the poor scalability may be a problem for some
applications."

Thread lifecycle, condition variables and mutexes
"The results of these benchmarks for the basic POSIX thread primitives
clearly shows that the NetBSD thread implementation contains many
impressive optimizations."

Thread context-switch time
"The creation of the first thread incurs a significant penalty on
NetBSD. However, the time to complete the game is significantly higher
for FreeBSD over NetBSD. This is due to increased latency in the thread
lifecycle and the much longer mutex acquire time for FreeBSD over
NetBSD."

Conclusions
"Significant performance improvements are obviously visible in the
thread implementation....Microbenchmarks are not always the best
indicators to make judgments on the overall performance of one
operating system over another...For many applications, the results
presented in the paper may never affect performance. For others, the
scalability of the operating system may simply not permit the
application to run suitably."


Clearly, the claims in the Abstract are not supported by the tests. The
author even points out the inapplicability of the benchmarks that he
ran. But even if they were, the author points out that only in the case
of threading performance does NetBSD do appreciably better than
FreeBSD. In all other tests, FreeBSD either does better, as well as, or
only "marginally" worse than NetBSD.
So what this should have been called is, "Meaningless Tests Show NetBSD
2.0 is Finally About as Good as FreeBSD 5.3." I want the 20 minutes
that I spent reading this "paper" and responding to it back. If you are
unwilling or unable to return the piece of my life you caused me to
waste, I demand to personally apologise to everyone who reads not only
the post, but the "paper", as well.

stheg


		
__________________________________ 
Do you Yahoo!? 
Yahoo! Mail - Easier than ever with enhanced search. Learn more.
http://info.mail.yahoo.com/mail_250