SMP on FreeBSD 6.x and 7.0: Worth doing?

[Martin Cracauer]

Brett Glass wrote on Fri, Dec 21, 2007 at 10:31:24PM -0700: 
> 
> As has been reported in some other messages on this list, Linux is 
> currently blowing FreeBSD away. It's taking as much as 20% less 
> time to get through the benchmark, depending on exactly how the 
> random shuffle came out. This is with 4 GB RAM, the GENERIC FreeBSD 
> SMP kernel (using SCHED_ULE), and aufs as the storage schema for Squid.

How much CPU load is there on the server during this benchmark?

As I have measured and reported since 1997, if you have CPU load and
do throughput timing at the same time, then FreeBSD always defaults to
satisfy the CPU eating processes and starves network (at least TCP)
and Linux is the other way round, keeps the throughput up but CPU
needing processes lose out.

Kind of opposite of what you'd expect from the ninja macho networker
OS (FreeBSD) and the desktop friendly OS (Linux).

Anyway, the real question is whether you get anything in return, aka
is there anything running on the server that uses CPU but is not
directly bound to the network throughput and hence gets work done
quicker.

> It appears, though I'd need to instrument the code more to be sure, 
> that the slowdown is coming from file I/O.

I could measure the above effects with things coming out of nowhere,
aka data made up by a process.  If you are observing the same
phaenomenon then it has nothing to do with file I/O.

> Could it be that there 
> less concurrency or more overhead in FreeBSD file operations than 
> there is in Linux? Even with SoftUpdates turned on, the cache 
> volume mounted with -noatime, and aufs (which uses kqueues -- a 
> FreeBSD invention -- to optimize multithreaded disk access), the 
> benchmark shows FreeBSD losing out. Why?

All that async and softupdates stuff only matters if you use a lot of
small files, which you shouldn't do in the first place if you are
building a cache of some sorts.

In all likelyness the cache that you are building is satisfying
requests out of the RAM, even if the cache is file-backed.  You need
to verify that before you can blame file I/O.

So the real questions right now are:
- how much CPU load (post top output)
- how much disk activity (post iostat or vmstat output)
- can't hurt to have a netstat throughput output, too

Martin
-- 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Martin Cracauer <cracauer at cons.org>   http://www.cons.org/cracauer/
FreeBSD - where you want to go, today.      http://www.freebsd.org/