SMP on FreeBSD 6.x and 7.0: Worth doing?

[Scott Long]

Brett Glass wrote:
> I will need to build several Web caches over the next few months, and 
> just took advantage of the Christmas lull (and a snowy day, when I 
> couldn't work outside) to test FreeBSD 7.0 BETA 4 to see how it will 
> perform at this task. I built up a 4 core FreeBSD box, and asked a 
> friend who's a Linux fanatic to do the same with Linux on identical 
> hardware. I didn't watch closely how he installed everything, but asked 
> him not to tune  it beyond setting it up properly for SMP.
> 
> We then ran a test suite in which a client starts several processes. 
> Each uses wget to fetch a series of objects in rapid succession via the 
> cache. The fetches done by each process are the same batch of URLS, but 
> shuffled differently, so each URL will get a miss the first time and 
> then hits each time it comes up thereafter unless the cache overflows. 
> We're doing all GETs, with no tricky stuff like subranges.
> 
> As has been reported in some other messages on this list, Linux is 
> currently blowing FreeBSD away. It's taking as much as 20% less time to 
> get through the benchmark, depending on exactly how the random shuffle 
> came out. This is with 4 GB RAM, the GENERIC FreeBSD SMP kernel (using 
> SCHED_ULE), and aufs as the storage schema for Squid.
> 
> It appears, though I'd need to instrument the code more to be sure, that 
> the slowdown is coming from file I/O. Could it be that there less 
> concurrency or more overhead in FreeBSD file operations than there is in 
> Linux? Even with SoftUpdates turned on, the cache volume mounted with 
> -noatime, and aufs (which uses kqueues -- a FreeBSD invention -- to 
> optimize multithreaded disk access), the benchmark shows FreeBSD losing 
> out. Why?
> 

Brett,

There could be several problems here:

1. WITNESS, INVARIANTS, malloc debugging.  Are any of these turned on 
for you?  I don't recall if malloc debugging got turned off yet for the
7.0 snapshots.
2. Disk subsystem.  What kind of disk controller are you using?  Not all
drivers work well in FreeBSD.  Are linux and freebsd using identical
hardware?
3. Directory hashing.  If you're using squid, you __must__ tune the 
DIRHASH, otherwise you'll spend a lot of time doing pathname lookups. 
What filesystem is linux using?

Would you mind if I logged into your test system and looked around to
help diagnose the problem?



Scott