SMP on FreeBSD 6.x and 7.0: Worth doing?

[Scott Long]

Brett Glass wrote:
> At 07:14 AM 12/24/2007, Scott Long wrote:
> 
>> Brett,
>>
>> There could be several problems here:
>>
>> 1. WITNESS, INVARIANTS, malloc debugging.  Are any of these turned on for you?  I don't recall if malloc debugging got turned off yet for the
>> 7.0 snapshots.
> 
> I nuked debugging when I recompiled the kernel with SCHED_ULE.

Did you also nuke malloc debugging?

> 
>> 2. Disk subsystem.  What kind of disk controller are you using?  Not all
>> drivers work well in FreeBSD.  Are linux and freebsd using identical
>> hardware?
> 
> They were. The drives are SATA.

Connected to what controller?

> 
>> 3. Directory hashing.  If you're using squid, you __must__ tune the DIRHASH, otherwise you'll spend a lot of time doing pathname lookups. What filesystem is linux using?
> 
> Whatever comes standard with Ubuntu.

Which is???

> As for directory hashing: Squid doesn't
> use more than 256 entries in each one, so that's what I normally set. I
> also normally do a newfs with parameters that favor the distribution of 
> object sizes found in Web caches. (We did this on both Linux and FreeBSD.)

newfs tuning has little to do with this.

> 
>> Would you mind if I logged into your test system and looked around to
>> help diagnose the problem?
> 
> The system isn't online now, because it's been a week since the tests and 
> I also wanted to try the 6.3 beta and a few hardware changes.
> 
> My guess, based on what I saw, is that UFS2 doesn't take as much advantage of
> SMP as Linux's file system does and threads are blocking on file I/O. 

That's really just speculation on your part, though.  UFS is SMP 
capable, but there really are a whole lot of factors that some into
play here so it's really hard to speculate with any chance of success.
I can tell you from my experience that a thrashed namei cache looks a
lot like slow disk I/O to the casual observer, and that tuning the 
dirhash is highly important.  Thus why I'd like to help out here and
see for myself what is going on.

Scott