SMP on FreeBSD 6.x and 7.0: Worth doing?

[Brett Glass]

At 09:10 AM 12/24/2007, Scott Long wrote:

>Did you also nuke malloc debugging?

I believe I did. I tried to take out all debugging to make it a fair test.

>>They were. The drives are SATA.
>
>Connected to what controller?

Whatever comes standard on the Intel S5000 motherboards. I believe that it
is Intel's own embedded SATA controller, which according to the spec sheet
is called their "ESB2-E" embedded SATA RAID controller. We are not using the
RAID capability.

>>>3. Directory hashing.  If you're using squid, you __must__ tune the DIRHASH, otherwise you'll spend a lot of time doing pathname lookups. What filesystem is linux using?
>>Whatever comes standard with Ubuntu.
>
>Which is???

EXT3.

>>My guess, based on what I saw, is that UFS2 doesn't take as much advantage of
>>SMP as Linux's file system does and threads are blocking on file I/O. 
>
>That's really just speculation on your part, though.

Yes, it is -- though I'd like to think it is an educated guess. ;-)
I'd need to instrument either the benchmark code or the kernel to 
tell for sure. But the test is, by its nature, bound by file I/O.

>I can tell you from my experience that a thrashed namei cache looks a
>lot like slow disk I/O to the casual observer, and that tuning the dirhash is highly important.

That's always been true. However, Squid's file layout tends to be
gentle on the file system. It lays out directories with no more than
256 items in each, and the names of the files are just sequential
hexadecimal numbers -- which I'd expect ought to bring about near-perfect 
if not perfect hashing. If you set the kernel to expect 256 or more entries 
per directory (I know that Adrian Chadd is on this list; do you recommend
256 or 512?), you seem to get the best performance you are going to get.

We have thought about using COSS for small Web objects, but are not sure how
it would interact with AUFS or how much it would impact stability by decreasing
the size of Squid's "CACHE_MEM" memory pool (which is used for "hot" objects 
and objects in transit). Squid tends to crash horribly if this pool isn't
kept quite big.

--Brett Glass