SMP on FreeBSD 6.x and 7.0: Worth doing?

[Brett Glass]

At 10:12 AM 12/24/2007, Scott Long wrote:

>It's not the same kind of hashing.  The kind of "hashing" that squid
>does on the filesystem is sub optimal for UFS performance.

Squid doesn't do any "hashing" on the file system, as far as I know.
It does, of course, have a hashed directory of cached Web objects.
What it does, however, is lay out files and subdirectories so that
there are no more than 256 entries in each directory. (The top level 
directory usually has only 16 subdirectories in it.) What I am saying 
is that by keeping directory size down and using hexadecimal numbers 
as the file names, the Squid storage conventions will likely allow the 
UFS hashing scheme to work very well -- possibly optimally. Which 
isn't a surprise, because they were designed with UFS in mind.

>It sounds like you're pretty convinced you know what the problem is.

Again, I'd have to instrument either the FreeBSD kernel or Squid to
be 100% sure. But it APPEARS that it's a problem with large
numbers of threads trying to do file I/O simultaneously and blocking.

(Aside: I wish that Squid used FreeBSD's sendfile(2) API once it got past 
the headers and was just streaming out a cached page. I suspect that 
this would make delivery of cache hits a lot more efficient, because
sendfile(2) does "zero copy" transfers and works entirely within
the kernel.)

>For others who might want help with this, tweaking
>vfs.ufs.dirhash_maxmem is what is needed.  A bit of a balancing act is
>needed if you're on i386 since you'll risk exhausting KVA unless you
>also tweak KVA_PAGES.

If you'd like, I can reassemble the system and try this. What would
your suggested values for these tunables be?

--Brett