Channel bonding.

[Sean]

> Are the gig nics in 64bit slots?  32bit slots can slow you down a bunch.
>   Also, I've seen some cases where the PCI bus itself is the bottleneck
> with multiple high-IO boards installed on the same bus.
> 
Yes.  The 3ware card and the Dual Port em are both in the PCI-X 100
slots (the first two).
There are no 32bit cards in the machine.

Is there some method I can use to determine if the PCI bus is the
bottleneck or even to determine the level of contention on the bus?

I've been looking at the newer Opteron boards with multiple PCI buses
but that's a lot of money to spend on my hunch.

-- 
Sean