Tuning GigE network for cluster computing?

[Danial Thom]

--- Steve Kargl
<sgk at troutmask.apl.washington.edu> wrote:

> First, I'm running FreeBSD 6.1-stable on a
> small cluster
> with 6 nodes that contain Tyan motheriboards. 
> These broads
> have Broadcom GigE NICs that use the bge
> device.
> 
> The cluster will be using MPI to possibly shove
> large data
> sets through a GigE switch, so I'm trying to
> determine how
> best to optimize the transfer.
> 
> Would DEVICE_POLLING and/or jumbo frames likely
> methods for
> enhanced speeds?
> 
> -- 
> Steve


--- Steve Kargl
<sgk at troutmask.apl.washington.edu> wrote:

> First, I'm running FreeBSD 6.1-stable on a
> small cluster
> with 6 nodes that contain Tyan motheriboards. 
> These broads
> have Broadcom GigE NICs that use the bge
> device.
> 
> The cluster will be using MPI to possibly shove
> large data
> sets through a GigE switch, so I'm trying to
> determine how
> best to optimize the transfer.
> 
> Would DEVICE_POLLING and/or jumbo frames likely
> methods for
> enhanced speeds?


Firstly we've been discussing how bad and
inefficient the broadcom controllers are. If you
have a tyan MB with onboard controllers they are
incredibly slow (ie inefficient), as well as
being quirky. If you have a PCI-X slot put in an
intel card. They have built-in interrupt
moderation so you don't have dick around with
polling. A 133Mhz intel controller (make sure
they have the GB chips on them and not the EB)
will use half the cpu of the on-board broadcoms.
If you don't believe try it with one box and
measure the cpu usage before and after. Its worth
the $50. investment in the card, believe me.

While polling may marginally decrease the cpu
load (depending on how bad FreeBSD 6.1 is on
interrupt overhead), its also going to add
latency to the processing of packets, which is
the opposite of what you want to do. 

DT

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com