Network performance in a dual CPU system

[Gleb Smirnoff]

On Fri, Feb 10, 2006 at 08:46:00AM -0500, Marcos Bedinelli wrote:
M> We have a 2.4GHz Intel Xeon machine running FreeBSD 6.0-RELEASE-p2. Due 
M> to heavy network traffic, CPU utilization on that machine is 100%:
M> 
M> ===
M> 
M> mull [~]$top -S
M> last pid: 94989;  load averages:  3.69,  4.02,  4.36           up 
M> 25+07:21:34  14:51:43
M> 105 processes: 2 running, 46 sleeping, 57 waiting
M> CPU states:  0.0% user,  0.0% nice,  0.3% system, 99.4% interrupt,  
M> 0.3% idle
M> Mem: 20M Active, 153M Inact, 84M Wired, 4K Cache, 60M Buf, 237M Free
M> Swap: 999M Total, 999M Free
M> 
M>   PID USERNAME   THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
M>    60 root         1 -44 -163     0K     8K WAIT   355.6H 72.17% swi1: 
M> net
M>    39 root         1 -68 -187     0K     8K WAIT    52.3H  5.22% irq28: 
M> bge0
M>    40 root         1 -68 -187     0K     8K WAIT    28.3H  2.25% irq29: 
M> bge1
M>    11 root         1 171   52     0K     8K RUN    166.6H  0.00% idle
M>    63 root         1 -16    0     0K     8K -      121:55  0.00% yarrow
M>    61 root         1 -32 -151     0K     8K WAIT    46:21  0.00% swi4: 
M> clock sio
M> [...]

You didn't described what the box is actually. According to top(1) output,
looks like your system is routing between bge0 and bge1. If this is true,
then you should try:

  sysctl net.inet.ip.fastforwarding=1

This will split the work into the ithreads and also this will use a faster
code path. The performance will improve, and obtaining a second CPU will
give you a benefit, since you will have two heavy loaded threads.

You should also try upgrade to 6.1, since bge(4) driver has undergone some
improvements.

If my guess was incorrect, and you aren't routing traffic, then you should
follow Robert's advice and try:

  sysctl net.isr.direct=1

And see whether load is splitted betweeen processes. If it is, then purchase
second CPU.

M> Does anyone know whether a dual CPU system can help us improve the 
M> situation? I was wondering if the software interrupt threads would be 
M> divided between the two processors.

-- 
Totus tuus, Glebius.
GLEBIUS-RIPN GLEB-RIPE