[fbsd] Re: [fbsd] Network performance in a dual CPU system

[Marcos Bedinelli]

Hi,

On 27-Apr-06, at 10:38, Jeremie Le Hen wrote:

> Hi, Robert,
>
> On Thu, Apr 27, 2006 at 02:54:21PM +0100, Robert Watson wrote:
>>
>> On Thu, 27 Apr 2006, Jeremie Le Hen wrote:
>>
>>>>  PID USERNAME   THR PRI NICE   SIZE    RES STATE    TIME   WCPU 
>>>> COMMAND
>>>>   60 root         1 -44 -163     0K     8K WAIT   355.6H 72.17% 
>>>> swi1:
>>>> net
>>>>   39 root         1 -68 -187     0K     8K WAIT    52.3H  5.22% 
>>>> irq28:
>>>> bge0
>>>>   40 root         1 -68 -187     0K     8K WAIT    28.3H  2.25% 
>>>> irq29:
>>>> bge1
>>>>   11 root         1 171   52     0K     8K RUN    166.6H  0.00% idle
>>>>   63 root         1 -16    0     0K     8K -      121:55  0.00% 
>>>> yarrow
>>>>   61 root         1 -32 -151     0K     8K WAIT    46:21  0.00% 
>>>> swi4:
>>>> clock sio
>>>> [...]
>>>>
>>>> Does anyone know whether a dual CPU system can help us improve the
>>>> situation? I was wondering if the software interrupt threads would 
>>>> be
>>>> divided between the two processors.
>>
>> I missed the original thread, but in answer to the question: if you 
>> set
>> net.isr.direct=1, then FreeBSD 6.x will run the netisr code in the 
>> ithread
>> of the network device driver.  This will allow the IP forwarding and
>> related paths in two threads instead of one, potentially allowing 
>> greater
>> parallelism. Of course, you also potentially contend more locks, you 
>> may
>> increase the time it takes for the ithread to respond to new 
>> interrupts,
>> etc, so it's not quite cut and dry, but with a workload like the one 
>> shown
>> above, it might make quite a difference.
>
> Actually you already replied in the original thread, explaining mostly
> the same thing.  The whole thread [1] brought up multiple valuable
> network performance tuning knobs, such as polling, fastforwarding,
> net.isr.direct but there is no happy end to the thread.  Given this is
> a real world situation, I wanted to know how Marcos revolved his
> problem.


Shortly after I opened the thread, most replies I received (from the 
list or privately) were saying that a second CPU either wouldn't help 
to improve the situation, or that the performance gain was unclear. We 
had an immediate problem to solve and the management wasn't comfortable 
in throwing money at new and expensive hardware that, in the end, might 
or might not solve the issue.

The performance gain with fastforwarding enabled was almost 20% and 
that's what we've been running since.

Note, however, that I am also using virtual interface disc0 and that 
fastforwarding doesn't work with that interface (at least prior to and 
including FreeBSD 6.0-release-p4). Gleb Smirnoff sent me a one-line 
patch that fixed the problem.

Hope this helps.

Regards,

--
Marcos