throughput and interrupts

[Bachilo Dmitry]

В сообщении от Вторник 15 августа 2006 19:02 Robert Watson написал(a):
> On Tue, 15 Aug 2006, Bachilo Dmitry wrote:
> > I am completely stuck. I have a router, that rules three subnets. At
> > first there were Allied Telesyn's netcards with realtek chipsets. I am
> > using ipfw+natd and I had like 3,5 megabytes per second with 20 per cent
> > of interrupt load. I had "discarded oversized frame" too often so I've
> > replaced two main cards with 3Com 905 and now my xl0 and xl1 never
> > discard frames. But I saw no speed boost and interrupts were also very
> > high. I have discovered some IRQ conflicts and removed them, now I have
> > single device per IRQ, but still i have near 20-30 % of interrupt load,
> > when i download something through this router (and natd takes rest % of
> > CPU). Some people advised me to use polling, so I've built kernel with
> > HZ=1000, polling and tried it. Interrupts grew to 40-50% and speed
> > decreased to 200-300 kb/s, so I had to turn the polling off.
> >
> > I just don't know what to try, to gain at least 9 mb/s, because this is
> > what I get at home with the same cards, the same processor, with no
> > polling and throgh the same provider.
>
> I think it would be useful to compare the settled loads of the two systems,
> if they are otherwise identical, using systat -vmstat 1 and top -S.  I'd
> load the systems up, leave the tools running, and wait a couple of minutes,
> then compare snapshots of the output from both systems.  top -S is
> particularly informative, as it will tell you which ithreads are burning
> the most CPU, and systat -vmstat will tell us the context switch rate and
> interrupt rates of various sources.
>
> There are some general optimization techniques that can be applied here --
> for example, if it's an SMP box, you might find that net.isr.direct=1
> improves performance.  However, I think the real problem here is the
> performance difference, which suggests a problem that needs resolving,
> rather than optimization being required up front.
>
> Thanks,
>
> Robert N M Watson
> Computer Laboratory
> University of Cambridge

Here is what I see:

CPU states:  9.6% user,  0.0% nice, 58.5% system, 31.9% interrupt,  0.0% idle
Mem: 127M Active, 664M Inact, 138M Wired, 48M Cache, 111M Buf, 21M Free
Swap: 64M Total, 272K Used, 64M Free

  PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU    CPU COMMAND
   11 root     171   52     0K     8K RUN    422.7H 33.06% 33.06% idle
  462 root     109    0  1496K   952K RUN    202:44 21.53% 21.53% natd
   37 root     -44 -163     0K     8K WAIT    99:09  5.08%  5.08% swi1: net
   29 root     -68 -187     0K     8K RUN     59:11  3.52%  3.52% irq19: xl0
   30 root     -68 -187     0K     8K WAIT    40:38  3.47%  3.47% irq20: xl1

This machine is Celeron 1,7 Ghz and 1 GB RAM and it gives me 3,7 MB per second 
max. While my home router is Celeron 1,7 Ghz with 512 MB RAM, the same xl 
NICs and I have almost 10 MB per second there with at least 48% of free CPU. 
I'll keep on testing this stuff, so i'll keep reporting the results.
-- 
------------------------
С уважением, Бачило Дмитрий
Руководитель отдела системной интеграции
ООО "Компания СоЛинк"