Polling slows down bandwidth

[Pyun YongHyeon]

On Thu, Oct 28, 2010 at 11:21:11PM +0300, ?????????????? ?????????????? wrote:
> Hello,
> w/0 polling:
> 
> 
> serv1# ifconfig nfe0
> nfe0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         options=10b<RXCSUM,TXCSUM,VLAN_MTU,TSO4>
>         ether 00:13:d4:ce:82:16
>         inet 10.11.8.17 netmask 0xfffffc00 broadcast 10.11.11.255
>         inet 10.11.15.15 netmask 0xffffff00 broadcast 10.11.15.255
>         inet 10.11.8.15 netmask 0xfffffc00 broadcast 10.11.11.255
>         media: Ethernet autoselect (1000baseTX <full-duplex>)
>         status: active
> 
> serv2# ifconfig re0
> re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
>         ether 00:1c:c0:c8:5a:4e
>         inet 192.168.255.254 netmask 0xffffffff broadcast 192.168.255.254
>         media: Ethernet autoselect (1000baseTX <full-duplex>)
>         status: active
> 
> 
> serv1# systat -v
>     2 users    Load  0.38  0.49  0.38                  Oct 28 23:10
> 
> Mem:KB    REAL            VIRTUAL                       VN PAGER   SWAP PAGER
>         Tot   Share      Tot    Share    Free           in   out     in   out
> Act  324452   20196   490400    25112  938576  count
> All  346596   21060 1074286k    28104          pages
> Proc:                                                            Interrupts
>   r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Flt        cow   29953 total
>           1  47       71k   18 5022  27k 2029    1        zfod        atkbd0 1
>                                                           ozfod     7 ata0 irq14
> 62.4%Sys  18.7%Intr  0.4%User  0.0%Nice 18.4%Idle        %ozfod 27944 nfe0 irq23
> |    |    |    |    |    |    |    |    |    |    |       daefr  2001 cpu0: time
> ===============================++++++++++                 prcfr       igb0 256
>                                          1 dtbuf        6 totfr     1 igb0 257
> Namei     Name-cache   Dir-cache    100000 desvn          react       igb0 258
>    Calls    hits   %    hits   %     84106 numvn          pdwak
>        5       5 100                 24824 frevn          pdpgs
>                                                           intrn
> Disks   ad0                                        373808 wire
> KB/t  16.00                                        292088 act
> tps       7                                        417568 inact
> MB/s   0.10                                           200 cache
> %busy     0                                        938376 free
> 
> iperf result:
> [ ID] Interval       Transfer     Bandwidth
> [  4]  0.0-10.3 sec    450 MBytes    368 Mbits/sec
> 
> 
> after enable POLLING:
> serv1# ifconfig nfe0 polling
>     2 users    Load  0.32  0.39  0.35                  Oct 28 23:13
> 
> Mem:KB    REAL            VIRTUAL                       VN PAGER   SWAP PAGER
>         Tot   Share      Tot    Share    Free           in   out     in   out
> Act  324464   20196   490656    25112  938428  count
> All  346624   21060 1074286k    28104          pages
> Proc:                                                            Interrupts
>   r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Flt        cow    2006 total
>           1  47       28k   19 2691    6 2036    2        zfod        atkbd0 1
>                                                           ozfod     4 ata0 irq14
> 24.7%Sys  18.6%Intr  0.7%User  0.0%Nice 55.9%Idle        %ozfod       nfe0 irq23
> |    |    |    |    |    |    |    |    |    |    |       daefr  2001 cpu0: time
> ============++++++++++                                    prcfr       igb0 256
>                                         10 dtbuf        2 totfr     1 igb0 257
> Namei     Name-cache   Dir-cache    100000 desvn          react       igb0 258
>    Calls    hits   %    hits   %     84106 numvn          pdwak
>       20      20 100                 24824 frevn          pdpgs
>                                                           intrn
> Disks   ad0                                        373944 wire
> KB/t  16.00                                        292104 act
> tps       4                                        417564 inact
> MB/s   0.07                                           200 cache
> %busy     0                                        938228 free
> 
> I get bad results (((
> [ ID] Interval       Transfer     Bandwidth
> [  4]  0.0-10.3 sec    180 MBytes    147 Mbits/sec
> 

nfe(4) controllers are one of rare gigabit controllers that lacks
efficient interrupt moderation mechanism. So it's normal to see
high number of interrupts under load.
hz controls how frequently checks controller's RX/TX queue so you
may have to increase hz to get reasonable performance under high
network load with polling(4). One of important performance factor
for NIC is how many frames should be processed for given network
load pattern. If you have to process more frames you have to
increase hz in polling(4).
I don't know what nfe(4) controller you have but it seems it's
somewhat low-end model because MSI/MSIX is not used at all.