7.1-PRERELEASE : bad network performance (nfe0)

Mon Sep 29 14:13:02 UTC 2008

Dear Pyun,

thanx for your prompt answer (as usual).

Pyun YongHyeon <pyunyh at gmail.com> writes:

> On Sat, Sep 27, 2008 at 11:21:00PM +0200, Arno J. Klaassen wrote:
>  > 
>  > 
>  > Hello,
>  > 
>  > I've serious network performance problems on a HP Turion X2
>  > based brand new notebook; I only used a 7-1Beta CD and
>  > 7-STABLE on this thing.
>  > 
>  > Scp-ing ports.tgz from a rock-stable 7-STABLE server to it gives :
>  > 
>  >   # scp -p ports.tgz login at mv:/tmp/
>  >     ports.tgz                             100%   98MB  88.7KB/s   18:49 
>  > 
>  > (doing the same thing by copy from an nfs-mounted disk even
>  >  takes mores than an hour ...)
>  > 
>  > 
>  > Doing a top(1) aside, just shows the box 100% idle :
>  > 
>  >   PID USERNAME PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
>  >    12 root     171 ki31     0K    16K CPU0   0  38:55 100.00% idle: cpu0
>  >    11 root     171 ki31     0K    16K RUN    1  38:55 100.00% idle: cpu1
>  >    13 root     -32    -     0K    16K WAIT   0   0:02  0.00% swi4: clock sio
>  >    29 root     -68    -     0K    16K -      0   0:00  0.00% nfe0 taskq
>  >    34 root     -64    -     0K    16K WAIT   1   0:00  0.00% irq23: atapci1
>  >  1853 root       8    0  7060K  1920K wait   0   0:00  0.00% sh
>  >   878 nono      44    0  8112K  2288K CPU1   1   0:00  0.00% top
>  >   884 root       8    -     0K    16K -      1   0:00  0.00% nfsiod 0
>  >     4 root      -8    -     0K    16K -      1   0:00  0.00% g_down
>  >    16 root     -16    -     0K    16K -      1   0:00  0.00% yarrow
>  >    46 root      20    -     0K    16K syncer 0   0:00  0.00% syncer
>  >     3 root      -8    -     0K    16K -      0   0:00  0.00% g_up
>  >    30 root     -68    -     0K    16K -      0   0:00  0.00% fw0_taskq
>  > 
>  > 
>  > I tested :
>  > 
>  >   Update Bios
>  >   ULE /4BSD
>  >   PREEMPTION on/off
>  >   PREEMPTION + IPI_PREEMPTION
>  >   hw.nfe.msi[x]_disable=1
>      ^^^^^^^^^^^^^^^^^^^^^^^
>      This has no effect as MCP65 lacks MSI/MSI-X capability. 
>  > 
>  > All don't seem to matter to the problem.
>  > 
>  > I put two tcpdumps (server and client during another scp(1) ) on 
>  >   http://bare.snv.jussieu.fr/temp/tcpdump-s1518.server
>  >   http://bare.snv.jussieu.fr/temp/tcpdump-s1518.client
>  > 
>  > I'm far from an expert on TCP/IP, but wireshark "expert info" shows
>  > lots of sequences like :
>  > 
>  >   TCP Previous segment lost
>  >   TCP Duplicate ACK 1
>  >   TCP Window update
>  >   TCP Duplicate ACK 2
>  >   TCP Duplicate ACK 3
>  >   TCP Duplicate ACK 4
>  >   TCP Duplicate ACK 5
>  >   TCP Fast retransmission (suspected)
>  >   TCP ...
>  >   TCP Out-of-Order segment
>  >   TCP ...
>  > 
>  > 
>  > As usual, feel free to contact me for further info/tests.
>  > 
> 
> AFAIK it seems that you're the first one that reports poor
> performance issue of MCP65.

someone must be ;) no kiddin, I am not convinced this is (only)
a driver issue (cf. "bad NFS/UDP performance" thread on -hackers).

I just have no experience on this notebook, so I can't say " it worked
great before" and my only other 7-stable-amd64 I have does not
show the probs, having a cheap re0 *and* being UP.

> MCP65 has no checksum offload/TSO
> capability so nfe(4) never try to take advantage of the hardware
> capability. So you should have no checksum offload/TSO related
> issue here. 
> Also note, checking network performance with scp(1) wouldn't
> show real numbers as scp(1) may involve other system activities.
> Use one of network benchmark programs in ports(e.g.
> benchmarks/netperf) to measure network performance.

quite funny (even taken with lots of salt since the LAN is used
for "normal work" as well in parallel, but differences are
rather significant) :

I test to same server (7-stable-amd64 from Jun  7 (using
nfe0 as well btw, but another chip), either from a
6-stable-x86 (Jul 14, sk0) or the notebook (7-stable-x64 below), using

 for i in <SOME-TESTS>  ; do
    echo $i; /usr/local/bin/netperf -H push -i 4,2 -I 95,10 -t $i;  echo;
 done

streaming results are OK for both :

TCP_STREAM
              Throughput  
              10^6bits/sec  

6-stable-x86  349.57   
7-stable-x64  939.47 

UDP_STREAM
              Throughput
              10^6bits/sec

6-stable-x86  388.45
7-stable-x64  947.89    

However, the "request/respones" tests are awfull for my notebook 
(test repeated on the notebook for the sake of conviction) :

TCP_RR
              Trans.
              Rate          
              per sec        

6-stable-x86  9801.58                
7-stable-x64   137.61
7-stable-x64    89.35
7-stable-x64   102.29

TCP_CRR
              Trans.
              Rate                                        
              per sec   

6-stable-x86  4520.98   
7-stable-x64     7.00
7-stable-x64     8.10
7-stable-x64    18.49

UDP_RR
              Trans.
              Rate                                        
              per sec   

6-stable-x86  9473.20   
7-stable-x64     9.60
7-stable-x64     0.90
7-stable-x64     0.10

I can send you complete results if wanted.

> Other possible cause of issue could be link speed/duplex mismatch
> or excessive MAC control frames(e.g. pause frames). Does nfe(4)
> agree on resolved speed/duplex with link partner?

yes (1000baseTX <full-duplex>)

> If they all agree on resolved speed/duplex, would you check number
> of pause frames sent/received from link partner? Even though MCP65
> supports hardware MAC statistics for pause frames nfe(4) has no
> support code yet so you may have to resort to managed switch that
> can show Tx/Rx statistics of each port.

aargh; I do have a Netgear GS724TS around where I can connect it to.
This thing should be manageable, but give me some time to
find out how ....

Thanx, Arno