5.3-STABLE/alpha: re(4) slowdown

[Scott Long]

Christian Weisgerber wrote:
> Robert Watson:
> 
> 
>>Could you use a tool like netperf to see whether the slowdown is specific
>>to TCP, or affects UDP also?  There have been some TCP tweaks and
>>bugfixes, and this would help isolate that.  Seeing the results of a
>>netperf run with the UDP_RR and UDP_STREAM tests in the "before" and
>>"after" scenarios would be helpful.
> 
> 
> 5.3-STABLE (RELENG_5) on alpha; GENERIC plus if_re.ko.
> The transmitting side is OpenBSD-current/amd64 (GENERIC, sk).
> The network is a no-name dumb gigabit switch.  No jumbo frames.
> 
> Results from five runs in a row.
> 
> Kernel as of December 4:
> 
>     netperf -H 172.16.0.3 -t TCP_STREAM
>     TCP STREAM TEST to 172.16.0.3
>     Recv   Send    Send                          
>     Socket Socket  Message  Elapsed              
>     Size   Size    Size     Time     Throughput  
>     bytes  bytes   bytes    secs.    10^6bits/sec  
> 
>      65536  16384  16384    10.00     142.11   
>      65536  16384  16384    10.01     142.58   
>      65536  16384  16384    10.01     137.59   
>      65536  16384  16384    10.01     135.01   
>      65536  16384  16384    10.01     139.23   
> 
> December 22:
> 
>     netperf -H 172.16.0.3 -t TCP_STREAM
>     TCP STREAM TEST to 172.16.0.3
>     Recv   Send    Send                          
>     Socket Socket  Message  Elapsed              
>     Size   Size    Size     Time     Throughput  
>     bytes  bytes   bytes    secs.    10^6bits/sec  
> 
>      65536  16384  16384    10.08       2.57   
>      65536  16384  16384    10.66       4.50   
>      65536  16384  16384    10.06       6.16   
>      65536  16384  16384    12.10       6.73   
>      65536  16384  16384    10.10       2.84   
> 
> Both the switch lights and the feeling from, say, scrolling in a
> Firefox window from a remote host suggest that there are short
> periods where no packets are transmitted.
> 
> 
> December 4:
> 
>     netperf -H 172.16.0.3 -t UDP_STREAM
>     UDP UNIDIRECTIONAL SEND TEST to 172.16.0.3
>     Socket  Message  Elapsed      Messages                
>     Size    Size     Time         Okay Errors   Throughput
>     bytes   bytes    secs            #      #   10^6bits/sec
> 
>       9216    9216   10.00       22783 219677     167.94
>      42080           10.00           4              0.03
> 
> (The large socket size tests all fail, which I presume is irrelevant in
> this context.  Figures from further runs:)
> 
>       9216    9216   10.00       22285 225815     164.22
>       9216    9216   10.00       23852 224714     175.78
>       9216    9216   10.01       23635 225654     174.13
>       9216    9216   10.01       24139 224133     177.86
> 
> December 22:
> 
>     netperf -H 172.16.0.3 -t UDP_STREAM
>     UDP UNIDIRECTIONAL SEND TEST to 172.16.0.3
>     Socket  Message  Elapsed      Messages                
>     Size    Size     Time         Okay Errors   Throughput
>     bytes   bytes    secs            #      #   10^6bits/sec
> 
>       9216    9216   10.01       24410 221169     179.84
>      42080           10.01          20              0.15
> 
> I would have said that UDP is not affected, except that at this
> point the FreeBSD box locked up solid.
> 

I suspect that the buffers are being bounced all over the place in the
if_re driver.  Can you send me the output of 'sysctl hw.busdma' after
the system has been under load?

Scott