packet drop with intel gigabit / marwell gigabit

[Gary Thorpe]

Jin Guojun [VFFS] wrote:
> OxY wrote:
> 
>>
>> ----- Original Message ----- From: "Jin Guojun [VFFS]" <g_jin at lbl.gov>
>> To: "OxY" <oxy at field.hu>
>> Cc: <freebsd-performance at freebsd.org>
>> Sent: Monday, March 20, 2006 4:05 AM
>> Subject: Re: packet drop with intel gigabit / marwell gigabit
>>
>>>>> ....
>>>>> First let's clear the notation -- Is 30MB/s (MBytes/s) = 240Mb/s 
>>>>> (Mbit/s) or MB/s means Mbits/s
>>>>> If  MB/s is MBytes/s and you also write this amount data to a disk, 
>>>>> plus other traffic on fxp0 to disk too,
>>>>> then your problem may be bonded by memory bandwidth because CPU 
>>>>> utilization is low:
>>>>>    (240 + 24~32) x 2 is about 535 Mbit/s (some chipset/motherboard 
>>>>> has low memory BW for AMD)
>>>>> If this is true, then this no thing you can tune. What does the 
>>>>> chipset (Motherboard) this machine have?
>>>>
>>>>
>>>>
>>>>
>>>> 30MB/s is Megabytes/sec, currently i have 18-20MB/s peak and 15MB/s 
>>>> avg.
>>>> it's not 535Mbit/s, because i only download it to my machine, no 
>>>> upload.
>>>> disks are different from apache disks, these disks have own 
>>>> controller in one pci slot.
>>>> the packet drop is 5-7% at 200Mbit iperf test, 100Mbit drop is 
>>>> around zero.
>>>> i have <ASUS A7V8X> on motherboard which has VIA KT400 northbridge
>>>> http://uk.asus.com/products4.aspx?modelmenu=2&model=226&l1=3&l2=13&l3=62 
>>>>
>>>
>>>
>>>
>>> Yes, this is one of problem chipset. I bought one about 3 years ago.
>>> After one day testing, I returned it for changing a A7V600 (VIA KT600 
>>> chipset),
>>> which is 30% more memory bandwidth than KT400. A7V600 can only 
>>> receive max
>>> 604 Mb/s TCP, so You can imagine what the KT400 can do :-)
>>> I do not have a record (because it is too bad), but taking minimum 
>>> 25% off,
>>> it probably about 420-430 Mb/s (50MB/s). Now you can do the math when 
>>> the
>>> machine also writing data to a disk (assume disk a fast enough). I 
>>> would expect
>>> 2/3 of 430 Mb/s, which is about 280~290 Mb/s (35 MB/s).
>>> If you experiment these numbers, you are at there. No improvement you 
>>> can make
>>> further.
>>
>>
>>
>> i have doubts, because when i have 3-4MB/s traffic on fxp0 then em0 peak
>> is 18MB/s, but when fxp0 is almost idle, have 500kB/s traffic, then 
>> em0 can only
>> do 20MB/s..
> 
> 
> Since you did not get anything better than 35MB/s, then, what is your 
> doubt --
>    the maximum I/O A7V8X can do?
> 
> The 35 MB/s is the theoretical ceiling based on 2100+ CPU. 2000+ will be 
> slower.
> In previous email, you mentioned you had 240 Mb/s (30 MB/s) on em0 with 
> some
> traffic on fxp0, it is pretty much close to your hardware physical 
> limitation.

I thought all modern NICs used bus mastering DMA i.e. not dependent on CPU for data transfers? In addition, the available memory bandwidth for modern CPU's/systems is well over 100 MB/s. DDR400 is 400 MB/s (megabytes per second). Bus mastering DMA will be limited by the memory or IO bus bandwidth primarily. The system bus bandwidth cannot be the problem either: his motherboard's lowest front side bus speed is 200 MHz * 64-bit width = 1.6 GB/s (gigabytes per second) of peak system bus bandwidth.

The limitation of 32-bit/33 MHz PCI is 133 MB/s (again, megabytes not bits) maximum. Gigabit ethernet requires 125 MB/s (not Mb/s) maximum bandwidth: 32/33 PCI has enough for bursts but bus contention with disk bandwidth will reduce the sustained bandwidth. The motherboard in question has an option for integrated gigabit LAN which may bypass the shared PCI bus altogether (or it might not).

Anyway, the original problem was packet loss and not bandwidth. His CPU is mostly idle, so that cannot be the reason for packet loss. If 32/33 PCI can sustain 133 MB/s then it cannot be a problem because he needs 
less than this. If it cannot, then packets will arrive too fast from the network before they can be moved from the board into memory and would cause the packet loss. Otherwise, his system is capable of achieving what he wants in theory and the suboptimal behavior may be due to hardware (e.g. PCI bus bandwidth not being able to reach 133 MB/s sustained) or software limitations (e.g inefficient operating system).

> Forget drop in this figure, because this demonstrated how much hardware 
> can do,
> rather than lossless transmission.
> Once you have determined the ceiling, you need to keep a margin for 
> lossless Tx.
> for other overhead, such as context switch, etc.
> 20 MB/s is not good enough for this board, you may expect 28-30 MB/s with
> fine tuning. Unless you will be happy with 28 MB/s, it does not make 
> sense to
> waste time to try to bump I/O above 30 MB/s for your application if you 
> have
> another motherboard.
> Again, this motherboard is designed for entertainment boxes not for network
> I/O based applications.
> 
>    -Jin
> _______________________________________________
> freebsd-performance at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to 
> "freebsd-performance-unsubscribe at freebsd.org"
> 
>