increasing em(4) buffer sizes

[rihad]

On 05/20/2010 10:19 AM, Eugene Grosbein wrote:
> On 20.05.2010 11:33, rihad wrote:
>
>> BTW, I can't read the current value:
>> $ sysctl hw.em.rxd
>> sysctl: unknown oid 'hw.em.rxd'
>> $
>>
>> Is this a write-only value? :)
>
> They are loader tunnables that have no sysctl variables
> with same names - as an example of such rare case :-)
> They are documented in man em, though.
>
>>> The price is amount of kernel memory the driver may consume.
>>> Maxumum MTU=16110 for em(4), so it can consume about 64Mb of kernel memory
>>> for that long input buffer, in theory.
>>>
Oh, if only this would solve the problem :) I understand the packets 
can't stay in the buffer for long, so an extremely large input buffer 
holding on to packets for several seconds is practically useless for 
TCP, as it will retransmit by that time. But packets dropped on input 
feel just as bad for the end user due to slower web browsing.

>>> Some more useful tunnables for loader.conf:
>>>
>>> dev.em.0.rx_int_delay=200
>>> dev.em.0.tx_int_delay=200
>>> dev.em.0.rx_abs_int_delay=200
>>> dev.em.0.tx_abs_int_delay=200
>>> dev.em.0.rx_processing_limit=-1
>>>
>> So this interrupt delay is the much talked about interrupt moderation?
>> Thanks, I'll try them. Is there any risk the machine won't boot with
>> them if rebooted remotely?
>
> Yes, for hw.em.rxd/hw.em.txd only. em interfaces will not function
> if you set these two too high for your nic chip, so read man em
> and mentioned Intel document carefully.
>
> It's pretty safe to play with other tunnables.
>
Can I even set the mentioned dev.em.0.* settings on a live system 
remotely? You seem to also have an em card, have you tried doing that? 
I've already set dev.em.0.rx_processing_limit=-1, but to see its effects 
I'll have to wait a couple of ours for the load to get heavier.

>> So I scratched it and
>> rebuilt the kernel for no apparent reason. Maybe you're right, polling
>> would've worked just fine, so I may go back to that too.
>
> For ng_source-based outgoing flood of 64 byte UDP packets in test lab
> I've obtained best results with polling enabled using FreeBSD 7.1
>
> For Pentium-D 2.8Ghz dualcore and desktop motherboard Intel D975XBX
> with integrated NIC:
>
> em0 at pci0:4:0:0: class=0x020000 card=0x30a58086 chip=0x109a8086 rev=0x00
> hdr=0x00
>      vendor     = 'Intel Corporation'
>      device     = '82573L Intel PRO/1000 PL Network Adaptor'
>      class      = network
>      subclass   = ethernet
>

Ours is:
em0 at pci0:5:0:0: class=0x020000 card=0x346c8086 chip=0x10968086 rev=0x01 
hdr=0x00
     vendor     = 'Intel Corporation'
     device     = 'PRO/1000 EB Network Connection'
     class      = network
     subclass   = ethernet

      hw.em.rxd
              Number of receive descriptors allocated by the driver.  The
              default value is 256.  The 82542 and 82543-based adapters can
              handle up to 256 descriptors, while others can have up to 
4096.

We have a EB with an unknown numeric ID. The only EB in the em manual is 
82546EB, so I guess it does support hw.em.rxd=4096. I now have several 
things to try (rxd, polling).

> I've got 742359pps / 772Mbps at wire. Note, that was not forwarding
> speed, only outgoing UDP flood. The CPU horsepower was limiting point,
> one core was fully loaded and another one idle.
>
I believe FreeBSD 8.x is better in this regard as it spreads the ISR 
load across multiple cores evenly.

> Eugene Grosbein
>
>