increasing em(4) buffer sizes

[Eugene Grosbein]

On 20.05.2010 11:33, rihad wrote:

>> First, read: http://www.intel.com/design/network/applnots/ap450.htm
>> You'll see you may be restricted with your NIC's chip capabilities.
>>
> Likely sooner than later these cards will be upgraded to 10 GigE ones, I 
> just want to make sure that the delays imposed by traversing the 
> firewall never cause traffic drops on input.

That document is needed to understand tunnables mentioned later.

>> There are loader tunnables, set them in /etc/loader.conf:
> 
> Do you mean /boot/loader.conf ?

Yes, /boot/loader.conf

>>
>> hw.em.rxd=4096
>> hw.em.txd=4096
>>
> 
> BTW, I can't read the current value:
> $ sysctl hw.em.rxd
> sysctl: unknown oid 'hw.em.rxd'
> $
> 
> Is this a write-only value? :)

They are loader tunnables that have no sysctl variables
with same names - as an example of such rare case :-)
They are documented in man em, though.

>> The price is amount of kernel memory the driver may consume.
>> Maxumum MTU=16110 for em(4), so it can consume about 64Mb of kernel memory
>> for that long input buffer, in theory.
>>
>> Some more useful tunnables for loader.conf:
>>
>> dev.em.0.rx_int_delay=200
>> dev.em.0.tx_int_delay=200
>> dev.em.0.rx_abs_int_delay=200
>> dev.em.0.tx_abs_int_delay=200
>> dev.em.0.rx_processing_limit=-1
>>
> So this interrupt delay is the much talked about interrupt moderation? 
> Thanks, I'll try them. Is there any risk the machine won't boot with 
> them if rebooted remotely?

Yes, for hw.em.rxd/hw.em.txd only. em interfaces will not function
if you set these two too high for your nic chip, so read man em
and mentioned Intel document carefully.

It's pretty safe to play with other tunnables.

>> Alternatively, you may try kernel polling (ifconfig em0 polling)
>> with other tunnables:
>>
>> kern.hz=4000				# for /boot/loader.conf
>> kern.polling.burst_max=1000		# for /etc/sysctl.conf
>> kern.polling.each_burst=500
>>
> Wow, I successfully used polling a couple of years ago when the load was 
> low, but then I read some posting on this list claiming that Intel cards 
> have the ability to do fast-interrupts (interrupt moderation),

Yes, they have but some tuning for interrupt moderation may be needed.

> but for that DEVICE_POLLING needs to be out of the kernel.

That may be true long time ago but not for 7.x afaik.
Just use ifconfig to enable/disable polling per-device.

> So I scratched it and
> rebuilt the kernel for no apparent reason. Maybe you're right, polling 
> would've worked just fine, so I may go back to that too.

For ng_source-based outgoing flood of 64 byte UDP packets in test lab
I've obtained best results with polling enabled using FreeBSD 7.1

For Pentium-D 2.8Ghz dualcore and desktop motherboard Intel D975XBX
with integrated NIC:

em0 at pci0:4:0:0: class=0x020000 card=0x30a58086 chip=0x109a8086 rev=0x00
hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82573L Intel PRO/1000 PL Network Adaptor'
    class      = network
    subclass   = ethernet

I've got 742359pps / 772Mbps at wire. Note, that was not forwarding
speed, only outgoing UDP flood. The CPU horsepower was limiting point,
one core was fully loaded and another one idle.

Eugene Grosbein