increasing em(4) buffer sizes

[rihad]

Hi there,

We have a FreeBSD 7.2 Intel Server System 4GB RAM box doing traffic 
shaping and accounting. It has two em gigabit interfaces: one used for 
input, the other for output, servicing around 500-600 mbps load through 
it. Traffic limiting is accomplished by dynamically setting up IPFW 
pipes, which in turn work fine for our per-user traffic accounting needs 
thanks to byte counters. So the firewall is basically a longish string 
of pipe rules. This worked fine when the number of online users was low, 
but now, as we've slowly begun servicing 2-3K online users netstat -i's 
Ierrs column is growing at a rate of 5-15K per hour for em0, the 
interface used for input. Apparently searching through the firewall 
linearly for _each_ arriving packet locks the interface for the duration 
of the search (even though net.isr.direct=0), so some packets are 
periodically dropped on input. To mitigate the problem I've set up a 
two-level hash by means of skipto rules, dropping the number of up to 
several thousand rules to be searched for each packet to a mere 85 max, 
but the rate of Ierrs has only increased to 40-50K per hour, I don't 
know why. I've also tried setting these sysctls:

hw.intr_storm_threshold=10000
dev.em.0.rx_processing_limit=3000

but they didn't help at all. BTW, the other current settings are:
kern.hz=4000
net.inet.ip.fw.verbose=0
kern.ipc.nmbclusters=111111
net.inet.ip.fastforwarding=1
net.inet.ip.dummynet.io_fast=1
net.isr.direct=0
net.inet.ip.intr_queue_maxlen=5000

net.inet.ip.intr_queue_drops is always zero.

I think the problem lies in the buffer size of em not being large enough 
to buffer the packets as they're arriving. I looked in 
/sys/dev/e1000/if_em.c and found this:

in em_attach():
         adapter->rx_buffer_len = 2048;

and later in em_initialize_receive_unit():
         switch (adapter->rx_buffer_len) {
         default:
         case 2048:
                 rctl |= E1000_RCTL_SZ_2048;
                 break;
         case 4096:
                 rctl |= E1000_RCTL_SZ_4096 |
                     E1000_RCTL_BSEX | E1000_RCTL_LPE;
                 break;
         case 8192:
                 rctl |= E1000_RCTL_SZ_8192 |
                     E1000_RCTL_BSEX | E1000_RCTL_LPE;
                 break;
         case 16384:
                 rctl |= E1000_RCTL_SZ_16384 |
                     E1000_RCTL_BSEX | E1000_RCTL_LPE;
                 break;
         }


So apparently the default buffer size is 2048 bytes, and as much as 
16384 is supported. But at what price? Those constants do look 
suspicious. Can I blindly change rx_buffer_len in em_attach()? Sorry, 
I'm not a kernel hacker :(

Thanks for reading and thanks for any tips. Any help is much appreciated.