tuning for high connection rates

[Philipp Wuensche]

Mike Tancsa wrote:
> At 12:23 PM 12/5/2007, Philipp Wuensche wrote:
>> Mike Tancsa wrote:
>> > At 07:14 PM 12/4/2007, Philipp Wuensche wrote:
>> >
>> >> The debug output of em0 looks like this:
>> >>
>> >> em0: CTRL = 0x40140248 RCTL = 0x8002
>> >> em0: Packet buffer = Tx=20k Rx=12k
>> >> em0: Flow control watermarks high = 10240 low = 8740
>> >> em0: tx_int_delay = 66, tx_abs_int_delay = 66
>> >> em0: rx_int_delay = 32, rx_abs_int_delay = 66
>> >> em0: fifo workaround = 0, fifo_reset_count = 0
>> >> em0: hw tdh = 183, hw tdt = 183
>> >> em0: hw rdh = 139, hw rdt = 139
>> >> em0: Num Tx descriptors avail = 223
>> >> em0: Tx Descriptors not avail1 = 6225
>> >> em0: Tx Descriptors not avail2 = 3
>> >> em0: Std mbuf failed = 0
>> >> em0: Std mbuf cluster failed = 0
>> >> em0: Driver dropped packets = 0
>> >> em0: Driver tx dma failure in encap = 0
>> >
>> > If you do a
>> > sysctl -w dev.em.0.stats=1
>> >
>> > It will spit the nic stats to syslog.  What are the results ?
>>
>> em0: Excessive collisions = 0
>> em0: Sequence errors = 0
>> em0: Defer count = 0
>> em0: Missed Packets = 12876719
>> em0: Receive No Buffers = 5950326
>> em0: Receive Length Errors = 0
>> em0: Receive errors = 0
>> em0: Crc errors = 0
>> em0: Alignment errors = 0
>> em0: Collision/Carrier extension errors = 0
>> em0: RX overruns = 56256
>> em0: watchdog timeouts = 0
>> em0: XON Rcvd = 0
>> em0: XON Xmtd = 0
>> em0: XOFF Rcvd = 0
>> em0: XOFF Xmtd = 0
>> em0: Good Packets Rcvd = 3384408375
>> em0: Good Packets Xmtd = 2657550034
>> em0: TSO Contexts Xmtd = 6925441
>> em0: TSO Contexts Failed = 0
>>
>>
>> > Also,
>> > what does ifconfig em0 look like (i.e. what options do you have set,
>> > speed etc)
>>
>> Its on a 100Mbit/s switch, we haven't changed the options:
>>
>> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>        
>> options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
>>         media: Ethernet autoselect (100baseTX <full-duplex>)
> 
> Some people have reported that TSO is a "bad thing" on 100Mb.  Can you
> try disabling that ? Also, you seem to have a lot of RX overruns and
> missed packets such that the nic cannot process things fast enough.  I
> havent done any benchmarks yet, but the Yandex people claim their
> modified EM driver can handle higher PPS rates than the stock em
> driver.    Not sure if they have a RELENG_7 port or not but they might
> have some insight.

After switching to net.isr.direct=0 and 346609775 good packets later, RX
overruns haven't increased by one! Thats nice. Still interrupt is using
up the CPU. I'm not quite sure if polling would help now!?

current stats:
em0: Excessive collisions = 0
em0: Sequence errors = 0
em0: Defer count = 0
em0: Missed Packets = 12885592
em0: Receive No Buffers = 5953357
em0: Receive Length Errors = 0
em0: Receive errors = 0
em0: Crc errors = 0
em0: Alignment errors = 0
em0: Collision/Carrier extension errors = 0
em0: RX overruns = 56296
em0: watchdog timeouts = 0
em0: XON Rcvd = 0
em0: XON Xmtd = 0
em0: XOFF Rcvd = 0
em0: XOFF Xmtd = 0
em0: Good Packets Rcvd = 3731018150
em0: Good Packets Xmtd = 2936659631
em0: TSO Contexts Xmtd = 7734211
em0: TSO Contexts Failed = 0

We will try disabling TSO to see if anything changes.

greetings,
cryx