tuning for high connection rates

[Mike Tancsa]

At 12:23 PM 12/5/2007, Philipp Wuensche wrote:
>Mike Tancsa wrote:
> > At 07:14 PM 12/4/2007, Philipp Wuensche wrote:
> >
> >> The debug output of em0 looks like this:
> >>
> >> em0: CTRL = 0x40140248 RCTL = 0x8002
> >> em0: Packet buffer = Tx=20k Rx=12k
> >> em0: Flow control watermarks high = 10240 low = 8740
> >> em0: tx_int_delay = 66, tx_abs_int_delay = 66
> >> em0: rx_int_delay = 32, rx_abs_int_delay = 66
> >> em0: fifo workaround = 0, fifo_reset_count = 0
> >> em0: hw tdh = 183, hw tdt = 183
> >> em0: hw rdh = 139, hw rdt = 139
> >> em0: Num Tx descriptors avail = 223
> >> em0: Tx Descriptors not avail1 = 6225
> >> em0: Tx Descriptors not avail2 = 3
> >> em0: Std mbuf failed = 0
> >> em0: Std mbuf cluster failed = 0
> >> em0: Driver dropped packets = 0
> >> em0: Driver tx dma failure in encap = 0
> >
> > If you do a
> > sysctl -w dev.em.0.stats=1
> >
> > It will spit the nic stats to syslog.  What are the results ?
>
>em0: Excessive collisions = 0
>em0: Sequence errors = 0
>em0: Defer count = 0
>em0: Missed Packets = 12876719
>em0: Receive No Buffers = 5950326
>em0: Receive Length Errors = 0
>em0: Receive errors = 0
>em0: Crc errors = 0
>em0: Alignment errors = 0
>em0: Collision/Carrier extension errors = 0
>em0: RX overruns = 56256
>em0: watchdog timeouts = 0
>em0: XON Rcvd = 0
>em0: XON Xmtd = 0
>em0: XOFF Rcvd = 0
>em0: XOFF Xmtd = 0
>em0: Good Packets Rcvd = 3384408375
>em0: Good Packets Xmtd = 2657550034
>em0: TSO Contexts Xmtd = 6925441
>em0: TSO Contexts Failed = 0
>
>
> > Also,
> > what does ifconfig em0 look like (i.e. what options do you have set,
> > speed etc)
>
>Its on a 100Mbit/s switch, we haven't changed the options:
>
>em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
>         media: Ethernet autoselect (100baseTX <full-duplex>)

Some people have reported that TSO is a "bad thing" on 100Mb.  Can 
you try disabling that ? Also, you seem to have a lot of RX overruns 
and missed packets such that the nic cannot process things fast 
enough.  I havent done any benchmarks yet, but the Yandex people 
claim their modified EM driver can handle higher PPS rates than the 
stock em driver.    Not sure if they have a RELENG_7 port or not but 
they might have some insight.

         ---Mike