Intel 82574L Gigabit Ethernet Controller

Shtorm admin at shtorm.com
Tue Jul 6 08:25:37 UTC 2010


On Mon, 2010-07-05 at 22:02 -0700, Jack Vogel wrote:
> Cleaner in the Makefile, let me know how it goes.
> 
> Jack
> 
> 
> On Mon, Jul 5, 2010 at 12:06 PM, Yuriy A. Korobko
> <administrator at shtorm.com>wrote:
> 
> > On Mon, 2010-07-05 at 11:11 -0700, Jack Vogel wrote:
> > > Are  you defining 'EM_MULTIQUEUE', its off by default and needs to be
> > > defined somewhere by you.
> > >
> > > You will only see the two queues used if you have two different
> > connections
> > > operating at once.
> > >
> > > Jack

Recompiled module (actually whole kernel) with EM_MULTIQUEUE, but got a
some watchdog timeouts. 

Here is example of netstat when it happens:

   2346     0     0     583370       2642     0    2988379     0
   2250     0     0     550961       2384     0    2835276     0
   2634     0     0     703410       2733     0    2971417     0
   2741     0     0     695884       2748     0    3061573     0
   2811     0     0     618520       2274     0    3273868     0
   3220     0     0     687145       2068     0    3417857     0
    901     0     0     231694        166     0    1565839     0
    752     0     0     216135          0     0     270247     0
    685     0     0     208745          0     0     243442     0
    713     0     0     178089          0     0     230472     0
    735     0     0     159555          0     0     178435     0
    616     0     0     145222          0     0     179022     0
         input          (em1)           output
packets  errs idrops      bytes    packets  errs      bytes colls
    616     0     0     129608          0     0     120929     0
    659     0     0     113806          0     0     105707     0
    622     0     0     106247          0     0     107825     0
    645     0     0     101593          0     0      38023     0
    483     0     0      61681          0     0       8547     0

Watchdog timeout on em1 message on console 

      0     0     0          0          0     1      19095     0
      0     0     0          0          0     0        222     0
      0     0     0          0          0     0          0     0
    243     0     0      30224       1783     0    1708875     0
    576     0     0      62350        105     0       7710     0
    467     0     0      49758          4     0       4388     0
    500     0     0      53415         23     0      24743     0
    437     0     0      50135         21     0      16733     0

After this traffic increased again up to 3 kpps and another watchdog
timeout happened.

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
	options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
	ether 00:30:48:bc:ab:ca
	inet x.x.x.x netmask 0xffffffe0 broadcast x.x.x.y
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active

em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
	options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
	ether 00:30:48:bc:ab:cb
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active

em1 have 30 vlans and mpd5 as pppoe server listening on each vlan.

Here is sysctl dev.em

dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.0.5
dev.em.0.%driver: em
dev.em.0.%location: slot=0 function=0
dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9
subdevice=0x10d3 class=0x020000
dev.em.0.%parent: pci1
dev.em.0.nvm: -1
dev.em.0.rx_int_delay: 200
dev.em.0.tx_int_delay: 200
dev.em.0.rx_abs_int_delay: 500
dev.em.0.tx_abs_int_delay: 500
dev.em.0.rx_processing_limit: 100
dev.em.0.link_irq: 2
dev.em.0.mbuf_alloc_fail: 0
dev.em.0.cluster_alloc_fail: 0
dev.em.0.dropped: 0
dev.em.0.tx_dma_fail: 0
dev.em.0.fc_high_water: 18432
dev.em.0.fc_low_water: 16932
dev.em.0.mac_stats.excess_coll: 0
dev.em.0.mac_stats.symbol_errors: 0
dev.em.0.mac_stats.sequence_errors: 0
dev.em.0.mac_stats.defer_count: 0
dev.em.0.mac_stats.missed_packets: 0
dev.em.0.mac_stats.recv_no_buff: 0
dev.em.0.mac_stats.recv_errs: 0
dev.em.0.mac_stats.crc_errs: 0
dev.em.0.mac_stats.alignment_errs: 0
dev.em.0.mac_stats.coll_ext_errs: 0
dev.em.0.mac_stats.rx_overruns: 0
dev.em.0.mac_stats.watchdog_timeouts: 0
dev.em.0.mac_stats.xon_recvd: 0
dev.em.0.mac_stats.xon_txd: 0
dev.em.0.mac_stats.xoff_recvd: 0
dev.em.0.mac_stats.xoff_txd: 0
dev.em.0.mac_stats.total_pkts_recvd: 732916
dev.em.0.mac_stats.good_pkts_recvd: 732916
dev.em.0.mac_stats.bcast_pkts_recvd: 1238
dev.em.0.mac_stats.mcast_pkts_recvd: 0
dev.em.0.mac_stats.rx_frames_64: 84622
dev.em.0.mac_stats.rx_frames_65_127: 128831
dev.em.0.mac_stats.rx_frames_128_255: 34037
dev.em.0.mac_stats.rx_frames_256_511: 30206
dev.em.0.mac_stats.rx_frames_512_1023: 24919
dev.em.0.mac_stats.rx_frames_1024_1522: 430301
dev.em.0.mac_stats.good_octets_recvd: 0
dev.em.0.mac_stats.good_octest_txd: 0
dev.em.0.mac_stats.total_pkts_txd: 678078
dev.em.0.mac_stats.good_pkts_txd: 678078
dev.em.0.mac_stats.bcast_pkts_txd: 109
dev.em.0.mac_stats.mcast_pkts_txd: 0
dev.em.0.mac_stats.tx_frames_64: 324803
dev.em.0.mac_stats.tx_frames_65_127: 196866
dev.em.0.mac_stats.tx_frames_128_255: 47362
dev.em.0.mac_stats.tx_frames_256_511: 31917
dev.em.0.mac_stats.tx_frames_512_1023: 44997
dev.em.0.mac_stats.tx_frames_1024_1522: 32133
dev.em.0.mac_stats.tso_txd: 0
dev.em.0.mac_stats.tso_ctx_fail: 0
dev.em.0.interrupts.asserts: 0
dev.em.0.interrupts.rx_pkt_timer: 0
dev.em.0.interrupts.rx_abs_timer: 0
dev.em.0.interrupts.tx_pkt_timer: 0
dev.em.0.interrupts.tx_abs_timer: 0
dev.em.0.interrupts.tx_queue_empty: 0
dev.em.0.interrupts.tx_queue_min_thresh: 0
dev.em.0.interrupts.rx_desc_min_thresh: 0
dev.em.0.interrupts.rx_overrun: 0
dev.em.0.host.breaker_tx_pkt: 0
dev.em.0.host.host_tx_pkt_discard: 0
dev.em.0.host.rx_pkt: 0
dev.em.0.host.breaker_rx_pkts: 0
dev.em.0.host.breaker_rx_pkt_drop: 0
dev.em.0.host.tx_good_pkt: 0
dev.em.0.host.breaker_tx_pkt_drop: 0
dev.em.0.host.rx_good_bytes: 0
dev.em.0.host.tx_good_bytes: 0
dev.em.0.host.length_errors: 0
dev.em.0.host.serdes_violation_pkt: 0
dev.em.0.host.header_redir_missed: 0
dev.em.1.%desc: Intel(R) PRO/1000 Network Connection 7.0.5
dev.em.1.%driver: em
dev.em.1.%location: slot=0 function=0
dev.em.1.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9
subdevice=0x10d3 class=0x020000
dev.em.1.%parent: pci2
dev.em.1.nvm: -1
dev.em.1.rx_int_delay: 200
dev.em.1.tx_int_delay: 200
dev.em.1.rx_abs_int_delay: 500
dev.em.1.tx_abs_int_delay: 500
dev.em.1.rx_processing_limit: 100
dev.em.1.link_irq: 7
dev.em.1.mbuf_alloc_fail: 0
dev.em.1.cluster_alloc_fail: 0
dev.em.1.dropped: 0
dev.em.1.tx_dma_fail: 0
dev.em.1.fc_high_water: 18432
dev.em.1.fc_low_water: 16932
dev.em.1.mac_stats.excess_coll: 0
dev.em.1.mac_stats.symbol_errors: 0
dev.em.1.mac_stats.sequence_errors: 0
dev.em.1.mac_stats.defer_count: 0
dev.em.1.mac_stats.missed_packets: 0
dev.em.1.mac_stats.recv_no_buff: 0
dev.em.1.mac_stats.recv_errs: 0
dev.em.1.mac_stats.crc_errs: 0
dev.em.1.mac_stats.alignment_errs: 0
dev.em.1.mac_stats.coll_ext_errs: 0
dev.em.1.mac_stats.rx_overruns: 0
dev.em.1.mac_stats.watchdog_timeouts: 3
dev.em.1.mac_stats.xon_recvd: 0
dev.em.1.mac_stats.xon_txd: 0
dev.em.1.mac_stats.xoff_recvd: 0
dev.em.1.mac_stats.xoff_txd: 0
dev.em.1.mac_stats.total_pkts_recvd: 638131
dev.em.1.mac_stats.good_pkts_recvd: 638131
dev.em.1.mac_stats.bcast_pkts_recvd: 49276
dev.em.1.mac_stats.mcast_pkts_recvd: 0
dev.em.1.mac_stats.rx_frames_64: 8091
dev.em.1.mac_stats.rx_frames_65_127: 472864
dev.em.1.mac_stats.rx_frames_128_255: 46165
dev.em.1.mac_stats.rx_frames_256_511: 33998
dev.em.1.mac_stats.rx_frames_512_1023: 44560
dev.em.1.mac_stats.rx_frames_1024_1522: 32453
dev.em.1.mac_stats.good_octets_recvd: 0
dev.em.1.mac_stats.good_octest_txd: 0
dev.em.1.mac_stats.total_pkts_txd: 668981
dev.em.1.mac_stats.good_pkts_txd: 668981
dev.em.1.mac_stats.bcast_pkts_txd: 0
dev.em.1.mac_stats.mcast_pkts_txd: 12
dev.em.1.mac_stats.tx_frames_64: 5594
dev.em.1.mac_stats.tx_frames_65_127: 166028
dev.em.1.mac_stats.tx_frames_128_255: 33061
dev.em.1.mac_stats.tx_frames_256_511: 28853
dev.em.1.mac_stats.tx_frames_512_1023: 24279
dev.em.1.mac_stats.tx_frames_1024_1522: 411166
dev.em.1.mac_stats.tso_txd: 0
dev.em.1.mac_stats.tso_ctx_fail: 0
dev.em.1.interrupts.asserts: 0
dev.em.1.interrupts.rx_pkt_timer: 0
dev.em.1.interrupts.rx_abs_timer: 0
dev.em.1.interrupts.tx_pkt_timer: 0
dev.em.1.interrupts.tx_abs_timer: 0
dev.em.1.interrupts.tx_queue_empty: 0
dev.em.1.interrupts.tx_queue_min_thresh: 0
dev.em.1.interrupts.rx_desc_min_thresh: 0
dev.em.1.interrupts.rx_overrun: 0
dev.em.1.host.breaker_tx_pkt: 0
dev.em.1.host.host_tx_pkt_discard: 0
dev.em.1.host.rx_pkt: 0
dev.em.1.host.breaker_rx_pkts: 0
dev.em.1.host.breaker_rx_pkt_drop: 0
dev.em.1.host.tx_good_pkt: 0
dev.em.1.host.breaker_tx_pkt_drop: 0
dev.em.1.host.rx_good_bytes: 0
dev.em.1.host.tx_good_bytes: 0
dev.em.1.host.length_errors: 0
dev.em.1.host.serdes_violation_pkt: 0
dev.em.1.host.header_redir_missed: 0

loader.conf variables:

if_em_load=YES
hw.em.rxd=4096
hw.em.txd=4096
hw.em.tx_int_delay=200
hw.em.rx_int_delay=200
hw.em.tx_abs_int_delay=500
hw.em.rx_abs_int_delay=500
hw.em.enable_msix=1
hw.em.msix_queues=2
hw.em.rx_process_limit=100
hw.em.fc_setting=0


And here is vmstat -i 

interrupt                          total       rate
irq16: uhci0                        2673          2
irq18: ehci0 uhci5                     2          0
irq19: uhci2 uhci4                    70          0
irq23: uhci3 ehci1                  8835          9
cpu0: timer                      3743479       3957
irq256: em0                       662219        700
irq257: em0                       717531        758
irq259: em0                        12142         12
irq260: em0                         6100          6
irq261: em1                       694169        733
irq262: em1                       217388        229
irq264: em1                       463023        489
irq265: em1                           11          0
cpu1: timer                      3743411       3957
cpu6: timer                      3743408       3957
cpu7: timer                      3743407       3957
cpu4: timer                      3743408       3957
cpu5: timer                      3743407       3957
cpu3: timer                      3743408       3957
cpu2: timer                      3743409       3957
Total                           32731500      34599


If you need more information just let me know, I can use this box for
tests at night until others can handle traffic.




More information about the freebsd-net mailing list