em(4)/82571eb: fifo not dma'ed to host memory

Wed Jul 18 06:39:55 UTC 2007

thanks for your feedback jack - i tried the MSI stuff as first task this
morning
the interface init'ed without a problem and did not complain about MSI
what i like most is that no IRQ is assigned now, everything seems to get
handled in a TASKQ (IRQs were quite weird sometimes when having a bunch of
interfaces...)

but now the bad news: even with MSI it is not possible to recv any frames
what makes me curious is, that the head pointer for receive fifo is *not*
counting up after receiving a frame (good_packets counter incrementing)
as far as i understand the 82571EB chipset manual, recv head counter inc
is done in hardware after receiving a good_packet and having dma'ed it to
host memory
from if_em.c/em_process_receive_interrupts:
 *  This routine executes in interrupt context. It replenishes
 *  the mbufs in the descriptor and sends data which has been
 *  dma'ed into host memory to upper layer.

i did a quick test to check this on 6.1-RELENG by inserting a return
at the top of em_process_receive_interrupts (see diff)
register RDH0 is still incrementing for each good_packet received - even
without receive interrupts being processed at the driver level

  freebsd61# sysctl dev.em.0.debug_info=1
  dev.em.0.debug_iemnfo: 0:-1 Adapter hardware address = 0xc4b7c928 
  em0: CTRL = 0x81c0241 RCTL = 0x8002 
  em0: RDBAH0 = 0x0 RDBAL0 = 0x3e0ff000 
  em0: RDLEN0 = 0x1000 RDH0 = 0x3 
  em0: RDT0 = 0xff RXDCTL = 0x10000

...so IMHO this increment (and also dma transfer to host memory) is done
in hardware - RDH0 is not counting up when running under 7.0-CURRENT...

  erroneous init of chipset?
  anyone else running 7.0-CURRENT on a 82571EB pci express chipset?

TIA for any help on this
pat

details for dmesg/if_stats/diff:

  pcib9: <PCI-PCI bridge> irq 19 at device 11.0 on pci2
  pcib9:   secondary bus     9
  pcib9:   subordinate bus   9
  pcib9:   I/O decode        0x6000-0x6fff
  pcib9:   memory decode     0xfd700000-0xfd7fffff
  pcib9:   prefetched decode 0xfce00000-0xfcefffff
  pci9: <PCI bus> on pcib9
  pci9: physical bus=9
  found-> vendor=0x8086, dev=0x105e, revid=0x06
          bus=9, slot=0, func=0
          class=02-00-00, hdrtype=0x00, mfdev=1
          cmdreg=0x0000, statreg=0x0010, cachelnsz=16 (dwords)
          lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
          intpin=a, irq=255
          powerspec 2  supports D0 D3  current D0
          MSI supports 1 message, 64 bit
          map[10]: type Memory, range 32, base 0, size 17, memory disabled
          map[14]: type Memory, range 32, base 0, size 17, memory disabled
          map[18]: type I/O Port, range 32, base 0, size  5, port disabled
  found-> vendor=0x8086, dev=0x105e, revid=0x06
          bus=9, slot=0, func=1
          class=02-00-00, hdrtype=0x00, mfdev=1
          cmdreg=0x0000, statreg=0x0010, cachelnsz=16 (dwords)
          lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
          intpin=b, irq=255
          powerspec 2  supports D0 D3  current D0
          MSI supports 1 message, 64 bit
          map[10]: type Memory, range 32, base 0, size 17, memory disabled
          map[14]: type Memory, range 32, base 0, size 17, memory disabled
          map[18]: type I/O Port, range 32, base 0, size  5, port disabled
  em0: <Intel(R) PRO/1000 Network Connection Version - 6.5.3> at device
    0.0 on pci9
  pcib9: em0 requested memory range 0xfd700000-0xfd7fffff: good
  pcib2: em0 requested memory range 0xfd700000-0xfd7fffff: good
  pcib1: em0 requested memory range 0xfd700000-0xfd7fffff: good
  em0: Lazy allocation of 0x20000 bytes rid 0x10 type 3 at 0xfd700000
  em0: attempting to allocate 1 MSI vectors (1 supported)
  msi: routing MSI IRQ 256 to vector 49
  em0: using IRQ 256 for MSI
  em0: bpf attached
  em0: Ethernet address: 00:10:f3:0c:5b:2a
  em0: [FILTER]
  pci9: <network, ethernet> at device 0.1 (no driver attached)

  freebsd61# diff -u sys/dev/em/if_em.c.orig sys/dev/em/if_em.c
  --- sys/dev/em/if_em.c.orig     Wed Jul 18 07:32:53 2007
  +++ sys/dev/em/if_em.c  Wed Jul 18 07:48:42 2007
  @@ -277,6 +277,11 @@

          INIT_DEBUGOUT("em_probe: begin");

  +        // prevent init of all em(4) devices except pci9/0/0 
  +        if(pci_get_bus(dev) != 0x09 || pci_get_slot(dev) != 0x00 ||
  +           pci_get_function(dev) != 0x00)
  +                return (ENXIO);
  +
          pci_vendor_id = pci_get_vendor(dev);
          if (pci_vendor_id != EM_VENDOR_ID)
                  return(ENXIO);
  @@ -2906,6 +2911,8 @@
          u_int8_t            eop = 0;
          u_int16_t           len, desc_len, prev_len_adj;
          int                 i;
  +
  +       return;

          /* Pointer to the receive descriptor being examined. */
          struct em_rx_desc   *current_desc;

On Tue, Jul 17, 2007 at 09:17:36AM -0700, Jack Vogel wrote:
> As an experiment, search in if_em.c for 'msix' look at the logic there.
> If the adapter is 82575 it will use msix, but the else clause has it
> use msi ONLY if its '>82571', change that to >=82571 and let's
> see if maybe using MSI will help you with your problems as I'm
> pretty confident its interrupt related.
> 
> I had intended on making this change shortly anyway, my tests
> have shown the 571 to work fine this way.
> 
> Let me know of the results.
> 
> Jack