Myrinet 10Gb odd behavior - SOLVED

[aurfalien]

Spoke to soon. Fine for a while (doing a 5 day rsync of 38TB) but getting those errors every 7 min.  And I'm only getting 1.24Gb/s over a 10Gb jumbo link.

Definitely causing connection issues.

Using it for ethernet.

Gonna go in tomorrow and give my Solarflare another shot as it was giving me issues but the rel notes say to try this, so I will;

 - The driver uses mbufs to store packet data which come from a set of pools
   of limted size. See man 7 tuning for more details. The following command
   can display the number of used and free mbufs within the pools the Solarflare
   driver uses

# vmstat -z | head -n 1; vmstat -z | grep mbuf
ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS  FAILURES
mbuf_cluster:            2048,    25600,     1408,      658,    31604,        0
mbuf_jumbo_page:         4096,    12800,        0,       76,     2063,        0
mbuf_jumbo_9k:           9216,     6400,        0,        0,        0,        0
mbuf_jumbo_16k:         16384,     3200,        0,        0,        0,        0

  If a pool is exhausted (i.e. the failure count in the right hand column is
  non-zero, networking applications may hang or received packets may be dropped.
  Hence you may need to increase these limits using the following sysctls:
     kern.ipc.nmbclusters (for mbuf_cluster)
     kern.ipc.nmbjumbop   (for mbuf_jumbo_page)
     kern.ipc.nmbjumbo9   (for mbuf_jumbo_9k)
     kern.ipc.nmbjumbo16  (for mbuf_jumbo_16k)


- aurf



On Aug 17, 2013, at 8:14 PM, iamatt wrote:

> Wow myricom still around...  used to use the lanai stuff never on bsd though.  All FDR Infiniband these days.  Are you using the myrinet protocol or ethernet,  just curious.  Glad you got it working!
> 
> On Aug 16, 2013 8:12 PM, "aurfalien" <aurfalien at gmail.com> wrote:
> 
> On Aug 16, 2013, at 8:47 AM, aurfalien wrote:
> 
> > Forgot to mention my loader.conf;
> >
> > if_mxge_load="YES"
> > mxge_ethp_z8e_load="YES"
> > mxge_eth_z8e_load="YES"
> > mxge_rss_ethp_z8e_load="YES"
> > mxge_rss_eth_z8e_load="YES"
> >
> >
> > I blindly added these w/o thinking what they do.
> >
> > Should I simply only load the first line?
> >
> > - aurf
> >
> >
> > On Aug 16, 2013, at 8:18 AM, aurfalien wrote:
> >
> >> Hi,
> >>
> >> I've been suspecting my NIC is not up to par and notice this in the logs every few minutes;
> >>
> >> Aug 16 08:05:06 prometheus kernel: mxge0: slice 0 struck? ring state:
> >> Aug 16 08:05:06 prometheus kernel: mxge0: tx.req=1914503981 tx.done=1914503810, tx.queue_active=0
> >> Aug 16 08:05:06 prometheus kernel: mxge0: tx.activate=0 tx.deactivate=0
> >> Aug 16 08:05:06 prometheus kernel: mxge0: pkt_done=1824019832 fw=1824019931
> >> Aug 16 08:05:06 prometheus kernel: mxge0: Watchdog reset!
> >> Aug 16 08:05:06 prometheus kernel: mxge0: NIC did not reboot, not resetting
> >>
> >> Could tis be effecting throughput?
> >>
> >> My card is a Myri-10G-PCIE-8A
> >>
> >> I did install the Myrinet dev tools for FreeBSD and ran myri_info which yields;
> >>
> >> pci-dev at 05:00.0 vendor:product(rev)=14c1:0008(00)
> >>      behind bridge root-port: 00:03.0 8086:3c08 (x8.1/x16.3)
> >> Myri-10G-PCIE-8A -- Link x8
> >>  EEPROM String-spec:
> >>      MAC=00:60:dd:45:73:23
> >>      SN=413665
> >>      PWR=100
> >>      PC=10G-PCIE-8A-R
> >>      PN=09-03852
> >>      XFI=AEL1010
> >>      TAG=ze_tools-1_4_45
> >>
> >>  EEPROM MCP, PRESENT, length = 103384, crc=0x119daf46
> >>      ETHZ::1.4.45 2009/08/22 18:57:06 self extracting firmware
> >>      Bundle: exec_len=72144, PCI-ROM-len = 31232
> >>  Running MCP:
> >>      ETH ::1.4.55 -P- 2012/04/21 01:48:34 myri10ge firmware
> >>
> >> Any insights are appreciated.
> >>
> >> - aurf
> 
> 
> Did the ole RTFM and re programmed the firmware, all good now.
> 
> - aurf
> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freebsd.org"