FreeBSD 10 network flapping, ix driver unreliable?

[Kevin Bowling]

On 2/15/2014 1:14 PM, Kevin Bowling wrote:
> Hi,
>
> I have FreeBSD 10.0-RELEASE installed on two Dell C6100 nodes.  Each
> node has an Intel X520-DA2 dual port 10gig card.  One of the ports on
> each go to a switch using direct attach coaxial cables.  The other port
> is directly connected between the two nodes (think crossover in twisted
> pair terminology) again using direct attach coaxial cables.
>
> On both machines, and on both ports (including the "crossover"), the
> links flap several times per day.
>
> I've pasted the output of lspci -vv and dmesg here:
> https://gist.github.com/kev009/9024442
>
> There's nothing outstanding about the setup otherwise.  I suspected some
> interaction with the switch initially but the "crossover" has eliminated
> that suspicion.
>
> It seems the ix driver is not very reliable under common conditions,
> i.e. https://forums.freebsd.org/viewtopic.php?f=7&t=44570 and a search
> of this list.  Any recommendations or tests?
>
> Regards,
> Kevin Bowling
>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>

After descending a rather dark rabbit hole, I'm pleased to have found a 
simple solution!

On some of these cards, there are known firmware problems.  The driver 
sometimes tries to compensate, but these code paths probably receive far 
less testing and look pretty fragile.  It seems some version of NIC 
firmware are particularly flaky with DA cables.

Some spam in the Linux dmesg led me to this, which does not appear to be 
in the FreeBSD ixgbe driver, although I did not stay in Linux long 
enough to see if it fully fixed the problem:
http://markmail.org/message/ivsjxoyfbvzv7mvo

Instead, I found a way to update the microcode.

My card and server are Dell and I was able to use this live cd to do the 
firmware upgrade: 
http://linux.dell.com/files/openmanage-contributions/om-firmware-live/ 
which applies a NIC firmware package like 
http://www.dell.com/support/drivers/us/en/19/driverdetails?driverid=HKK1W

Both of my systems appear to be stable; I'll comment if there are issues 
over the next few days.

Intel is less than forthcoming about these microcode updates; I'm not 
sure if the preboot code 
(https://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&ProdId=3591&DwnldID=19186&ProductFamily=Network+Connectivity&ProductLine=Intel%C2%AE+Server+Adapters&ProductProduct=Intel%C2%AE+Ethernet+Converged+Network+Adapter+X520+Series&lang=eng) 
contains the NIC microcode or if you must obtain it from a vendor.

Regards,
Kevin Bowling