em problems on supermicro 5015M-MT+

[Jack Vogel]

On 3/8/07, Sven Petai <hadara at bsd.ee> wrote:
> Hi
>
> We have 3 supermicro 5015M-MT+ machines that are identical hw
> wise and are all running 6.2R. One of them was installed in 32
> bit mode and others are running 64 bit systems.
> They are all connected to the same Cisco switch (2960G).
>
> 32 bit system can communicate with all the other systems besides
> the 64 bit boxes just fine.
> When we transfer files between the two 64 bit systems everything
> seems to be just fine and we get reasonable speeds.
>
> When one of the 64 bit systems tries to communicate with the 32 bit box or any
> other system whatsoever it gets:
>   * ~40KB/s when interface is in gigabit mode
>   * 320KB/s when interface is in 100baseTX full-duplex
>   * 500KB/s when interface is in 100baseTX half-duplex
>
> Tcpdump is full of lost packets, out of order packets, retransmits, duplicate
> acks (probably all caused by fast-retransmit) and all other kinds of
> nastiness.
>
> We also see the infamous "em0 watchdog timeout -- resetting" messages from
> time to time, but not always when doing the tests so resets alone can't be
> the reason.
>
> We have tried various network settings from polling to debug.mpsafenet=0
> and nothing seems to help much. To exclude possibilty of broken switch port/
> bad cable we also tried the same thing with em1 interfaces (with other cables
> of course) and still got identical symptoms.
>
> So what can possibly cause the fact that systems can communicate fine with
> eachother and fail so miserably when communicating with anything else ?
>
> If nobody can come up with any good ideas we will probably try installing 32
> bit systems on the problamatic machines since it seems to be the only
> relevant difference between the working and non-working ones.
>
> The 64 bit boxes are currently in preproduction testing and we can provide
> access to for developers who wish to debug it.
>
> Full dmesg & pciinfo is available @
> http://bsd.ee/~hadara/debug/supermicro/
>
> The most relevant parts are:
>
> pciconf:
> em0 at pci13:0:0:  class=0x020000 card=0x108c15d9 chip=0x108c8086 rev=0x03
> hdr=0x00
>     vendor   = 'Intel Corporation'
>     device   = 'PRO/1000 PM'
>     class    = network
>     subclass = ethernet
> em1 at pci14:0:0:  class=0x020000 card=0x109a15d9 chip=0x109a8086 rev=0x00
> hdr=0x00
>     vendor   = 'Intel Corporation'
>     class    = network
>     subclass = ethernet
>
> dmesg:
> em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x4000-0x401f
> mem 0xe0200000-0xe021ffff irq 16 at device 0.0 on pci13
> em0: Ethernet address: 00:30:48:8c:52:78
> pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.5 on pci0
> pci14: <ACPI PCI bus> on pcib5
> em1: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x5000-0x501f
> mem 0xe0300000-0xe031ffff irq 17 at device 0.0 on pci14
> em1: Ethernet address: 00:30:48:8c:52:79

I am not sure why the 32/64 bit architecture is making a difference, however,
you have the 573 NIC, so before doing anything else run this DOS app, it
will patch your eeprom if needed, the problem is a misprogramed MANC
register that tends to eat packets when it shouldnt.

Please let me know if that helps. Oh, you arent using TSO by some chance
are you?

Jack
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dcgdis.ThisIsZip
Type: application/octet-stream
Size: 158727 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20070308/b0473c4e/dcgdis-0001.obj