strange arp problem with bge nics

[Nikos Vassiliadis]

On Friday 31 August 2007 21:55, Tobias Ernst wrote:
> Hi,
>
> I have further news on this problem. It really seems to be a
> driver/hardware issue.
>
> As I said, the two servers have 6 NICs each. These are:
>
> bge0, bge1: BCM5750, integrated on the motherboard
> bge2, bge3: BCM5704, PCIX card
> bge4, bge5: BCM5704, PCIX card
>
> I have now greatly simplified the test case: Only connect any two
> interfaces with the same number with a crosslink cable or an otherwise
> unused switch. Assign two IP addresses from within the same subnet.
> E.g., make bge0 on machine #1 10.0.0.1 and bge0 on machine #2 10.0.0.2.
> Don't connect anything else.
>
> I can instantly ping the other machine after booting up when using bge0,
> bge1 or bge2 on both machines.
>
> I cannot initially ping the other machine when using bge3, bge4 or bge5.
>  In this case, I first have to put one of the interfaces into
> promiscuous mode, wait for the ping to come through, then disable
> promiscuous mode.
>
> Incidentally, the working interfaces all sit on IRQ3, while the other
> three sit on IRQ7, IRQ11 and IRQ5, respectively.
>
> Where do I take this from here? I need at least four interfaces working
> for the configuration I need to implement. I could do away with the
> other two, but four is the minimum I need.
>
> Incidentally, another option to "wake up" the ping, apart from setting
> and unsetting promiscous modem, is to connect any Windows machine to the
> same switch. As soon as a Windows machine is present on the switch, the
> ping between the two FreeBSD machines works right out from the start.
>
> This looks like a minor issue at first glance, because everything seems
> to be normal once the ping is set going, and I could just write a script
> that enables promiscuous mode on startup for a certain amount of time,
> and there will always be Windows boxes on the network anyway. However, I
> am now wary that there might be other hidden bugs or hardware problems,
> and I have no use for those in a production machine ...
>

If you take a look here http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/dev/bge/if_bge.c
you will see some problems with some chipsets regarding auto negotiation.
http://www.freebsd.org/cgi/query-pr.cgi?pr=94833

How all these apply to your case?
Did you try "down-ing" and "up-ing" the interfaces?
Did you try without forcing a link speed(check ifconfig -m)

Just wild guesses...

HTH, Nikos