Problem with receiving packets right after remote-interface is up

[Artem Belevich]

On Sat, Sep 5, 2015 at 2:44 PM, Luigi Rizzo <rizzo at iet.unipi.it> wrote:

> You may be able to avoid the issue if you avoid autonegotiation and
> manually set the media type, speed and duplex.
>

Technically, 1G ethernet *requires* autonegotiation.
http://standards.ieee.org/findstds/interps/IEEE802.3af-2003interp-6.pdf

While you can indeed set speed/duplex manually, you will also need to make
sure both have have proper clock master/slave  selection  which is normally
done via autonegotiation. On Freebsd some interfaces support "mediaopt
master" option -- check if that's the case for your interface with ifconfig
-m.

You will also need to make sure that the other end of the link is also
configured to set matching link parameters manually.

Even with all of that, copper PHYs take a while to bring link up as PHYs
need to tune their parameters. Ethernet MAC will not be able to send or
receive any packets *on the wire* until PHYs are done which may take
relatively long time (~hundreds of milliseconds).

>
> On Saturday, September 5, 2015, M. V. via freebsd-net <
> freebsd-net at freebsd.org> wrote:
>
> > Hi,
> >
> > Our product is being tested with Spirent TestCenter, and we're facing an
> > unusual problem with the tests.
> > We use NICs with intel 82574 and 82576 on FreeBSD 9.2 with latest em and
> > igb drivers (we also tested this on FreeBSD-10.1) It seems what Spirent
> > TestCenter does to start any individual test is, it disables its own
> > interface, and at the beginning of the new test, it suddenly "up"s its
> > interface and sends test packet right after that without any delay.
>

You may want to go over spirent box settings and look for something along
the lines of "wait for the link to come up". It may be somewhere in
per-interface settings.


> This is
>
> where we have problem, and we don't receive this first packets most of the
> > time (result is vary, in 100 tests, we lose about 60~70% of this "first"
> > packets on each test, so we FAIL most of tests because apparently we need
> > about 0.5~1 seconds after setup and renegotiation before we can receive
> > packets)
>

That's in the ballpark of time it takes for a copper link to come up.


> > we checked both em and igb drivers and both have this problem.
> > this is how we reproduce the problem:
> > - connect sender-system's first port to receiver-system's first port.
> > - In sender-system:
> > -- ifconfig em0 down
> > -- sleep 5
> > -- ifconfig em0 up && tcpreplay -l 1 -i em0 one_packet.pcap
> >
> > We checked, and even though sender sends packet successfully (tx led
> > blinks),
>

It's not a conclusive proof that the packet made it to the wire. Depending
on where the LED is connected, it may mean different things -- if it's set
up to indicate that MAC is sending a packet to PHY, for instance, it would
happily blink if the link is down.


> in receiver even driver doesn't receive it (rx led doesn't blink,
> > and "rxeof" function in "if_em" tells us no new packet is received)
> > we removed optional interface options like msix, tso, rxcsum, ... but no
> > luck.
>

Which is another point that the link is not up and the packet did not get
received by the PHY.


> >
> > So, does anyone know any solution for this problem? or any suggestions on
> > where we should look or what we should do that might solve it and prevent
> > our tests from being failed?
>

It sounds to me like the problem is with your test setup. If your test
requires no packet drops, then it implicitly assumes having established
link and sending the packets right after the interface (and not necessarily
the physical link) is up is really pushing it. You will either need to
configure your test gear to wait some time after the interface is up, or
redesign your test so it can tolerate some packet loss while the physical
link is being established.

To make things more interesting, consider that ethernet does not guarantee
packet delivery at all.

--Artem