82576 + NETMAP + VLAN

Slawa Olhovchenkov slw at zxy.spb.ru
Wed Feb 10 13:53:23 UTC 2016

On Wed, Feb 10, 2016 at 02:33:20PM +0100, Giuseppe Lettieri wrote:

> Il 10/02/2016 12:59, Slawa Olhovchenkov ha scritto:
> > Can you look also on second issue?
> >
> > PS: What need from me? May be open PR?
> May you provide some example code that triggers the issue?

This is about 700 lines of code (not very clear), may be I can describe it?
First: this is thread code. I think it impotant point.
All sync with netmap go throw kevent() interface.

Main thread got information about two NIC (determine number of rings)
and spawn 1 balancer thread and NCPU worker thread.
All thread pinned to cores.

Balancer thread open all rings on both NIC separate (i.e. open 6+6
NETMAP fds for hardware rings) and allocate on first NIC and open
NCPU*2 master NETMAP pipes.
All fds register in kevent for read event as

EV_SET(kp, fd, EVFILT_READ, EV_ADD | EV_ENABLE | EV_CLEAR, 0, 0, kidx);

and for future write event as

EV_SET(kp, fd, EVFILT_WRITE, EV_ADD  | EV_DISABLE, 0, 0, 0);

after write occur next event will be added:


Each worker thread open two slave NETMAP pipes in same maner as above.

Balancer got packet from inside hardware ring, dispatch (zero-copy) to inside master pipe.
Worker got packet from inside slave pipe and zero-copy to outside slave pipe.
Balancer got  packet from outside master pipe and dispatch (zero-copy) to outside hardware ring.
And vise versa.

After program start I have deadlock instantly (with WITNESS enabled kernel).
W/o WITNESS deadloc occur only under load.

W/o EV_DISABLE/EV_DISPATCH NETAMP don't see output packets.
NETMAP don't forward incoming packet to program when open and attach to NETMAP do in main thread.

More information about the freebsd-stable mailing list