virtual NIC rings (Re: FreeBSD I/OAT (QuickData now?) driver)

[Luigi Rizzo]

Two previous discussions on direct NIC access evolved on how
to share one interface among multiple clients (including the
host stack) some of which can be less trusted than others.

I have not addressed these problems yet in netmap, because
i see them as orthogonal to the API for accessing data, 
nor i think that the number of virtual queues supported by
the hardware is fundamental in solving the problem.

As i see it, the important thing is understand who does:

- replication (needed for multicast, broadcast, sniffing)
  The main reason to put the NIC in charge of replication is
  that it saves contention on locks. Other than that, there are
  only disadvantages, because the PCI-e bus has a lot less capacity
  than the memory bus, and (perhaps) a CPU-assisted replication
  could be based on shared readonly copies.

- packet filtering (which packet goes where);
  some nics already support routing of packets to queues according to
  various fields. This is nice to have, but not essential.
  I'd argue that it mostly saves lock contention, but otherwise,
  with the table sizes available in the NIC, even filtering in software
  is not prohibitively expensive.

- "access control" (is client X allowed to send/receive certain traffic).
  Do we want an untrusted client to be able to send packets with
  someone else's source MAC or IP ?  Perhaps the right answer is
  "who cares, it can happen anyways on the PC next to you", and
  this saves a per-packet decision.  On the incoming side the problem
  is easier, as programming the packet filter/replicator is done
  once in a while and not on a per-packet basis.

- memory protection.
  Right now netmap uses a single buffer area for all rings on all
  cards. While this is extremely efficient at runtime (you can move
  packets around by just shuffling buffer indexes; validity checks
  require a single comparison and perhaps a lookup; you can do
  zero-copy forwarding, or modify buffers if needed), it is clearly
  vulnerable to data corruption from misbehaving processes. It will
  be interesting to see how we can insert some protection without
  killing the option of zero-copy operation.

Once we have addresses these issues, of course any hardware
support will be welcome.

Just to refer to the one implementation i know, netmap creates one
virtual ring for each hardware ring, plus one ring pair for the
host stack.  I think the next step would be to put some additional
field (e.g. a MAC address) to drive the filtering and replication
engines (either on-nic, or done in software).  Then one could e.g.
say "give me all traffic for MAC X," and have the kernel decide
whether this needs partial or full traffic replication, etc.

cheers
luigi

http://info.iet.unipi.it/~luigi/netmap/