RFC: adding 'proxy' nodes to provider ports (with patch)

[Luigi Rizzo]

Hi,

Fabio Checconi and I have been thinking on how to implement "proxy"
geom nodes, i.e. nodes that have exactly 1 provider and 1 consumer
port, do not do any data transformation, and can be transparently
inserted or removed on top of a provider port while the tree is
actively moving data.

Our immediate need was to add/remove a scheduler while a disk is
mounted, but there are possibly other uses e.g. if one wants to
"sniff" the traffic through a disk, or do other ops that are
transparent for the data stream.

We would like opinion on the following solution, which seems
extremely simple in terms of implementation.

The idea is to intercept requests coming on a provider port, pp, and
redirect them to a geom node acting as a proxy if the port
is configured in this way:

     +=====...===...======+
     H                    H
     H                    H
     H                    H
     +=====...====== cp ==+
                     |          +---------------+
                     V          |               V
     +=====.....==== pp ==+     |    +======= proxy_pp ==+
     H           'ad0s1'  H     |    H                   H
     H                ------->--+    H                   H
     H        gp      -------<--+    H    proxy_node     H
     H                    H     |    H                   H
     +=======....===...===+     |    +======= proxy_cp ==+
                                |               V
                                +---------------+

Normally the proxy does not exist, and the geom tree works as it does now.

When we create a 'proxy' node, with something like

	geom my_proxy_class proxy ad0s1

we do something very similar to a 'create', but:

- the proxy node is marked specially in gp->flags, so the core will
  not generate a g_new_provider_event when the provider port is created
  (this means there is no taste() call and nobody should be able
  to attach to the port).

- the provider port we attach to is linked, with two pointers,
  to the provider and consumer ports of the proxy_node.

In this situation, g_io_request() finds that port pp has a proxy attached
to it, and immediately redirects the requests to the proxy, which
does everything a geom node does (cloning requests, etc).
   When the proxy wants to pass the request down, it sends it again to pp,
but now there is no redirection because the source can be identified
as the proxy.  The pointers in the bio insure a correct flow of the
requests on the reverse path.

Disconnecting a proxy is almost trivial: apart from handling possible
races on the data path, we just need to clear pp->proxy_pp and pp->proxy_cp.
After that, we can just send the regular destroy events to the proxy
node, who will have to take care of flushing any pending bio's (e.g.
see our geom_sched node that already does this).

Overall the change is very small (see attached patch):
a couple of lines in g_io_request, two extra fields in the g_provider,
and the addition of a flag to gp->flags to control the generation
of g_new_provider_event.
There is basically no overhead on regular operation, and only
a couple of extra pointers in struct g_provider (we use a spare
bit in gp->flags to mark G_GEOM_PROXY nodes).

The only things missing in the patch should be:

- a check to avoid races on creation&destruction of a proxy.
  I am not so sure on how to achieve this, but creation and destruction
  are rare and can normally wait, so we could just piggyback the
  small critical section (manipulating pp->proxy_cp and pp->proxy_cp)
  into some other piece of code that is guaranteed to be race-free.

- a check to prevent attaching to a provider port of a proxy
  (not a problem, i believe);

- a check to prevent attaching a proxy to a provider port that already
  has one. Of course you can attach a proxy to another proxy, and
  if you want to change the order it is as simple as removing the
  existing proxy and reattaching it after the new one.

Feedback welcome.

	cheers
	luigi and fabio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: geom-proxy.patch
Type: text/x-diff
Size: 3166 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-arch/attachments/20090319/d051c8bb/geom-proxy.bin