RFC: adding 'proxy' nodes to provider ports (with patch)

[Marius Nünnerich]

2009/3/19 Luigi Rizzo <rizzo at iet.unipi.it>:
> Hi,
>
> Fabio Checconi and I have been thinking on how to implement "proxy"
> geom nodes, i.e. nodes that have exactly 1 provider and 1 consumer
> port, do not do any data transformation, and can be transparently
> inserted or removed on top of a provider port while the tree is
> actively moving data.
>
> Our immediate need was to add/remove a scheduler while a disk is
> mounted, but there are possibly other uses e.g. if one wants to
> "sniff" the traffic through a disk, or do other ops that are
> transparent for the data stream.
>
> We would like opinion on the following solution, which seems
> extremely simple in terms of implementation.
>
> The idea is to intercept requests coming on a provider port, pp, and
> redirect them to a geom node acting as a proxy if the port
> is configured in this way:
>
>     +=====...===...======+
>     H                    H
>     H                    H
>     H                    H
>     +=====...====== cp ==+
>                     |          +---------------+
>                     V          |               V
>     +=====.....==== pp ==+     |    +======= proxy_pp ==+
>     H           'ad0s1'  H     |    H                   H
>     H                ------->--+    H                   H
>     H        gp      -------<--+    H    proxy_node     H
>     H                    H     |    H                   H
>     +=======....===...===+     |    +======= proxy_cp ==+
>                                |               V
>                                +---------------+
>
> Normally the proxy does not exist, and the geom tree works as it does now.
>
> When we create a 'proxy' node, with something like
>
>        geom my_proxy_class proxy ad0s1
>
> we do something very similar to a 'create', but:
>
> - the proxy node is marked specially in gp->flags, so the core will
>  not generate a g_new_provider_event when the provider port is created
>  (this means there is no taste() call and nobody should be able
>  to attach to the port).
>
> - the provider port we attach to is linked, with two pointers,
>  to the provider and consumer ports of the proxy_node.
>
> In this situation, g_io_request() finds that port pp has a proxy attached
> to it, and immediately redirects the requests to the proxy, which
> does everything a geom node does (cloning requests, etc).
>   When the proxy wants to pass the request down, it sends it again to pp,
> but now there is no redirection because the source can be identified
> as the proxy.  The pointers in the bio insure a correct flow of the
> requests on the reverse path.
>
> Disconnecting a proxy is almost trivial: apart from handling possible
> races on the data path, we just need to clear pp->proxy_pp and pp->proxy_cp.
> After that, we can just send the regular destroy events to the proxy
> node, who will have to take care of flushing any pending bio's (e.g.
> see our geom_sched node that already does this).
>
> Overall the change is very small (see attached patch):
> a couple of lines in g_io_request, two extra fields in the g_provider,
> and the addition of a flag to gp->flags to control the generation
> of g_new_provider_event.
> There is basically no overhead on regular operation, and only
> a couple of extra pointers in struct g_provider (we use a spare
> bit in gp->flags to mark G_GEOM_PROXY nodes).
>
> The only things missing in the patch should be:
>
> - a check to avoid races on creation&destruction of a proxy.
>  I am not so sure on how to achieve this, but creation and destruction
>  are rare and can normally wait, so we could just piggyback the
>  small critical section (manipulating pp->proxy_cp and pp->proxy_cp)
>  into some other piece of code that is guaranteed to be race-free.
>
> - a check to prevent attaching to a provider port of a proxy
>  (not a problem, i believe);
>
> - a check to prevent attaching a proxy to a provider port that already
>  has one. Of course you can attach a proxy to another proxy, and
>  if you want to change the order it is as simple as removing the
>  existing proxy and reattaching it after the new one.
>
> Feedback welcome.

I wonder if it's really necessary to alter the GEOM infrastructure or
if it is possible to do this with what's there already. Just an idea:
Lock g_topology, put g_down and g_up to sleep, alter the consumer and
provider pointers where you need it so the everything is routed
through your proxy class (which isn't special in any way) and restart
g_down and g_up.