On the transparent insertion of geoms, again

[Fabio Checconi]

Hi all,
  some time ago there was a discussion [1, 2] on this list on the
transparent insertion of geoms in an open geom path.  There was some
kind of agreement on a possible way of doing that, this is the diagram
used at the time:

>   BEFORE        ---> [ pp    --> old_gp ...]
> 
> Then we can do either "geom xx create ad0" which results in
> 
>   AFTER create  ---> [ newpp --> gp --> cp ] ---> [ pp    --> old_gp ... ]
> 
> or "geom xx insert ad0", which results in
> 
>   AFTER insert  ---> [ pp    --> gp --> cp ] ---> [ newpp --> old_gp ... ]
> 

[ see the original threads for more details and a draft of the code ]

 
The solution relied on the fact that bios keep a reference to their
way back up into the geom path, so it is possible to change on-the-fly
the contents of the providers they go through without affecting the
pending bios.

Unfortunately a little problem hides behind this assumption; considering
e.g., g_disk_done():

%  if ((bp->bio_cmd & (BIO_READ|BIO_WRITE|BIO_DELETE)) &&
%      (dp = bp2->bio_to->geom->softc)) {
%          devstat_end_transaction_bio(dp->d_devstat, bp);
%  }

if bp2->bio_to->geom is dereferenced after the ``geom insert'' above,
pp->geom points to the new geom, thus the softc of the new class is used
as a struct disk pointer (with all the consequent breakage).

To fix that we have fallen back to waiting for the completion of all
the pending bios in the ``geom insert'' path, which involves sleeping
(with a timeout) with topology held, from the event thread.  The reasons
behind this (admittedly ugly) design are:

  - waiting unconditionally may lead to stall, if the transparent
    insertion is done on top of a geom which serves its bios from the
    event thread (like geom_slice hotspot users may do), so we need a
    timeout to limit this effect;

  - new requests for the old provider/geom couple may arrive, we need
    to store them in a temporary queue;

  - we need to sleep inside topology, releasing it to allow progress
    to the event thread would mean rechecking *everything* after we
    reacquire it to verify if there are no more pending bios; we tried
    this and the complexity seemed to be excessive.

So the question is: does anyone see a better/simpler way of doing the
same hot-insertion without the spurious failures this method may introduce?

Any suggestion is welcome, thank you in advance.


[1] http://lists.freebsd.org/pipermail/freebsd-geom/2009-March/003400.html
[2] http://lists.freebsd.org/pipermail/freebsd-geom/2009-March/003407.html