On the transparent insertion of geoms, again

[Pawel Jakub Dawidek]

On Tue, Aug 11, 2009 at 05:30:50AM +0200, Fabio Checconi wrote:
> Hi all,
>   some time ago there was a discussion [1, 2] on this list on the
> transparent insertion of geoms in an open geom path.  There was some
> kind of agreement on a possible way of doing that, this is the diagram
> used at the time:
> 
> >   BEFORE        ---> [ pp    --> old_gp ...]
> > 
> > Then we can do either "geom xx create ad0" which results in
> > 
> >   AFTER create  ---> [ newpp --> gp --> cp ] ---> [ pp    --> old_gp ... ]
> > 
> > or "geom xx insert ad0", which results in
> > 
> >   AFTER insert  ---> [ pp    --> gp --> cp ] ---> [ newpp --> old_gp ... ]
> > 
> 
> [ see the original threads for more details and a draft of the code ]
> 
>  
> The solution relied on the fact that bios keep a reference to their
> way back up into the geom path, so it is possible to change on-the-fly
> the contents of the providers they go through without affecting the
> pending bios.
> 
> Unfortunately a little problem hides behind this assumption; considering
> e.g., g_disk_done():
> 
> %  if ((bp->bio_cmd & (BIO_READ|BIO_WRITE|BIO_DELETE)) &&
> %      (dp = bp2->bio_to->geom->softc)) {
> %          devstat_end_transaction_bio(dp->d_devstat, bp);
> %  }
> 
> if bp2->bio_to->geom is dereferenced after the ``geom insert'' above,
> pp->geom points to the new geom, thus the softc of the new class is used
> as a struct disk pointer (with all the consequent breakage).
> 
> To fix that we have fallen back to waiting for the completion of all
> the pending bios in the ``geom insert'' path, which involves sleeping
> (with a timeout) with topology held, from the event thread.  The reasons
> behind this (admittedly ugly) design are:
> 
>   - waiting unconditionally may lead to stall, if the transparent
>     insertion is done on top of a geom which serves its bios from the
>     event thread (like geom_slice hotspot users may do), so we need a
>     timeout to limit this effect;
> 
>   - new requests for the old provider/geom couple may arrive, we need
>     to store them in a temporary queue;
> 
>   - we need to sleep inside topology, releasing it to allow progress
>     to the event thread would mean rechecking *everything* after we
>     reacquire it to verify if there are no more pending bios; we tried
>     this and the complexity seemed to be excessive.
> 
> So the question is: does anyone see a better/simpler way of doing the
> same hot-insertion without the spurious failures this method may introduce?
> 
> Any suggestion is welcome, thank you in advance.

I also found another problem. Some classes (mine, for example) reference
provider in their private metadata (sc_provider field in g_mirror_softc
structure), once this provider is moved to another geom this reference
will become invalid.

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd at FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-geom/attachments/20090811/51aee736/attachment.pgp