gmirror synchronising is very slow due to frequent metadata updates

[Andre Albsmeier]

On Sun, 19-Nov-2017 at 22:38:28 -0500, Mark Johnston wrote:
> On Sun, Nov 19, 2017 at 11:32:41AM +0100, Andre Albsmeier wrote:
> > I am running a 2 TB gmirror with two disks. I replaced one disk and the
> > mirror started to synchronise. I noticed that the sync rate was about
> > 50 MB/s which is a lot lower than the more than 120 MB/s which are
> > possible with this hardware.
> > 
> > The reason for this are the very frequently happening metadata updates
> > done in sys/geom/mirror/g_mirror.c:
> > 
> >                if (sync->ds_offset_done + (MAXPHYS * 100) < offset) {
> >                         /* Update offset_done on every 100 blocks. */
> >                         sync->ds_offset_done = offset;
> >                         g_mirror_update_metadata(disk);
> > 
> > With MAXPHYS being 128kB this would mean that metadata is updated every
> > 12 MB -- or 4 times a second @ 50 MB/s.
> > 
> > Out of curiosity I changed this to MAXPHYS * 2000 which means that the
> > updating is done every 240 MB. Now the disks are synchronising with
> > approx. 105 MB/s...
> > 
> > Is there any reason why these metadata updates are done so often and
> > what are the risks having them more infrequently (as I do now)?
> 
> I believe that the metadata updates are done only so that it's possible
> to resume a synchronization after a crash. In particular, ds_offset_done

That's what I thought I had figured out as well when looking at the code.

> is translated into the sync_offset field in the metadata block of a
> synchronizing disk. Therefore, there is no real risk in making the
> updates less frequent; it'll just take marginally more time to complete
> an interrupted rebuild of a mirror (which would hopefully be a rare
> event anyway).
> 
> We should probably decrease the update interval based on the size of a
> mirror. For mirrors larger than say, 1GB, we might just update the
> metadata block once per 1% of the synchronization operation's progress.

I think best would be to have it updated every <whatever> seconds.
I think of very fast or very slow drives, or drives which are used
heavily during rebuild (where the effective rebuild speed is quite
low)...