gmirror synchronising is very slow due to frequent metadata updates

[Mark Johnston]

On Sun, Nov 19, 2017 at 11:32:41AM +0100, Andre Albsmeier wrote:
> I am running a 2 TB gmirror with two disks. I replaced one disk and the
> mirror started to synchronise. I noticed that the sync rate was about
> 50 MB/s which is a lot lower than the more than 120 MB/s which are
> possible with this hardware.
> 
> The reason for this are the very frequently happening metadata updates
> done in sys/geom/mirror/g_mirror.c:
> 
>                if (sync->ds_offset_done + (MAXPHYS * 100) < offset) {
>                         /* Update offset_done on every 100 blocks. */
>                         sync->ds_offset_done = offset;
>                         g_mirror_update_metadata(disk);
> 
> With MAXPHYS being 128kB this would mean that metadata is updated every
> 12 MB -- or 4 times a second @ 50 MB/s.
> 
> Out of curiosity I changed this to MAXPHYS * 2000 which means that the
> updating is done every 240 MB. Now the disks are synchronising with
> approx. 105 MB/s...
> 
> Is there any reason why these metadata updates are done so often and
> what are the risks having them more infrequently (as I do now)?

I believe that the metadata updates are done only so that it's possible
to resume a synchronization after a crash. In particular, ds_offset_done
is translated into the sync_offset field in the metadata block of a
synchronizing disk. Therefore, there is no real risk in making the
updates less frequent; it'll just take marginally more time to complete
an interrupted rebuild of a mirror (which would hopefully be a rare
event anyway).

We should probably decrease the update interval based on the size of a
mirror. For mirrors larger than say, 1GB, we might just update the
metadata block once per 1% of the synchronization operation's progress.