Re: Quick question about geom/g_mirror.c g_mirror_sync_stop() function

On Mon, Aug 22, 2022 at 10:30:15PM +0000, Cavallo, Joseph wrote:
> Hi Mark,
> 
> Thank you for your response while traveling.
> 
> In g_mirror_sync_stop() it quickly calls “free(disk->d_sync.ds_bios, M_MIRROR);”. But at that point can’t these bios still be in progress in the g_mirror_disk.d_sync.ds_consumer or in the g_mirror_disk.ds_consumer?

They can, yes.  Note though that disk->d_sync.ds_bios is an array of
pointers to BIOs, not an array of BIOs, so freeing ds_bios doesn't
impact pending synchronization I/O.  g_mirror_sync_request() will notice
in the BIO_WRITE path that the sync GEOM has gone away and will free the
BIO at that point, or earlier if an I/O error occurred.  Alternately, if
the mirror itself is being destroyed, g_mirror_destroy_provider() will
free any BIOs queued for the worker thread (see the special handling of
synchronization BIOs there).

> In the g_mirror_ctl_remove() path it calls “g_mirror_event_send(disk, G_MIRROR_DISK_STATE_DESTROY, G_MIRROR_EVENT_DONTWAIT);” That leads to “g_mirror_destroy_disk()” which quickly calls “g_mirror_sync_stop()”. I can’t find where it quiesces those bios so that it can free them right then and there.  Later it does the g_mirror_kill_consumer() but that would just jump out when it got to the g_mirror_is_busy() test unless it was all quiesced (I think).
> 
> I’ve run FreeBSD 13.1 with 2 64GB SCSI disks mirror (data volume). It looks like it defaults to 2 sync bios.  I did the remove of the disk getting synched in the middle of a sync, and of course it works fine 100% of the time. It does not crash. But I’m trying to understand how that is. How does it avoid the pitfall that I think I see?
> 
> Many thanks in advance. Take your time. Enjoy your travels.
> 
> --Joe
> 
> From: Mark Johnston <markj@freebsd.org>
> Sent: Saturday, August 20, 2022 2:20 PM
> To: Cavallo, Joseph <Joseph.Cavallo@stratus.com>
> Subject: [EXTERNAL] Re: Quick question about mirror.c
> 
> 
> [EXTERNAL SENDER: This email originated from outside of Stratus Technologies. Do not click links or open attachments unless you recognize the sender and know the content is safe.]
> 
> ________________________________
> Hi Joe,
> 
> I am happy to try to answer questions about that code, but I'm traveling and might take a few days to reply. Please include freebsd-hackers@freebsd.org<mailto:freebsd-hackers@freebsd.org> in the email, in case someone else is able to answer.
> 
> Thanks,
> -Mark
> 
> On Fri., Aug. 19, 2022, 12:08 Cavallo, Joseph, <Joseph.Cavallo@stratus.com<mailto:Joseph.Cavallo@stratus.com>> wrote:
> Hi Mark,
> 
> I reside in MA, USA.
> 
> I'm using the FreeBSD Geom/mirror project that you're a contributor to.
> 
> I have a quick question about the g_mirror_sync_stop() routine. Would you be able to help?
> 
> Thx,  Joe