Inspect pages created after a vm_object is marked as copy-on-write

[Mark Johnston]

On Sun, Jul 01, 2018 at 01:34:01AM +0300, Konstantin Belousov wrote:
> On Sat, Jun 30, 2018 at 05:59:56PM -0400, Mark Johnston wrote:
> > On Sat, Jun 30, 2018 at 10:38:21AM +0300, Mihai Carabas wrote:
> > > On Sat, Jun 30, 2018 at 1:52 AM, Mark Johnston <markj at freebsd.org> wrote:
> > > > On Fri, Jun 29, 2018 at 11:58:31AM +0300, Elena Mihailescu wrote:
> > > >> Is there anything I am doing wrong? Maybe I misunderstood something about
> > > >> the way the virtual memory works in FreeBSD.
> > > >
> > > > I'll note that inspecting and manipulating vm_map_entry and vm_object
> > > > structures in the bhyve code constitutes something of an abstraction
> > > > violation, though it's reasonable to proceed this way while working on a
> > > > prototype of the feature.  That is, I think you should keep trying your
> > > > current approach, but just be aware that you are using the copy-on-write
> > > > mechanism in a way that the VM system isn't really expecting.
> > > >
> > > 
> > > Can you point out the right approach in our case?
> > 
> > I am merely suggesting that once the required VM interactions are fully
> > understood, the mechanism implemented for bhyve should be generalized
> > and lifted into the VM code.  It's hard to say what the "right" approach
> > is, since I don't fully understand the proposed algorithm.  It sounds
> > like you might be attempting something like:
> > 
> > 1. mark the mappings of to-be-migrated objects as NEEDS_COW, so that a
> >    subsequent write fault triggers creation of a shadow object
> It is actually MAP_ENTRY_COW | MAP_ENTRY_NEEDS_COPY.

Indeed.

> Note that setting an entry to COW changes the behaviour of mprotect(2),
> at least.
> 
> > 2. invalidate all physical mappings of pages in the object to be copied,
> >    so that subsequent writes trigger a fault
> I do not think this is needed to detect writes after the COW is set.
> It is enough to remove the write permissions.  Same as fork() does,
> see the vm_map_copy_entry() code for the handling of MAP_ENTRY_NEEDS_COPY
> case.

Ah, right.

> > 3. copy pages from the backing object to the destination
> As I understand, this is done right after the entry is marked
> as COW.
> 
> > 4. copy any pages from the shadow object to the desination
> And this is done after all backing data is copied and the process is
> suspended.

Right, I see.  Some reading suggests that in general we might perform
multiple iterations of this procedure before suspending the process and
performing any remaining copies.

> > 5. collapse the backing object into the shadow
> > 6. if the shadow object exists and was non-empty before the collapse,
> >    goto 1
> Are you trying to describe how to undo the COW marking ?  Marking an
> entry as COW really changes its semantic, and we do not need the undo
> operation in the base so far.  Collapsing the objects would lesser
> the pressure on the system pollution with objects, but it does not
> change back the meaning of mappings, e.g. their behaviour on inheritance
> on fork.

I was just thinking about how to avoid creating a long chain of objects
if multiple iterations are in fact needed.