Inspect pages created after a vm_object is marked as copy-on-write

[Mihai Carabas]

On Sat, Jun 30, 2018 at 1:52 AM, Mark Johnston <markj at freebsd.org> wrote:
> On Fri, Jun 29, 2018 at 11:58:31AM +0300, Elena Mihailescu wrote:
>>  Hello,
>>
>> I am interested if there is a method to inspect what pages/objects were
>> created after a vm_object (the vm_map_entry associated with the object) is
>> marked as copy-on-write. More specifically, I'm interested only in the
>> pages that were copied when a write operation was proceed on a page that
>> belongs to the object marked copy-on-write.
>>
>> I need this for a live migration feature for bhyve in order to send the
>> pages that were modified between the iterations in which I migrate the
>> guest's memory(the guest's memory will be migrated in rounds - firstly, all
>> memory will be sent remote, then, only the pages that were modified and so
>> on).
>>
>> What I want to implement is the following:
>> Step 1: Given a vm_object *obj, mark its associated vm_map_entry *entry as
>> copy-on-write.
>> Step 2: After a while (a non-deterministic amount of time),
>> inspect/retrieve the pages that were created based on information existent
>> in the object.
>>
>> What I tried until now:
>>
>> I implemented a function in kernel that:
>> - gets the vmspace structure pointer for the current process
>> - gets the vm_map structure pointer for the vmspace
>> - iterates through each vm_map_entry and based on the vm_offset_start and
>> vm_offset_end determines vm_map_entry that contains the object I am
>> interested in.
>> - for this object, it prints some debug information such as: shadow_count,
>> ref_count, whether if it has a backing_object or not.
>> The code written is similar with the code from here (the way in which I get
>> vmspace for the current process and the way I am iterating through
>> vm_map_entry and objects):
>> [0]
>> https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration/sys/amd64/vmm/vmm_dev.c#L979
>>
>> I have read the following documentation about FreeBSD's implementation for
>> virtual memory:
>> [1] https://www.freebsd.org/doc/en/books/arch-handbook/vm.html
>> [2]
>> https://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/article.html
>> [3] https://people.freebsd.org/~neel/bhyve/bhyve_nested_paging.pdf
>> [4] http://www.cse.chalmers.se/edu/year/2011/course/EDA203/unix4.pdf
>>
>> As far as I could tell after reading the documentation presented above, I
>> should look for the object that the object I am interested in is a shadow
>> of or an object that my object is shadow for.
>
> Right. When a copy-on-write fault results in the creation of a new
> object, the new object is said to shadow the original object, which
> becomes the backing object for the shadow object.  When faults in the
> corresponding map entry occur, the fault handler first searches for a
> page in the map entry's object, and then falls back to the backing
> object if necessary.
>
>> To do that, I should inspect the following fields from the vm_object
>> structure (among others)(
>> https://github.com/freebsd/freebsd/blob/master/sys/vm/vm_object.h#L98) :
>>
>> - int shadow_count; /* how many objects that this is a shadow for */
>> - struct vm_object *backing_object; /* object that I'm a shadow of */
>>
>> But in all my tests, for the object I am interested in, the shadow_count is
>> 0 and the backing_object is NULL.
>>
>> The code I use to mark the vm_map_entry for the object I am interested in
>> copy-on-write is here:
>> [5]
>> https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration/sys/amd64/vmm/vmm_dev.c#L949
>
> MAP_ENTRY_NEEDS_COPY needs to be set in order for the copy-on-write
> machinery to work the way you expect.  Take a look at vm_map_lookup():
> when it sees that flag and the caller is attempting a write fault, it
> creates a shadow object, updates the entry and clears
> MAP_ENTRY_NEEDS_COPY, leaving MAP_ENTRY_COW set.
>
>> Is there anything I am doing wrong? Maybe I misunderstood something about
>> the way the virtual memory works in FreeBSD.
>
> I'll note that inspecting and manipulating vm_map_entry and vm_object
> structures in the bhyve code constitutes something of an abstraction
> violation, though it's reasonable to proceed this way while working on a
> prototype of the feature.  That is, I think you should keep trying your
> current approach, but just be aware that you are using the copy-on-write
> mechanism in a way that the VM system isn't really expecting.
>

Can you point out the right approach in our case?

Thanks,
Mihai

>> There is another way I could inspect what pages were created between the
>> moment I mark an object (its vm_map_entry) as copy-on-write and a later
>> moment?
> _______________________________________________
> freebsd-virtualization at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe at freebsd.org"