Inspect pages created after a vm_object is marked as copy-on-write

[Elena Mihailescu]

 Hello,

I am interested if there is a method to inspect what pages/objects were
created after a vm_object (the vm_map_entry associated with the object) is
marked as copy-on-write. More specifically, I'm interested only in the
pages that were copied when a write operation was proceed on a page that
belongs to the object marked copy-on-write.

I need this for a live migration feature for bhyve in order to send the
pages that were modified between the iterations in which I migrate the
guest's memory(the guest's memory will be migrated in rounds - firstly, all
memory will be sent remote, then, only the pages that were modified and so
on).

What I want to implement is the following:
Step 1: Given a vm_object *obj, mark its associated vm_map_entry *entry as
copy-on-write.
Step 2: After a while (a non-deterministic amount of time),
inspect/retrieve the pages that were created based on information existent
in the object.

What I tried until now:

I implemented a function in kernel that:
- gets the vmspace structure pointer for the current process
- gets the vm_map structure pointer for the vmspace
- iterates through each vm_map_entry and based on the vm_offset_start and
vm_offset_end determines vm_map_entry that contains the object I am
interested in.
- for this object, it prints some debug information such as: shadow_count,
ref_count, whether if it has a backing_object or not.
The code written is similar with the code from here (the way in which I get
vmspace for the current process and the way I am iterating through
vm_map_entry and objects):
[0]
https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration/sys/amd64/vmm/vmm_dev.c#L979

I have read the following documentation about FreeBSD's implementation for
virtual memory:
[1] https://www.freebsd.org/doc/en/books/arch-handbook/vm.html
[2]
https://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/article.html
[3] https://people.freebsd.org/~neel/bhyve/bhyve_nested_paging.pdf
[4] http://www.cse.chalmers.se/edu/year/2011/course/EDA203/unix4.pdf

As far as I could tell after reading the documentation presented above, I
should look for the object that the object I am interested in is a shadow
of or an object that my object is shadow for.
To do that, I should inspect the following fields from the vm_object
structure (among others)(
https://github.com/freebsd/freebsd/blob/master/sys/vm/vm_object.h#L98) :

- int shadow_count; /* how many objects that this is a shadow for */
- struct vm_object *backing_object; /* object that I'm a shadow of */

But in all my tests, for the object I am interested in, the shadow_count is
0 and the backing_object is NULL.

The code I use to mark the vm_map_entry for the object I am interested in
copy-on-write is here:
[5]
https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration/sys/amd64/vmm/vmm_dev.c#L949

Is there anything I am doing wrong? Maybe I misunderstood something about
the way the virtual memory works in FreeBSD.
There is another way I could inspect what pages were created between the
moment I mark an object (its vm_map_entry) as copy-on-write and a later
moment?

Thank you,
Elena