Inspect pages created after a vm_object is marked as copy-on-write

[Mark Johnston]

On Fri, Jun 29, 2018 at 11:58:31AM +0300, Elena Mihailescu wrote:
>  Hello,
> 
> I am interested if there is a method to inspect what pages/objects were
> created after a vm_object (the vm_map_entry associated with the object) is
> marked as copy-on-write. More specifically, I'm interested only in the
> pages that were copied when a write operation was proceed on a page that
> belongs to the object marked copy-on-write.
> 
> I need this for a live migration feature for bhyve in order to send the
> pages that were modified between the iterations in which I migrate the
> guest's memory(the guest's memory will be migrated in rounds - firstly, all
> memory will be sent remote, then, only the pages that were modified and so
> on).
> 
> What I want to implement is the following:
> Step 1: Given a vm_object *obj, mark its associated vm_map_entry *entry as
> copy-on-write.
> Step 2: After a while (a non-deterministic amount of time),
> inspect/retrieve the pages that were created based on information existent
> in the object.
> 
> What I tried until now:
> 
> I implemented a function in kernel that:
> - gets the vmspace structure pointer for the current process
> - gets the vm_map structure pointer for the vmspace
> - iterates through each vm_map_entry and based on the vm_offset_start and
> vm_offset_end determines vm_map_entry that contains the object I am
> interested in.
> - for this object, it prints some debug information such as: shadow_count,
> ref_count, whether if it has a backing_object or not.
> The code written is similar with the code from here (the way in which I get
> vmspace for the current process and the way I am iterating through
> vm_map_entry and objects):
> [0]
> https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration/sys/amd64/vmm/vmm_dev.c#L979
> 
> I have read the following documentation about FreeBSD's implementation for
> virtual memory:
> [1] https://www.freebsd.org/doc/en/books/arch-handbook/vm.html
> [2]
> https://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/article.html
> [3] https://people.freebsd.org/~neel/bhyve/bhyve_nested_paging.pdf
> [4] http://www.cse.chalmers.se/edu/year/2011/course/EDA203/unix4.pdf
> 
> As far as I could tell after reading the documentation presented above, I
> should look for the object that the object I am interested in is a shadow
> of or an object that my object is shadow for.

Right. When a copy-on-write fault results in the creation of a new
object, the new object is said to shadow the original object, which
becomes the backing object for the shadow object.  When faults in the
corresponding map entry occur, the fault handler first searches for a
page in the map entry's object, and then falls back to the backing
object if necessary.

> To do that, I should inspect the following fields from the vm_object
> structure (among others)(
> https://github.com/freebsd/freebsd/blob/master/sys/vm/vm_object.h#L98) :
> 
> - int shadow_count; /* how many objects that this is a shadow for */
> - struct vm_object *backing_object; /* object that I'm a shadow of */
> 
> But in all my tests, for the object I am interested in, the shadow_count is
> 0 and the backing_object is NULL.
> 
> The code I use to mark the vm_map_entry for the object I am interested in
> copy-on-write is here:
> [5]
> https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration/sys/amd64/vmm/vmm_dev.c#L949

MAP_ENTRY_NEEDS_COPY needs to be set in order for the copy-on-write
machinery to work the way you expect.  Take a look at vm_map_lookup():
when it sees that flag and the caller is attempting a write fault, it
creates a shadow object, updates the entry and clears
MAP_ENTRY_NEEDS_COPY, leaving MAP_ENTRY_COW set.

> Is there anything I am doing wrong? Maybe I misunderstood something about
> the way the virtual memory works in FreeBSD.

I'll note that inspecting and manipulating vm_map_entry and vm_object
structures in the bhyve code constitutes something of an abstraction
violation, though it's reasonable to proceed this way while working on a
prototype of the feature.  That is, I think you should keep trying your
current approach, but just be aware that you are using the copy-on-write
mechanism in a way that the VM system isn't really expecting.

> There is another way I could inspect what pages were created between the
> moment I mark an object (its vm_map_entry) as copy-on-write and a later
> moment?