Grant Table Userspace Device - Status Update

[Akshay Jaggi]

Carrying over discussion from IRC.

20:11 royger: ghost_rider: hello! I've been doing some testing with the
> device today, and it seems there's a memory leak somewhere, after shutting
> down all my domains I still see 1KB of memory used by the device, which
> AFAICT is not expected (you can check with `vmstat -m |grep gntdev`)
>

Nope. That's not a leak.

I ran `vmstat -m | grep gntdev` just after booting up Dom0, without any of
the DomU's running, and I still saw 1KB of memory being used by the device.

root at freebsd:~ # vmstat -m | grep gntdev
       gntdev     2     1K       -        2  64

That is, 2 requests have been made, out of which both are currently active,
without any DomU's active.

After this I fired up a DomU with qdisk backends, and vmstat returned:

root at freebsd:~/xen_test # vmstat -m | grep gntdev
       gntdev  2129   134K       -     2137  32,64,128

Well in line with expectations. Now, powering off the DomU and running
vmstat again, we get:

root at freebsd:~/xen_test # vmstat -m | grep gntdev
       gntdev     2     1K       -     2845  32,64,128

The initial 2 requests are still active, and this has nothing to do with
the DomU's. The first malloc() that happens in the device is in the device
open function at [1]. That means that someone has the device open. `fstat`
confirmed my suspicions.

`fstat` with DomU active:

root at freebsd:~/xen_test # fstat | grep xen/gntdev
root     qemu-system-i386  1266   29 /dev         62 crw-------  xen/gntdev
rw
root     qemu-system-i386  1266   32 /dev         62 crw-------  xen/gntdev
rw
root     qemu-system-i386  1266   34 /dev         62 crw-------  xen/gntdev
rw
root     xenconsoled   751    6 /dev         62 crw-------  xen/gntdev rw
root     xenstored    746   11 /dev         62 crw-------  xen/gntdev rw

`fstat` with DomU powered off:

root at freebsd:~ # fstat | grep xen/gntdev
root     xenconsoled   751    6 /dev         62 crw-------  xen/gntdev rw
root     xenstored    746   11 /dev         62 crw-------  xen/gntdev rw

So yep! It's no leak. Just that xenconsoled and xenstored keep the gntdev
device open. I guess this would be expected behaviour. Let me know if it is
not.

20:14 royger: ghost_rider: and I've also seen a "Can't find requested
> grant-map." after attaching 4 Qdisk to a domain and done heavy IO to to
> them.
> 20:16 royger: although this last one I haven't been able to reproduce
>

That's pretty strange. I have never noticed this in any of my manual or
stress tests.

At this point I would also like to mention, that the xen-gnttab code is
kind of buggy (putting it mildly, no offence).
Like I pointed out in the xen-devel patch thread, there is a place in code
where "-1" is being used to specify there is no CLEAR_BYTE notify. But this
is not being checked for inside the function, which would have caused a
clear-byte notification on a different page, causing data corruption. The
only reason this bug is not doing so, is because of another bug, where this
-1 is being passed on to an unsigned int32, which would keep it out of
bounds for most requests.

I don't think this has to do anything with our device. If we lost some
unmap request (which is where this message is generated) we would have
surely leaked the memory for the gmap structure associated with that
request (because, 1. ref-counting, 2. transferred to global clean list only
on an unmap request), and that would have been visible in `vmstat`.

Let me know if this repeats.


> 20:40 royger: and I'm not sure if you tested it, but if you attach a
> ramdisk to a VM (one created with `mdconfig -t malloc ...`) and try to run
> newfs against it, it doesn't work, a bunch of read errors appear on both
> the DomU console and Qemu log. Although it works with a plain file, so I
> guess this is probably some bad interation between Qemu and FreeBSD block
> devices...


Mhm. Sounds like that. I'll try it out on my setup and post the results.

Regards,
Akshay

[1] https://github.com/akshay1994/freebsd/blob/grant_table/sys/
dev/xen/gntdev/gntdev.c#L896

On 18 August 2016 at 23:21, Akshay Jaggi <akshay1994.leo at gmail.com> wrote:

> Hello Everyone,
>
> Hope everyone is doing great.
>
> This is the last week of GSoC-2016, and the grant table userspace device
> is complete. We are actively working to get the code reviewed and pushed.
>
> Here is the list of patches in the pipeline:
>
>    - https://reviews.freebsd.org/D7429
>    - https://reviews.freebsd.org/D7293
>    - https://lists.xenproject.org/archives/html/xen-devel/2016-
>    08/msg02125.html
>
> I'll be using this thread to discuss the overall progress of code-review
> for all the three patches, and I hope that would be fine.
>
> Looking forward to getting this stuff to mainline. :)
>
> Regards,
> Akshay
>
> On 20 July 2016 at 20:33, Akshay Jaggi <akshay1994.leo at gmail.com> wrote:
>
>> Hello Everyone,
>>
>>
>> I hope everyone is doing great.
>>
>>
>> First things first. I want to apologise for not updating my blogpost
>> after June 15. Some personal stuff happened on June 17, and since then all
>> my work routines are completely disturbed. Soc-admins know about this, and
>> I hope the community can pardon my irregularity this once.
>>
>>
>> Now, time for some status update on the project. We have a working Grant
>> Table User-space Device (with some caveats, of course). Yay!!
>>
>> Check out the code on the pull request: https://github.com/fr
>> eebsd/freebsd/compare/master...akshay1994:grant_table?expand=1
>>
>> Test the device by compiling a custom kernel; from my working branch:
>> https://github.com/akshay1994/freebsd/tree/grant_table
>>
>>
>> Caveats and work yet to be done:
>>
>>    - A kernel panic happens if you terminate the program without
>>    un-mmapping all the mmaps.
>>    - The interface for a grant map is not completely in sync with the
>>    Linux interface yet. Notably, we still map and subsequently mmap partially
>>    working grants. This is a bug since the pager will fail on a page fault for
>>    the non-working-grant.
>>    - I'm yet to add event channel code for the notify ioctls.
>>    - I'm yet to get in touch with the vm-guys. I spent a lot of time
>>    understanding the vm structure, but I surely do not have a complete
>>    picture. I'm using a MGTDEVICE cdev pager to handle managed, fictitious
>>    pages from mapped grants, and a custom PHYS vm object to manage wired
>>    physical memory for allocated grants. I have to
>>       - Confirm if the approach I am using is the best one.
>>       - The interface differs compared to Linux with regards to multiple
>>       mmaps, which I need a little help with.
>>
>>
>> Looking forward to some community testing of the working parts.
>> Comments are welcome and highly appreciated. :)
>>
>>
>> Regards,
>> Akshay Jaggi
>>
>
>