[Bug 246886] [sendfile] Nginx + NFS or FUSE causes VM stall

From: <bugzilla-noreply_at_freebsd.org>
Date: Fri, 17 Jun 2022 16:52:41 UTC

--- Comment #70 from firk@cantconnect.ru ---
So, the source of the problem seems was base r337165 . I didn't tested
revisions around, but I did tested rollback of this specific comit, setting
f_iosize back to PAGE_SIZE=4096 and the problem is gone. But this is not the
bug itself, it is just a trigger for another problem(s).

As already noted, backtrace for the deadlock is:

 allocbuf+0x371 (vfs_vmio_extend inlined inside)
 vn_sendfile+0xdf2 (sendfile_swapin inlined inside)

What happens:

1) sendfile_swapin() grabs and exclusively-busies bunch of pages via

2) it then scans them sequentially, unbusies already loaded ones, and calls
vm_pages_get_pages_async() for not yet loaded ones, which should load them and
call sendfile_iodone() callback, and that how it was in 11.x;

3) vm_pages_get_pages_async() calls some other nested functions, and now we are
in vfs_bio_getpages(). Note: despite the "async" name all this done

4) vfs_bio_getpages() still have vm_page[] array and its size as arguments,
passed unchanged straightly from sendfile_swapin(); it downgrades
exclusive-busy state to shared-busy for the given pages range;

5) the next step (bread_gb -> breadn_flags) is done using block index and size
obtained from fusefs driver via get_lblkno() and get_blksize() callbacks, and
the new block size is 65536 by default. And, going through getblkx() ->
allocbuf() -> vfs_vmio_extend(), the last one calls vm_page_grab_pages() again,
but the range is not the requested one, but the one matches fusefs block size,
effectively aligned to 16-block boundary (65536 = 16*4096). This leads to
deadlock because the pages after currently requested are still exclusively-busy
(see p.2)

What could be fixed:

1) easiest: rollback iosize to PAGE_SIZE, but this will reduce i/o speed back

2) rework sendfile_swapin() to first scan entire range for being loaded or not
and only then calling queued vm_pager_get_pages_async(); don't think it is good
because everythink already works when fusefs/nfs not used.

3) make "async" functions really async (see p.3) for fusefs; i don't know if it
easy or not - this will resolve deadlock too because vfs_bio_getpages() will
not block the sequential scan of requested pages by sendfile_swapin()

4) prevent partially loaded filesystem f_iosize blocks from happening; again, I
don't know is it easy or even desirable or not.

PS: I don't know how all this works or not in 13.x

You are receiving this mail because:
You are the assignee for the bug.