Process stuck in "vnread"

[Maxim Sobolev]

Andriy, this is file that gets copies from md(4)-baked UFS to file that is
located on ZFS zraid volume that is mounted via NULLFS. The file that backs
up md(4) is located on ZFS, so in a sense we have full cycle with the
backing block starting on ZFS and ending up on ZFS too. cp(1) calls
write(), with a pointer to mmaped area and the page is not mapped in, so it
triggers pagein and waits for the page to arrive. In any case, it looks
like your patch from D5738 might be working, I've put it into my 10.3-rc3
buildbox and give it some shake to see if it improves things or not. I've
also made sure that I have all debug symbols installed, so that I can poke
inside zfs.ko if I need to.

Thanks, guys, Andriy what keep you from pushing that patch into the tree?

On Mon, Mar 28, 2016 at 10:19 AM, Andriy Gapon <avg at freebsd.org> wrote:

> On 28/03/2016 19:23, Konstantin Belousov wrote:
> > On Mon, Mar 28, 2016 at 08:52:03AM -0700, Maxim Sobolev wrote:
> >> Done some head scratching, it looks like it's got page fault in the
> >> copyin() (cp(1) AFAIK mmaps source file). There might be some interlock
> >> issue between competing write to the same ZFS, the md0 device is locked
> >> forever waiting for the write operation to complete at the very same
> time.
> >> I am curious as to whether we are allowed to sleep in the
> dmu_write_uio_dbuf(),
> >> AFAIK dmu is ZFS's transaction layer, so maybe copyin() should be done
> >> earlier to avoid possible page fault in there?
>
> Maxim,
>
> is this copy from UFS to ZFS?
> It looks like that because the copyin() fault goes to
> vnode_pager_generic_getpages() -> bwait()...
>
> > No idea about ZFS, but if the issue is due to copyin(9) recursing into
> > VM and then VFS while owning file system locks, it is well-known and
> > long-standing issue. I sometimes call it 'ups deadlock', for some
> > reasons, see tools/test/upsdl/ for the distilled test case.
> >
> > It is handled for UFS and NFS, read the long comment starting with 'The
> > vn_io_fault() is a wrapper' in sys/kern/vfs_vnops.c, which describes the
> > deadlock in details and explains the mechanism which is used to prevent
> > it. Filesystems must opt-in into it by specifiying MNTK_NO_IOPF flag,
> > and then being ready to get an array of pages for io instead of the
> buffer
> > KVA.
>
>
> I don't have any idea why the thread would be stuck in bwait() and what
> locks
> and threads are involved here.  But, as Kostik said, there is a general
> problem
> and I have a patch for ZFS:
> https://reviews.freebsd.org/D2790
>
> --
> Andriy Gapon
>



-- 
Maksym Sobolyev
Sippy Software, Inc.
Internet Telephony (VoIP) Experts
Tel (Canada): +1-778-783-0474
Tel (Toll-Free): +1-855-747-7779
Fax: +1-866-857-6942
Web: http://www.sippysoft.com
MSN: sales at sippysoft.com
Skype: SippySoft