buf(9) woes: when does bcopy do nothing at all?

[Alan Somers]

On Thu, Apr 25, 2019 at 9:31 PM Alan Somers <asomers at freebsd.org> wrote:
>
> How is it possible that bcopy() doesn't affect its output array at
> all?  While investigating a data corruption issue in fuse, I narrowed
> the problem down to a bcopy operation that apparently has no affect.
> The code in question is:
>
> bcopy(cp, iov->iov_base, cnt);
> r = memcmp(cp, iov->iov_base, cnt);
> if (r)
>     printf("uiomove: miscompare\n");
>
> Rationally, I would expect that line never to be printed.  But it
> does.  The destination is always all zeros, even though the source is
> not.  I can only guess that there's something wrong about the way that
> I I'm using buf(9), because the output is part of a buffer allocated
> by bread(9).  I've been able to rule out:
>
> 1) Race conditions.  The bug is 100% reproducible, and doubling the
> bcopy or changing the timing in other ways has no effect.
> 2) Unmapped buffer.  I verified that the buf is not unmapped_buf.
> 3) Overlapping src and dst
> 4) Duplicated pages.  I verified that each of the buf's pages has a
> unique physical address
> 5) Bad RAM.  My machine passes memtest86, and anyway the failure is
> too specific and reproducible to be caused by bad hardware.
>
> What could I be missing?  Do I need to do something to prepare the buf
> before I can use it?  The code that allocates the buffer is here:
> https://svnweb.freebsd.org/base/projects/fuse2/sys/fs/fuse/fuse_io.c?view=markup#l240
>
> -Alan

To answer phk's questions, I checked that src and dst don't overlap,
and the kernel's bcopy is actually a wrapper around memmove.

To answer hps's question, this is on amd64, in a bhyve VM.

I solved the problem - part of it, anyway.  The user-visible problem
that originally led me down this rabbit-hole was an apparent cache
invalidation failure during writes on fusefs.  That turned out to be
caused by an off-by-one error that I just fixed in r346756.  However,
the miscompares remain.  Could those pages be mapped differently for
reading than for writing?  I don't know.  At this point, I'm not going
to put much more effort into investigating the problem; I've wasted
too much time already.

-Alan