ZFS file caching question

[Kris Kennaway]

Andrew Gallatin wrote:
> If I have, say, 512MB RAM and a 1GB file which is written or read
> sequentially on an otherwise idle system, I'd expect the last 512MB (-
> epsilon) of the file's pages to be cached in RAM.  This is true for
> UFS, but it does not appear to be the case with ZFS (see example
> below).
> 
> Can somebody explain how the arc cache in ZFS relates to the normal
> page cache used by traditional filesystems?  Are ZFS filesystems
> cached exclusively in the arc cache, and not in the page cache?  Is
> the arc cache per-filesystem, per-pool, or global for ZFS as a whole?

The ZFS arc cache is completely independent from the normal buffer cache 
on FreeBSD.  This is inefficient in a number of ways.  I have also seen 
things that make me suspicious that it is not caching properly even when 
you tune it to be "large enough" (if possible given memory constraints), 
but I haven't confirmed this.

> Hmm.. Could this be the cause of the problems with ZFS and mmap'ed files?

What problems do you mean?  There were coherency problems but I think 
they were fixed.

Kris