[ZFS] ARC accounting bug ?

[Ben RUBSON]

> On 30 Aug 2016, at 15:37, Gary Palmer <gpalmer at freebsd.org> wrote:
> 
> On Sat, Aug 27, 2016 at 06:15:18PM +0200, Ben RUBSON wrote:
>> Playing with these commands :
>> # dtrace -n 'sdt:zfs::arc-hit {@[execname, stack()] = count();}'
>> # dtrace -n 'sdt:zfs::arc-miss {@[execname, stack()] = count();}'
>> 
>> We can see that these are readdir calls which produce arc-misses, and that readdir calls also produce arc-hits.
>> 
>> It would be interesting to know why some lead to hits, and some lead to misses.
>> 
>> (note that ls -lR / rsync commands produces exactly the same dtrace results/numbers as find command)
> 
> If the *same* readdir() call produces both a hit and a miss, my guess would
> be that it is hitting two data sources for the data.

I investigated further, and found that :
- readdir() first produces a miss when it returns the "." entry.
- readdir() then does not produce any miss for the other entries.
- readdir() finally produces a miss when it does not return any value (no more entry).

I also noted that misses are not produced every time, it seems to depend on the folder content.

It is possible to generate so many misses when using find in a loop on a huge folder that if these misses were finally redirect to disks, zpool iostat would show use some IOs.
But this is not the case.
It looks strange.
Cache hits statistics can then be degraded even if there are no disk IOs at all.

> I am not familiar with ZFS, however UFS treats a directory as a special 
> type of file.  Filename information and the inode associated with that
> filename would be in the directory "file.  The inodes would be in the inode
> table.
> 
> If ZFS is similar, then the directory "file" reads would be misses and
> the inode table reads would be hits
> 
> Regards,
> 
> Gary

Thank you for your answer Gary !

Regards,

Ben