Poor ZFS prefetch performance

[Andrew Snow]

On latest 8-current with latest patches, I am seeing poor data prefetch 
hit rate with my workload of rsync on millions of files.

ZFS has been quite good apart from this - UFS was much faster at this 
particular workload.  ZFS just seems to be reading many megabytes over 
and over from the filesystem with little to show for it.

Arcstats reports the following:

kstat.zfs.misc.arcstats.prefetch_data_hits: 475476
kstat.zfs.misc.arcstats.prefetch_data_misses: 5325057
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 126411515
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 3998476


I don't really want to turn of prefetching completely because it does 
actually help in many situations.

 From this I deduce it would be really nice to provide an extra tunable 
that allows you to turn off prefetching for data but leave it on for 
metadata.

This would be easy to do because data prefetches come in through 
dmu_zfetch, but metadata comes through dmu_prefetch() in dmu.c


- Andrew