8.1-STABLE: zfs and sendfile: problem still exists

[Alexander Zagrebin]

> >> I apologize for my haste, it should have been VM_ALLOC_WIRED.
> > 
> > Ok, applied and tested under some load(~1200 active 
> connections, outgoing
> > ~80MB/s). Patch work as expected and i has noted no side 
> effects.  Just one
> > question - should grow Active memory counter, if some pages 
> is "hot"(during
> > multiple sendfile on one file)?
> 
> Pages used by sendfile are marked as Inactive for faster 
> reclamation on demand.

I have a question.
When we transfer a file via sendfile, then current code allocates
a memory, marked inactive. For example, if the file has size 100 MB,
then 100 MB of memory will be allocated.
If we have to transfer this file again later, then this memory will used
as cache, and no disk io will be required.
The memory will be freed if file will be deleted or operating system
will need an additional memory. 
I have correctly understood?
If it so, the i continue...
Such behaviour is good if we have files with relatively small size.
Suppose we have to transfer file with large size (for example, greater 
than amount of physical memory).
While transfering, the inactive memory will grow, pressing the ARC.
When size of the ARC will fall to its minimum (vfs.zfs.arc_min), then
inactive memory will be reused.
So, when transfer is complete, we have:
1. No free memory
2. Size of the ARC has minimal size (it is bad)
3. Inactive memory contains the _tail_ of the file only (it is bad too)
Now if we have to transfer this file again, then
1. there is no (or few) file's data in ARC (ARC too small)
2. The inactive memory doesn't contain a head part of the file
So the file's data will read from a disk again and again...
Also i've noticed that inactive memory frees relatively slowly,
so if there is a frequent access to large files, then system will run
at very unoptimal conditions.
It's imho...
Can you comment this?

-- 
Alexander Zagrebin