8.1-STABLE: zfs and sendfile: problem still exists

[Andriy Gapon]

on 31/10/2010 11:02 Alexander Zagrebin said the following:
> I have a question.
> When we transfer a file via sendfile, then current code allocates
> a memory, marked inactive. For example, if the file has size 100 MB,
> then 100 MB of memory will be allocated.
> If we have to transfer this file again later, then this memory will used
> as cache, and no disk io will be required.
> The memory will be freed if file will be deleted or operating system
> will need an additional memory. 
> I have correctly understood?
> If it so, the i continue...
> Such behaviour is good if we have files with relatively small size.
> Suppose we have to transfer file with large size (for example, greater 
> than amount of physical memory).
> While transfering, the inactive memory will grow, pressing the ARC.
> When size of the ARC will fall to its minimum (vfs.zfs.arc_min), then
> inactive memory will be reused.
> So, when transfer is complete, we have:
> 1. No free memory
> 2. Size of the ARC has minimal size (it is bad)
> 3. Inactive memory contains the _tail_ of the file only (it is bad too)
> Now if we have to transfer this file again, then
> 1. there is no (or few) file's data in ARC (ARC too small)
> 2. The inactive memory doesn't contain a head part of the file
> So the file's data will read from a disk again and again...
> Also i've noticed that inactive memory frees relatively slowly,
> so if there is a frequent access to large files, then system will run
> at very unoptimal conditions.
> It's imho...
> Can you comment this?
> 

First and foremost, the double-caching issue for ZFS+sendfile on FreeBSD is
still there and no resolution for this issue is on horizon.  So, you have to
account for the fact that twice as much memory is needed for this use-case.
Whether you plan your system, or configure it, or tune it.

Second, with recent head and stable/8 ARC should not be the primary victim of
memory pressure; ARC reclaim thread and the page daemon should cooperate in
freeing/recycling memory.

Nothing much to add.

-- 
Andriy Gapon