Very inconsistent (read) speed on UFS2

[Lev Serebryakov]

Hello, Kirk.
You wrote 31 августа 2011 г., 0:09:12:

> Now that you have defragmented your filesystem, we can factor that out
> of the equation. What is left is the management of the memory used for
  Yep.

> caching the files. I expect what is happening is that you are busily
> reading along and run out of free memory in which to read. This triggers
> a cleanup thread that churns through the memory pool to decide what
> should be thrown out to make room. Your reading process is demanding
> memory faster than the cleanup thread can produce it. The result is
> that your read idles (e.g., appears to run slowly). It is random because
> it depends on when you run out of memory.
  It is interesting. But this box have two real cores (yes, I know,
 now it is ``only two cores'' :)) and nothing to do but this test
 program (single-threaded). Of course, this program read data into
 same buffer and didn't allocate anything in process of benchmark.

  Yes, I know, that kernel prefer not to throw away data even if it is
not needed right now, but here decision looks very simple :)

 And, one more detail: I use O_DIRECT flag in open(2).

  Other interesting observing: this program consume about 20% of one
core. It is very strange for I/O bound process, isn't it?

> The cleanup is complex because it has to deal with all of memory and
> it wants to avoid a simple LRU which would cause the read of a large 
> file to throw out a lot of things you would rather keep (like your 
> window manager, browser, etc). Not sure what the best strategy is here.
  window manager and browser look different from one small (128Kb)
 buffer which is ovewritten again and again. As far as I understand,
 FreeBSD's VM (according to "Design and Implementation of...) should
 move such buffers (data read recently, not changed and not needed)
 into "Inact" state and it is easy task to re-use these buffers --
 they don't belong to any active process, they don't need to be paged
 or swapped out, etc.

  I'll try this experiment with mmap() and touching every 4096-th byte of
mapped memory instead of read(2).

-- 
// Black Lion AKA Lev Serebryakov <lev at FreeBSD.org>