Very inconsistent (read) speed on UFS2

[Kirk McKusick]

> Date: Wed, 31 Aug 2011 02:29:33 +0400
> From: Lev Serebryakov <lev at FreeBSD.org>
> To: Kirk McKusick <mckusick at mckusick.com>, freebsd-fs at FreeBSD.org
> Subject: Re: Very inconsistent (read) speed on UFS2
> 
> Hello, Kirk.
> 
> >   I'll try this experiment with mmap() and touching every 4096-th byte of
> > mapped memory instead of read(2).
> 
>   Strange enough, it gives only 40-50MiB/s and results are very
>  consistent.
> 
>   It really surprise me. I didn't think, that there will be so much
> difference, I was sure, that it will be almost equivalent speed.
> 
> --
> // Black Lion AKA Lev Serebryakov <lev at FreeBSD.org>

I had not realized that you were using O_DIRECT. That would in fact
avoid most of the caching / memory-recovery effects that I was blaming
earlier. Your test above is definitely hitting them though. My guess
is that the consistency is because you are measuring the rate at which
free memory can be created.

So, my new theory on why your O_DIRECT test is running slowly is due
to the single threading in the GEOM layer. Pawel Jakub Dawidek (pjd@)
gave a very interesting talk on this problem at this year's BSDCan.

	Kirk McKusick