FreeBSD and poor ata performance

[Bob Willcox]

On Fri, Oct 15, 2004 at 07:41:15PM +0200, fandino wrote:
> Kenneth Culver wrote:
> >>well, my usage pattern is write a big file and few seconds later read 
> >>it. So my tests
> >>were valid for the use of the computer.
> >>
> >>But you have reason, I must provide a more formal report. I redid all 
> >>test
> >>with bonnie++ and results shows Linux (56848 K/sec) two times faster than
> >>FreeBSD (26347 K/sec)
> >>
> >>Any help will be appreciated!
> >>
> >>
> >>Linux test  (slackware 8.1, kernel 2.4.18, ext2 filesystem):
> >
> >
> >This test isn't really a fair test either. The ext2 filesystem uses 
> >async io,
> >and doesn't do any kind of journaling to ensure data integrity in the 
> >event of
> >a crash. FreeBSD isn't using async, it uses softupdates. Because of this
> >FreeBSD SHOULD be slower... but it'll be a lot more reliable than linux 
> >in the
> >event of a power outage for example. The ext2 filesystem is extremely
> >unreliable, and will almost always lose data when there's a crash or power
> >outage.
> 
> but then why does read/write tests over raw devices performs so bad?
> AFAIK on raw devices not filesystem, journaling, caches, etc are involved.

Linux doesn't really have raw disk devices. There is a raw I/O disk
hack (rawio, but I don't recall much about it), but one would normally
not encounter it. All disk I/O is through the disk cache, and in my
experience the size of the cache appears to grow quite large (this was
with a 2.4.x kernel, though the 2.6.x kernel seems to be similar...see
my example below). Consequently, when writing to /dev/sda (for example)
and then quickly reading it back will more likely be measuring memory
performance (on a system with sufficient memory to hold all of the data
in its cache).

Here is an example of the caching behavior on a SUSE SLES9 system with
1GB of physical memory (REISERFS filesystem and a not particularly fast
SCSI disk):

bob at panaka:/tmp> time dd count=1024 bs=1024k if=/dev/zero of=aa; time dd bs=1024k if=aa of=/dev/null
1024+0 records in
1024+0 records out

real    0m35.914s
user    0m0.014s
sys     0m12.477s
1024+0 records in
1024+0 records out

real    0m52.892s
user    0m0.014s
sys     0m7.761s

bob at panaka:/tmp> time dd count=512 bs=1024k if=/dev/zero of=aa; time dd bs=1024k if=aa of=/dev/null
512+0 records in
512+0 records out

real    0m14.602s
user    0m0.006s
sys     0m6.870s
512+0 records in
512+0 records out

real    0m2.316s
user    0m0.004s
sys     0m2.166s

As you can see, in the first example I wrote and reread a 1 GB file. Due
to its size it could not be completely contained in memory (this system
has 1 GB of RAM) and the write and read times (esp. the read) is likely
representitive of the speed of the disk. With the second example I only
wrote and read 512 MB, all of which was apparently contained within the
cache (apparent by the read performance and the fact that it was compute
bound).

FreeBSD doesn't allow the disk cache to grow unbounded as apparently
Linux does. The Linux approach has the advantage of looking good in
relatively small file I/O benchmarks, especially where you write a file
and then immediately read it back. However, in my experience, it suffers
disadvantages in the real world where reading a single large file can
cause lots of data that you care about (e.g., VM pages for running apps)
to be flushed out of memory to accommodate the growing filesystem cache
(the tests of this were conducted on a 2.4.x based kernel a couple of
years ago, things may have changed so YMMV).

Bob

-- 
Bob Willcox             The boss who attempts to impress employees with his
bob at immure.com          knowledge of intricate details has lost sight of his
Austin, TX              final objective.