(LONG) ATA Benchmark: 5.x Reads Slower than Writes

Danny Howard dannyman at toldme.com
Fri Apr 8 13:41:37 PDT 2005


Hello,

BACKGROUND

I need to purchase a new system for our developers, for use as a
Postgres database test server.  Having a RAID, probably RAID1, is
desirable for performance and reliability.  I have recently set up a
system with a gmirror-based software RAID1 on a pair of 250GB ATA
drives.  I would like to stick with gmirror, because:

- We save money on extra hardware.
- Since gmirror is part of FreeBSD, maintenance is a lot easier than
  with a hardware solution.

SUMMARY

But before I go balls out, I should see how well it compares to hardware
RAID.  So, I do some benchmarks with bonnie++.  Since this simulates
create, write and read on thousands of random files, this sounds like a
good approximation of what Postgres does. :)

I don't have the time and hardware to do very scientific tests, but I
have been able to run a series of benchmarks using bonnie++ on some
systems I have available to me.  The ATA-based gmirror performs
extremely well, compared to a few Adaptec RAIDs that we have, EXCEPT
that the sequential and random reads are MUCH SLOWER than the hardware
solution, and even *slower than the preceding write operations*.  This
is counter-intuitive, especially since RAID1 implies slowed writes and
faster reads.  I tried the benchmark on my workstation (single 2.5" IDE
in a laptop) and got comparable write-faster-than-read results.

DATA

I was able to make use of the following test systems.  I ran tests in
multi-user, but tried to favor times when there wasn't much background
activity:

mito: (lone 2.5" ATA)
5.4-PRERELEASE
CPU: Intel(R) Pentium(R) M processor 1.50GHz (1495.16-MHz 686-class CPU)
atapci0: <Intel ICH4 UDMA100 controller>
ata0: channel #0 on atapci0
ata1: channel #1 on atapci0
ad0: 28615MB <TOSHIBA MK3021GAS/GA129D> [58140/16/63] at ata0-master UDMA100

amun: (gmirror RAID1 2 x WD250GB High Intensity)
FreeBSD 5.3-RELEASE
CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2799.22-MHz 686-class CPU)
atapci0: <SiI 3112 SATA150 controller>
ata2: channel #0 on atapci0
ata3: channel #1 on atapci0
ad4: 238475MB <WDC WD2500SD-01KCB0/08.02D08> [484521/16/63] at 
ata2-master SATA150
ad6: 238475MB <WDC WD2500SD-01KCB0/08.02D08> [484521/16/63] at 
ata3-master SATA150
atapci1: <SiI 3112 SATA150 controller>
acd0: CDROM <CD-224E/1.9A> at ata1-master UDMA33

janus: (Adaptec RAID1 2 x 72G 10,000 RPM SCSI)
FreeBSD 4.10-STABLE
CPU: Intel(R) Xeon(TM) CPU 2.40GHz (2399.33-MHz 686-class CPU) [DUAL]
aac0: <Adaptec SCSI RAID 2120S> mem 0xf8000000-0xfbffffff irq 16 at 
device 1.0 on pci2
aac0: i960RX 100MHz, 48MB cache memory, optional battery present
aac0: Kernel 4.0-0, Build 6011, S/N baec64
aac0: Supported 
Options=1f7e<CLUSTERS,WCACHE,DATA64,HOSTTIME,RAID50,WINDOW4GB,SOFTERR,NORECOND,SGMAP64,ALARM,NONDASD>
aacd0: <RAID 1 (Mirror)> on aac0
aacd0: 69998MB (143357184 sectors)

db2: (Adaptec RAID10 4 x 36G 15,000 RPM SCSI)
FreeBSD 4.8-STABLE
CPU: Intel(R) Xeon(TM) CPU 3.06GHz (3065.81-MHz 686-class CPU) [DUAL]
aac0: <Adaptec SCSI RAID 2120S> mem 0xf8000000-0xfbffffff irq 18 at 
device 2.0 on pci5
aac0: i960RX 100MHz, 48MB cache memory, optional battery present
aac0: Kernel 4.0-0, Build 6008, S/N b97ce8
aac0: Supported 
Options=1f7e<CLUSTERS,WCACHE,DATA64,HOSTTIME,RAID50,WINDOW4GB,SOFTERR,NORECOND,SGMAP64,ALARM,NONDASD>
aacp0: <SCSI Passthrough Bus> on aac0

The raw data can be viewed at
http://dannyman.toldme.com/scratch/benchmarks/

ANALYSIS

Unfortunately, my hardware RAIDs are on FreeBSD 4, and gmirror is on 5.
My hardware RAIDs are on dual CPU systems, with 2G RAM, and my gmirror
is on a single hyperthreaded CPU with 512M.  Yes, sorry, not especially
scientific.  Maybe the changes in FreeBSD make a big difference?  Maybe
RAM makes a big difference?

The first results show a serious advantage for the gmirror setup:

Sequential output (char)
  gmirror ATA RAID1: avg   320K/s
 Adaptec SCSI RAID1: avg   222K/s
Adaptec SCSI RAID10: avg   202K/s

Sequential input (char)
  gmirror ATA RAID1: avg   617K/s
 Adaptec SCSI RAID1: avg   345K/s
Adaptec SCSI RAID10: avg   336K/s

Sequential Output (block)
  gmirror ATA RAID1: avg 37893K/s
 Adaptec SCSI RAID1: avg 13829K/s
Adaptec SCSI RAID10: avg 40440K/s

The gmirror sees slightly poorer performance in random seeks:

Rndom Seeks
  gmirror ATA RAID1: avg  4144/s
 Adaptec SCSI RAID1: avg  5428/s
Adaptec SCSI RAID10: avg 13302/s

That all sounds great if I was streaming video, but I want to run a
database, opening and closing, reading, writing, and rewriting several
small files.  This is where things seem to go rotten.

We see the ATA performance go to heck on the File Create tests:

Sequential Create
    laptop 2.5" ATA: avg  101/s
  gmirror ATA RAID1: avg  365/s
 Adaptec SCSI RAID1: avg  160/s
Adaptec SCSI RAID10: avg  412/s

Sequential Read
    laptop 2.5" ATA: avg   76/s  # SLOWER than write!
  gmirror ATA RAID1: avg  251/s  # SLOWER than write!
 Adaptec SCSI RAID1: avg 7862/s
Adaptec SCSI RAID10: avg 7618/s

Random Create
    laptop 2.5" ATA: avg  124/s
  gmirror ATA RAID1: avg  354/s
 Adaptec SCSI RAID1: avg  155/s
Adaptec SCSI RAID10: avg  504/s

Random Read
    laptop 2.5" ATA: avg   57/s  # SLOWER than write!
  gmirror ATA RAID1: avg  144/s  # SLOWER than write!
 Adaptec SCSI RAID1: avg 7655/s
Adaptec SCSI RAID10: avg 7413/s

CONFUSION

Now, I could explain poor read performance by:
- Less RAM == Less buffer
- Bigger Disks == Slower Seeks
- Less CPU == ???

I DO have a 4.8-STABLE with a single IDE disk, no Soft Updates, and
faster read than write:

Version 1.93c       ------Sequential Create------ --------Random 
Create--------
anubis.xxxxxxxxxxxx -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
files:max            /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
      10:104884:0/5   183  32  1739  97   502  16   176  32  1624  94   
368  13
Latency               707ms   11487us   32824us     488ms     207ms     
117ms

However, seeing read SLOWER than write ... I have to wonder if something
fishy is going on.  Suggestions?  Ideas?  I'm fresh out, at the moment.
My suspicion is that something in 5.x is out-of-tune!?

Thanks a lot.

Sincerely,
-danny

-- 
http://dannyman.toldme.com/



More information about the freebsd-stable mailing list