(LONG) ATA Benchmark: 5.x Reads Slower than Writes

Fri Apr 8 14:34:03 PDT 2005

On April 8, 2005 04:41 pm, Danny Howard wrote:
> Hello,
>
> BACKGROUND
>
> I need to purchase a new system for our developers, for use as a
> Postgres database test server.  Having a RAID, probably RAID1, is
> desirable for performance and reliability.  I have recently set up a
> system with a gmirror-based software RAID1 on a pair of 250GB ATA
> drives.  I would like to stick with gmirror, because:
>
> - We save money on extra hardware.
> - Since gmirror is part of FreeBSD, maintenance is a lot easier than
>   with a hardware solution.
>
> SUMMARY
>
> But before I go balls out, I should see how well it compares to hardware
> RAID.  So, I do some benchmarks with bonnie++.  Since this simulates
> create, write and read on thousands of random files, this sounds like a
> good approximation of what Postgres does. :)
>
> I don't have the time and hardware to do very scientific tests, but I
> have been able to run a series of benchmarks using bonnie++ on some
> systems I have available to me.  The ATA-based gmirror performs
> extremely well, compared to a few Adaptec RAIDs that we have, EXCEPT
> that the sequential and random reads are MUCH SLOWER than the hardware
> solution, and even *slower than the preceding write operations*.  This
> is counter-intuitive, especially since RAID1 implies slowed writes and
> faster reads.  I tried the benchmark on my workstation (single 2.5" IDE
> in a laptop) and got comparable write-faster-than-read results.

May I suggest that you turn softupdates off and sync on for the filesystems 
you are testing. If you don't you are not really testing the hardware. You 
are testing the FreeBSD disk caching system in the kernel.

> DATA
>
> I was able to make use of the following test systems.  I ran tests in
> multi-user, but tried to favor times when there wasn't much background
> activity:
>
> mito: (lone 2.5" ATA)
> 5.4-PRERELEASE
> CPU: Intel(R) Pentium(R) M processor 1.50GHz (1495.16-MHz 686-class CPU)
> atapci0: <Intel ICH4 UDMA100 controller>
> ata0: channel #0 on atapci0
> ata1: channel #1 on atapci0
> ad0: 28615MB <TOSHIBA MK3021GAS/GA129D> [58140/16/63] at ata0-master
> UDMA100
>
> amun: (gmirror RAID1 2 x WD250GB High Intensity)
> FreeBSD 5.3-RELEASE
> CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2799.22-MHz 686-class CPU)
> atapci0: <SiI 3112 SATA150 controller>
> ata2: channel #0 on atapci0
> ata3: channel #1 on atapci0
> ad4: 238475MB <WDC WD2500SD-01KCB0/08.02D08> [484521/16/63] at
> ata2-master SATA150
> ad6: 238475MB <WDC WD2500SD-01KCB0/08.02D08> [484521/16/63] at
> ata3-master SATA150
> atapci1: <SiI 3112 SATA150 controller>
> acd0: CDROM <CD-224E/1.9A> at ata1-master UDMA33
>
> janus: (Adaptec RAID1 2 x 72G 10,000 RPM SCSI)
> FreeBSD 4.10-STABLE
> CPU: Intel(R) Xeon(TM) CPU 2.40GHz (2399.33-MHz 686-class CPU) [DUAL]
> aac0: <Adaptec SCSI RAID 2120S> mem 0xf8000000-0xfbffffff irq 16 at
> device 1.0 on pci2
> aac0: i960RX 100MHz, 48MB cache memory, optional battery present
> aac0: Kernel 4.0-0, Build 6011, S/N baec64
> aac0: Supported
> Options=1f7e<CLUSTERS,WCACHE,DATA64,HOSTTIME,RAID50,WINDOW4GB,SOFTERR,NOREC
>OND,SGMAP64,ALARM,NONDASD> aacd0: <RAID 1 (Mirror)> on aac0
> aacd0: 69998MB (143357184 sectors)
>
> db2: (Adaptec RAID10 4 x 36G 15,000 RPM SCSI)
> FreeBSD 4.8-STABLE
> CPU: Intel(R) Xeon(TM) CPU 3.06GHz (3065.81-MHz 686-class CPU) [DUAL]
> aac0: <Adaptec SCSI RAID 2120S> mem 0xf8000000-0xfbffffff irq 18 at
> device 2.0 on pci5
> aac0: i960RX 100MHz, 48MB cache memory, optional battery present
> aac0: Kernel 4.0-0, Build 6008, S/N b97ce8
> aac0: Supported
> Options=1f7e<CLUSTERS,WCACHE,DATA64,HOSTTIME,RAID50,WINDOW4GB,SOFTERR,NOREC
>OND,SGMAP64,ALARM,NONDASD> aacp0: <SCSI Passthrough Bus> on aac0
>
> The raw data can be viewed at
> http://dannyman.toldme.com/scratch/benchmarks/
>
> ANALYSIS
>
> Unfortunately, my hardware RAIDs are on FreeBSD 4, and gmirror is on 5.
> My hardware RAIDs are on dual CPU systems, with 2G RAM, and my gmirror
> is on a single hyperthreaded CPU with 512M.  Yes, sorry, not especially
> scientific.  Maybe the changes in FreeBSD make a big difference?  Maybe
> RAM makes a big difference?
>
> The first results show a serious advantage for the gmirror setup:
>
> Sequential output (char)
>   gmirror ATA RAID1: avg   320K/s
>  Adaptec SCSI RAID1: avg   222K/s
> Adaptec SCSI RAID10: avg   202K/s
>
> Sequential input (char)
>   gmirror ATA RAID1: avg   617K/s
>  Adaptec SCSI RAID1: avg   345K/s
> Adaptec SCSI RAID10: avg   336K/s
>
> Sequential Output (block)
>   gmirror ATA RAID1: avg 37893K/s
>  Adaptec SCSI RAID1: avg 13829K/s
> Adaptec SCSI RAID10: avg 40440K/s
>
> The gmirror sees slightly poorer performance in random seeks:
>
> Rndom Seeks
>   gmirror ATA RAID1: avg  4144/s
>  Adaptec SCSI RAID1: avg  5428/s
> Adaptec SCSI RAID10: avg 13302/s
>
> That all sounds great if I was streaming video, but I want to run a
> database, opening and closing, reading, writing, and rewriting several
> small files.  This is where things seem to go rotten.
>
> We see the ATA performance go to heck on the File Create tests:
>
> Sequential Create
>     laptop 2.5" ATA: avg  101/s
>   gmirror ATA RAID1: avg  365/s
>  Adaptec SCSI RAID1: avg  160/s
> Adaptec SCSI RAID10: avg  412/s
>
> Sequential Read
>     laptop 2.5" ATA: avg   76/s  # SLOWER than write!
>   gmirror ATA RAID1: avg  251/s  # SLOWER than write!
>  Adaptec SCSI RAID1: avg 7862/s
> Adaptec SCSI RAID10: avg 7618/s
>
> Random Create
>     laptop 2.5" ATA: avg  124/s
>   gmirror ATA RAID1: avg  354/s
>  Adaptec SCSI RAID1: avg  155/s
> Adaptec SCSI RAID10: avg  504/s
>
> Random Read
>     laptop 2.5" ATA: avg   57/s  # SLOWER than write!
>   gmirror ATA RAID1: avg  144/s  # SLOWER than write!
>  Adaptec SCSI RAID1: avg 7655/s
> Adaptec SCSI RAID10: avg 7413/s
>
> CONFUSION
>
> Now, I could explain poor read performance by:
> - Less RAM == Less buffer
> - Bigger Disks == Slower Seeks
> - Less CPU == ???
>
> I DO have a 4.8-STABLE with a single IDE disk, no Soft Updates, and
> faster read than write:
>
> Version 1.93c       ------Sequential Create------ --------Random
> Create--------
> anubis.xxxxxxxxxxxx -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
> files:max            /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
> /sec %CP
>       10:104884:0/5   183  32  1739  97   502  16   176  32  1624  94
> 368  13
> Latency               707ms   11487us   32824us     488ms     207ms
> 117ms
>
> However, seeing read SLOWER than write ... I have to wonder if something
> fishy is going on.  Suggestions?  Ideas?  I'm fresh out, at the moment.
> My suspicion is that something in 5.x is out-of-tune!?
>
> Thanks a lot.
>
> Sincerely,
> -danny

-- 
Ean Kingston

E-Mail: ean AT hedron DOT org
URL: http://www.hedron.org/