Inconsistent IO performance

Kevin Oberman oberman at es.net
Sat Aug 14 02:12:37 UTC 2010


> Date: Fri, 13 Aug 2010 23:32:05 +0200
> From: Roland Smith <rsmith at xs4all.nl>
> 
> On Fri, Aug 13, 2010 at 09:01:09AM -0700, Kevin Oberman wrote:
> > For some time I have seen very odd issues with IO performance on
> > 8-Stable. Going back to November of last year when 8.0 was released, I
> > see variations of up to 22% in identical operations. This is not a
> > degradation as the performance moves up and down.
> > 
> > This is a very simplistic case. I have two identical disks (Fujitsu 80G)
> > on a ThinkPad T43 with a 2 GHz CPU and 2G RAM. I run the command:
> > dd bs=516096 if=/dev/ad0 of=/dev/ad2
> 
> Why are you using this peculiar block size?

It was suggested by a co-worker with a similar system, but I have tried
larger values with no clear difference. It is the size of one cylinder
based on disk geometry. I was (and remain) sceptical since CHS has
nothing to do with real geometry in modern disks.

> > Note the dramatic differences even on the same kernel. For the December
> > 6 kernel, for example, I see a maximum of 23,676,086 and a minimum of
> > just 18,304,565. ????
> 
> Both figures seem quite low to me? I cannot exactly reproduce your test,
> because I don't have an empty second disk handy, but doing
> 
>     dd if=/dev/zero bs=1m count=100 of=/tmp/foo
> 
> yields the following writing speed on 8.1-RELEASE amd64, 
> WDC WD5001ABYS SATA harddisk @ 7200 rpm.:
> 
> 1) 87263174 bytes/sec
> 2) 87878728 bytes/sec
> 3) 86397125 bytes/sec
> 4) 86550094 bytes/sec
> 5) 86524741 bytes/sec
> 
> Th maximum variation in write speed is (87878728-86397125)/86397125*100% =
> 1.7%, which doesn't seem that much to me.
> 
> This is in multi-user, with X11 running but on an otherwise idling machine,
> and with filesystem overhead to boot. Still the numbers are a lot higher than
> yours, which puzzles me. 
> 
> Trying only reading does yield very inconsistent results because of caching, I
> think;
> 
>     dd if=/tmp/foo bs=1m count=100 of=/dev/null
> 
> 1) 1454216957 bytes/sec
> 2) 1003691691 bytes/sec
> 3) 1429956761 bytes/sec
> 4) 2324794646 bytes/sec
> 5) 1804563681 bytes/sec
> 
> This is a (2324794646-1003691691)/1003691691*100% = 132% difference. OTOH,
> your data set should be big enough to negate caching effects, I guess. :-)
> 
> What this does show is that writing seems to be the bottleneck.
> 
> If I both read from and write to a file (on the same disk & partition);
> 
>     dd if=/tmp/foo bs=1m count=100 of=/tmp/bar
> 
> gives
> 
> 1) 85161534 bytes/sec
> 2) 84978770 bytes/sec
> 3) 87966613 bytes/sec
> 4) 83036312 bytes/sec
> 5) 86536879 bytes/sec
> 
> This is a (87966613-83036312)/83036312*100% = 5.9% difference between largest
> and smallest. The speed seems to be dictated by the writing.
> 
> > Can anyone explain what might be causing such a dramatic difference?
> 
> Maybe there is a hardware component here? Are both disks on the same
> controller? Or if not are both controllers using the same interrupt line?

No. Each is on its one controller and is the only disk on that
controller. 

> You should have a look at 'systat -vmstat' with dd running in the
> background. That might give a clue as to where the bottleneck is.

I guess I can give this a try.

Thanks!
-- 
R. Kevin Oberman, Network Engineer
Energy Sciences Network (ESnet)
Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab)
E-mail: oberman at es.net			Phone: +1 510 486-8634
Key fingerprint:059B 2DDF 031C 9BA3 14A4  EADA 927D EBB3 987B 3751


More information about the freebsd-stable mailing list