4BSD process starvation during I/O
Sam Lawrance
boris at brooknet.com.au
Thu Nov 24 00:54:16 GMT 2005
On 24/11/2005, at 7:18 AM, Kris Kennaway wrote:
> I have noticed that when multiple identical processes (e.g. gtar, or
> dd) are run on 4BSD, when there are N CPUs on a machine there will be
> N processes that run with a higher CPU share than all the others. As
> a result, these N processes finish first, then another N, and so on.
>
> This is true under both 4.11 and 6.0 (so in that sense it's not so
> surprising), but the effect is much more pronounced on 6.0 (which may
> be possible to fix).
>
> Here are the exit times for 6 identical gtar processes (and same 4.11
> gtar binary) started together on a 2-CPU machine:
>
> 6.0:
>
> 1132776233
> 1132776235
> 1132776264
> 1132776265
> 1132776279
> 1132776279
> 238.86 real 10.87 user 166.00 sys
>
> You can see they finish in pairs, and there's a spread of 46 seconds
> from first to last.
>
> On 4.11:
>
> 1132775426
> 1132775429
> 1132775431
> 1132775432
> 1132775448
> 1132775449
> 275.56 real 0.43 user 336.26 sys
>
> They also finish in pairs, but the spread is half, at 23 seconds.
>
> This seems to be correlated to the rate at which the processes perform
> I/O. On a quad amd64 machine running 6.0 when I run multiple dd
> processes at different offsets in a md device:
>
> 268435456 bytes transferred in 1.734285 secs (154781618 bytes/sec)
> 268435456 bytes transferred in 1.737857 secs (154463501 bytes/sec)
> 268435456 bytes transferred in 1.751760 secs (153237575 bytes/sec)
> 268435456 bytes transferred in 3.263460 secs (82254865 bytes/sec)
> 268435456 bytes transferred in 3.295294 secs (81460244 bytes/sec)
> 268435456 bytes transferred in 3.349770 secs (80135487 bytes/sec)
> 268435456 bytes transferred in 4.716637 secs (56912467 bytes/sec)
> 268435456 bytes transferred in 4.850927 secs (55336941 bytes/sec)
> 268435456 bytes transferred in 4.953528 secs (54190760 bytes/sec)
>
> They finish in groups of 3 here since the 4th CPU is being used to
> drive the md worker thread (which takes up most of the CPU). In this
> case the first 3 dd processes get essentially 100% of the CPU, and the
> rest get close to 0% until those 3 processes finish.
>
> Perhaps this can be tweaked.
>
I tried this on a dual Xeon, with 12 processes like
mdconfig -a -t swap -s 320m
dd if=/dev/md0 of=1 bs=1m skip=0 count=40 &
dd if=/dev/md0 of=2 bs=1m skip=40 count=40 &
...
and all processes finished within a frame of about 1 second. No
interesting difference with HTT on or off. Generic SMP RELENG_6 from
8 November.
Is your dd test different somehow?
More information about the freebsd-current
mailing list