Context Switching Oddities

[Peter Jeremy]

I originally asked a variant of this in -sparc64 since the oddities
initially seemed to be related to sparc64 but it was suggested that
this might be a more appropriate list.  I've also added more results.

I have a tool that measures the rate at which a single-byte token can
be passed between two processes via a socketpair - basically counting
	while (1) {
		read(fd[0], buf, 1);
		write(fd[1], "a", 1);
	}
I was initially running it multiple copies of it on an otherwise idle
V890 (16-CPU, 64GB RAM running -current from about a week ago),
capturing 'vmstat -s' output.  In the process, I have found several
oddities.  That prompted me to additionally test it on a V440 (4-CPU,
8GB RAM, same world as the V890), a SB1500 (1 CPU, 4GB RAM, -current
from a few days ago) and an Athlon-64 (dual-core, 8GB RAM, -stable
amd64 from a few months ago).  The SPARC systems are all using 4BSD
and the amd64 is using ULE.  I don't have access to any large x86
boxes to cross-check.

1) The number of context switches doesn't match my expectations.
See http://i.imgur.com/28OHu.jpg (1 CPU)
    http://i.imgur.com/6YRh8.jpg (2 core)
    http://i.imgur.com/r0v7M.jpg (4 CPU)
    http://i.imgur.com/hkCA2.jpg (16 CPU)
    http://i.imgur.com/9Tt9Q.gif (combined)
Based on one process writing a token to a second process requiring one
context switch, I would expect the number of context switches to
roughly match the green (based on token passing rate) or blue (based
on syscall rate) lines.  Instead, it's generally far too low, though
the 4- and 16-CPU graphs start out unexpectedly high.

2) The transfer rate doesn't gradually tail off
See http://i.imgur.com/YoDQ5.jpg (1 CPU)
    http://i.imgur.com/0Wl1Y.jpg (2 core)
    http://i.imgur.com/699zr.jpg (4 CPU)
    http://i.imgur.com/0ujRN.jpg (16 CPU)
    http://i.imgur.com/omxG1.gif (combined & scaled)
I would expect a fairly flat peak from 1 to about n-CPU pairs (since
there are that many execution threads available) that then tailed off
as scheduler overheads increased.  Instead the 4- & 16-CPU tests show
a dip initially then rising to a peak before tailing off.  Each graph
shows both the rate reported by the program and the rate estimated
from the syscall rate.

Can anyone offer an explanation for this behaviour?

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-arch/attachments/20110624/c856fd68/attachment.pgp