Puzzling performance

[Guy Helmer]

On a FreeBSD 7.1 SCHED_ULE kernel, I have a large number of files opened and
mmapped (with MAP_NOSYNC option) for shared-memory communication between
processes.  Normally, memcpy() copies data into these shared-memory buffers
in a reasonable amount of time closely related to the size of the copy
(roughly 10us per 10KB).  However, due to performance issues I've found that
sometimes a memcpy() takes an abnormally long time (10ms for 40KB, and I
suspect longer times occurring when I have not had monitoring enabled).  The
system doesn't seem to be in memory overcommit -- there is just a minor
amount of swap in use, and I've not seen page-ins or page-outs while
watching systat or vmstat.

Since I'm using MAP_NOSYNC, I would not expect the pager to flush dirty
pages to disk and cause add delays.  Any ideas where to look?  Might it help
to pin threads to CPUs in case a thread is getting moved to a different
core?

Thanks,
Guy