Benchmarking 5.4

[Niall Douglas]

As promised, here is the writeup of the benchmarking of my code on 
various systems:

http://www.nedprod.com/Tn/

The code is heavily multithreaded, with some eleven to fourteen 
threads running most of these experiencing some load. It represents a 
good test of the scheduler and thread performance. It uses its own 
mutex implementation as to be honest, my own is far faster than any 
other I've found. This could change if FreeBSD made full use of the 
M:N scope for optimisation. The code uses a mixture of process scope 
and system scope threads but mostly process scope as that's the 
default on FreeBSD builds on my library. The kernel was a GENERIC 
amd64 build but only with SCHED_ULE changed (so no witness, invariant 
etc.).

The good news is that performance now matches Linux for the first 
time having previously been a fraction. The bad news is that Windows 
is still whooping both Linux and FreeBSD.

There is a curious thing on FreeBSD - during the memory mapped i/o 
test, the percent complete goes from 0%-60% very quickly indeed but 
then hangs at 60% for what seems like ages before completing at a 
slower rate than before. What's happening internally is that a 4.47Mb 
file is being mapped in in 1Mb segments and memcpy()'ed elsewhere. I 
initially thought it might be the 4BSD scheduler so I tried ULE but 
it made no difference (to the benchmarks or anything). The test file 
was on UFS with soft updates.

Even in the packet based transfers, the percent complete noticeably 
goes faster from 0%-60% and then halts momentarily before continuing 
at a slower pace though it's nothing like as bad as memory mapped. Of 
course internally the file is still being accessed as memory mapped 
ie; the server still maps the file, just it reads and packetises it 
for the client. If I watch the disk drive light, it does flicker 
briefly at the end of the test but otherwise remained unlit - surely 
under UFS the file would still be in the cache?

If that memory mapped i/o thing were sorted out, I'm quite sure that 
FreeBSD would at least reach half the Windows speeds. Any ideas? Is 
there some kernel sysctl for setting read ahead buffering or 
something? 60% of 4.47Mb is around 2.6Mb.

Cheers,
Niall