unsatisfying c++/boost::multi_index_container::erase performance on at least FreeBSD 6.0

[bert hubert]

Dear FreeBSD hackers,

I'm working on improving the PowerDNS recursor for a big FreeBSD-loving
internet provider in The Netherlands and I am hitting some snags. I also
hope this is the appropriate list to share my concerns.

Pruning the cache is very very slow on the providers FreeBSD 6.0 x86 systems
whereas it flies on other operating systems.

I've managed to boil down the problem to the code found on
http://ds9a.nl/tmp/cache-test.cc which can be compiled with:
'g++ -O3 -I/usr/local/include cache-test.cc -o cache-test' after installing
Boost from the ports.

The problem exists both with the system compiler and with a self-compiled
g++ 4.1.

Here are some typical timings:
$ ./cache-test
Creating..
Copying 499950 nodes
100                           345 usec            3.45 usec/erase
300                           3298 usec           10.99 usec/erase
500                           8749 usec           17.50 usec/erase
700                           72702 usec          103.86 usec/erase
900                           46521 usec          51.69 usec/erase

On another operating system with almost the same cpu:

$ ./cache-test
Creating..
Copying 499950 nodes
100                           62 usec             0.62 usec/erase
300                           187 usec            0.62 usec/erase
500                           347 usec            0.69 usec/erase
700                           419 usec            0.60 usec/erase
900                           575 usec            0.64 usec/erase

I've toyed with MALLOC_OPTIONS, especially the >> options, I've tried
GLIBCXX_FORCE_NEW, I've tried specifying a different STL allocator in the
c++ code, it all doesn't change a thing.

A quick gprof profile shows a tremendous number of calls to 'ifree' but that
may be due to the copying of the container going on between test runs.

Any help would be very appreciated as I am all out of clues. 

Thanks.

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://netherlabs.nl              Open and Closed source services