2 x quad-core system is slower that 2 x dual core on FreeBSD

[Kris Kennaway]

Alexey Popov wrote:
> Hi
> 
> Kris Kennaway wrote:
>>>>>>> CPU states:  9.5% user,  0.0% nice, 82.0% system,  0.5% 
>>>>>>> interrupt,  8.0% idle
>>>>>> A wild idea that might not help: try reducing kern.hz in 
>>>>>> loader.conf to
>>>>>> something like 100 and see if something significant changes.
>>> Usually on PHP backends slow PHP code eats most of the CPU time. I 
>>> have %user much bigger than %system in CPU states.
>>> But now %system is much bigger than %user and I can conclude that on 
>>> 8-core server FreeBSD consumes more CPU time than PHP.
>> That is one possibility, but you still need to look at the actual 
>> throughput on these machines before making conclusions about which is 
>> performing better.  Can you please provide those numbers for 6.x, 7.x 
>> with ULE and 4BSD on the 4-core and 8-core systems?
> Ok, here's results of practical research. The following is approximate 
> maximum qps that backends can survive with my workload:
> 
> 7-STABLE quad ULE    20
> 7-STABLE quad 4BSD    17
> 6-STABLE quad        14
> 6-STABLE dual        21
> Linux CentOS 5 quad    >50

OK, so 7.x is an improvement compared to 6.x on the 8 core machine, and 
ULE is an improvement over 4BSD.  This much is in line with expectations.

Neither shows an improvement vs 4 cores.  It is hard to say for certain 
without a direct profile comparison of the workload, but it is probably 
due to lockmgr contention.  lockmgr is used for various locking 
operations to do with VFS data structures.  It is known to have poor 
performance and scale very badly.  It is interesting that you are 
running into this on a real workload though, so far I have only 
encountered it as a limiting factor in synthetic microbenchmarks.

There was some work done over the summer on replacing lockmgr with 
something reasonable, but unfortunately it is not yet ready for testing. 
    I am CC'ing the developer who was working on that (Attilio Rao). 
Depending on his availability it will probably be at least a couple of 
months before it is ready though.

In the meantime there is unfortunately not a lot that can be done, 
AFAICT.  There is one hack that I will send you later but it is not 
likely to help much.  I will also think about how to track down the 
cause of the contention further (the profiling trace only shows that it 
comes mostly from vget/vput but doesn't show where these are called from).

Kris






> 
> With best regards,
> Alexey Popov
> 
>