2 x quad-core system is slower that 2 x dual core on FreeBSD

[Kris Kennaway]

Alexey Popov wrote:
> Hi
> 
> Kris Kennaway wrote:
>>>>> Now FreeBSD 7-STABLE ULE 8-core server without optimized PHP 
>>>>> realpath_cache_size (producing 2000+ lstats per request) can handle 
>>>>> up to ~24 rps as opposed to  max. 17 rps without your patch. %sys 
>>>>> never grows over %user with your patch. On the server with 
>>>>> optimized realpath_cache_size there's no visible influence of your 
>>>>> patch.
>>>> You said "20" before for this configuration, so I'm a bit suspicious 
>>>> about how seriously to treat your measurements :)
>>> Sorry, my mistake. s/ULE/4BSD.
>> OK, please compare ULE to ULE with and without my patch (and 
>> remembering to enable the sysctl), and obtain lock profiling traces in 
>> both cases under identical workloads & durations.  That is what I need 
>> to proceed with this issue.
> I didn't measured the exact values of requests per second on ULE with 
> patch and without patch, but at first glance the benefits of the patch 
> are similiar to 4BSD. If you need this values, I'll obtain them.
> 
> Here you can find lock profiling results for 7-BETA3 GENERIC kernel with 
> SCHED_ULE running optimized PHP and unoptimized, with your patch and 
> without it: http://83.167.98.162/gprof/lockmgr/
> 
> This data was collected by th following script:
> (sysctl debug.lock.prof.reset=1
> sysctl debug.lock.prof.enable=1
> sleep 60
> sysctl debug.lock.prof.enable=0
> sysctl debug.lock.prof.stats
> top -d 2 -b | tail -25)
> 
> AFAIU there's still high contention on "lockbuilder mtxpool" with patch 
> applied. But hopefully "lockmgr:ufs" contention which i believe produced 
> 80%sysCPU load is gone with your patch.

Looks to me like lockmgr-related contention was reduced by 1 to 2 orders 
of magnitude, which is the expected result.  This surely must have a 
measurable impact on your workload.  Further lockmgr improvement will 
have to wait until the lockmgr replacement work proceeds.

One more patch which may or may not help is:

   http://www.freebsd.org/~jhb/patches/namei_rwlock.patch

(may also require porting since it was against an older version of 
7.0-CURRENT).  When I have tested this in the past it was a performance 
loss for reasons that I think I understand (basically, it is locally a 
performance improvement for the name cache but also requires a fixed 
lockmgr to avoid an overall performance loss), but I don't remember if I 
tested it in conjunction with the lockmgr patch.

>> There are patches you need to enable it on woodcrest.  They are in my 
>> p4 branch (kris-contention) but I don't have time right now to extract 
>> them.
> I think it would be very useful because I can't see any other ways to 
> profile FreeBSD on the modern many-cores machines.

You can extract the changeset from my branch via 
http://perforce.freebsd.org.  Unfortunately I don't have time to do it 
myself.

Kris