Increasing MAXPHYS

[Ivan Voras]

Julian Elischer wrote:
> Ivan Voras wrote:
>> Julian Elischer wrote:
>>> Alexander Motin wrote:
>>>> Scott Long wrote:
>>>>> On Mar 20, 2010, at 11:53 AM, Matthew Dillon wrote:
>>>>>>    Diminishing returns get hit pretty quickly with larger MAXPHYS 
>>>>>> values.
>>>>>>    As long as the I/O can be pipelined the reduced transaction rate
>>>>>>    becomes less interesting when the transaction rate is less than a
>>>>>>    certain level.  Off the cuff I'd say 2000 tps is a good basis for
>>>>>>    considering whether it is an issue or not.  256K is actually quite
>>>>>>    a reasonable value.  Even 128K is reasonable.
>>>>> I agree completely.  I did quite a bit of testing on this in 2008 
>>>>> and 2009.
>>>>> I even added some hooks into CAM to support this, and I thought 
>>>>> that I had
>>>>> discussed this extensively with Alexander at the time.  Guess it 
>>>>> was yet another
>>>>> wasted conversation with him =-(  I'll repeat it here for the record.
>>>
>>> In the Fusion-io driver we find that the limiting factor is not the
>>> size of MAXPHYS, but the fact that we can not push more than
>>> 170k tps through geom. (in my test machine. I've seen more on some
>>> beefier machines), but that is only a limit on small transacrtions,
>>
>> Do the GEOM threads (g_up, g_down) go into saturation? Effectively all 
>> IO is serialized through them.
> 
> basically..
> 
> You can get better throughput by using TSC for timing because the geom 
> and devstat code does a bit of timing.. Geom can be told to turn off
> it's timing but devstat can't. The 170 ktps is with TSC as timer,
> and geom timing turned off.

I see. I just ran randomio on a gzero device and with 10 userland 
threads (this is a slow 2xquad machine) I get g_up and g_down saturated 
fast with ~~ 120 ktps. Randomio uses gettimeofday() for measurements.

Hmm, it looks like it could be easy to spawn more g_* threads (and, 
barring specific class behaviour, it has a fair chance of working out of 
the box) but the incoming queue will need to also be broken up for 
greater effect.