kernel: mps0: Out of chain frames, consider increasing hw.mps.max_chains.

[Scott Long]

> On Mar 8, 2016, at 10:48 AM, Slawa Olhovchenkov <slw at zxy.spb.ru> wrote:
> 
> On Tue, Mar 08, 2016 at 10:34:23AM -0800, Scott Long wrote:
> 
>> 
>>> On Mar 8, 2016, at 10:07 AM, Slawa Olhovchenkov <slw at zxy.spb.ru> wrote:
>>> 
>>> On Mon, Mar 07, 2016 at 02:10:12PM +0300, Slawa Olhovchenkov wrote:
>>> 
>>>>>>>> This allocated one for all controllers, or allocated for every controller?
>>>>>>> 
>>>>>>> It’s per-controller.
>>>>>>> 
>>>>>>> I’ve thought about making the tuning be dynamic at runtime.  I
>>>>>>> implemented similar dynamic tuning for other drivers, but it seemed
>>>>>>> overly complex for low benefit.  Implementing it for this driver
>>>>>>> would be possible but require some significant code changes.
>>>>>> 
>>>>>> What cause of chain_free+io_cmds_active << max_chains?
>>>>>> One cmd can use many chains?
>>>>> 
>>>>> Yes.  A request uses and active command, and depending on the size of the I/O,
>>>>> it might use several chain frames.
>>> 
>>> I am play with max_chains and like significant cost of handling
>>> max_chains: with 8192 system resonded badly vs 2048. Now try 3192,
>>> response like with 2048.
>> 
>> Hi, I’m not sure I understand what you’re saying.  You said that you tried 8192, but the system still complained of being out of chain frames?  Now you are trying fewer, only 3192?
> 
> With 8192 system not complained of being out of chain frames, but like
> need more CPU power to handle this chain list -- traffic graf (this
> host servered HTTP by nginx) have many "jerking", with 3192 traffic
> graf is more smooth.

Hi,

The CPU overhead of doing more chain frames is nil.  They are just objects in a list, and processing the list is O(1), not O(n).  What you are likely seeing is other problems with VM and VFS-BIO system struggling to deal with the amount of I/O that you are doing.  Depending on what kind I/O you are doing (buffered filesystem reads/writes, memory mapped I/O, unbuffered I/O) there are limits and high/low water marks on how much I/O can be outstanding, and when the limits are reached processes are put to sleep and then race back in when they are woken up.  This causes poor, oscillating system behavior.  There’s some tuning you can do to increase the limits, but yes, it’s a problem that behaves poorly in an untuned system.

Scott