Increasing MAXPHYS

[Scott Long]

On Mar 20, 2010, at 1:26 PM, Alexander Motin wrote:

> Scott Long wrote:
>> On Mar 20, 2010, at 11:53 AM, Matthew Dillon wrote:
>>>   Diminishing returns get hit pretty quickly with larger MAXPHYS values.
>>>   As long as the I/O can be pipelined the reduced transaction rate
>>>   becomes less interesting when the transaction rate is less than a
>>>   certain level.  Off the cuff I'd say 2000 tps is a good basis for
>>>   considering whether it is an issue or not.  256K is actually quite
>>>   a reasonable value.  Even 128K is reasonable.
>> 
>> I agree completely.  I did quite a bit of testing on this in 2008 and 2009.
>> I even added some hooks into CAM to support this, and I thought that I had
>> discussed this extensively with Alexander at the time.  Guess it was yet another
>> wasted conversation with him =-(  I'll repeat it here for the record.
> 
> AFAIR at that time you've agreed that 256K gives improvements, and 64K
> of DFLTPHYS limiting most SCSI SIMs is too small. That's why you've
> implemented that hooks in CAM. I have not forgot that conversation (pity
> that it quietly died for SCSI SIMs). I agree that too high value could
> be just a waste of resources. As you may see I haven't blindly committed
> it, but asked public opinion. If you think 256K is OK - let it be 256K.
> If you think that 256K needed only for media servers - OK, but lets make
> it usable there.
> 

I think that somewhere in the range of 128-512k is appropriate for a given platform.
Maybe big-iron gets 512k and notebooks and embedded systems get 128k?  It's
partially a platform architecture issue, and partially a platform application issue.
Ultimately, it should be possible to have up to 1M, and maybe even more.  I don't
know how best to make that selectable, or whether it should just be the default.

>> Besides the nswbuf sizing problem, there is a real problem that a lot of drivers
>> have incorrectly assumed over the years that MAXPHYS and DFLTPHYS are
>> particular values, and they've sized their data structures accordingly.  Before
>> these values are changed, an audit needs to be done OF EVERY SINGLE
>> STORAGE DRIVER.  No exceptions.  This isn't a case of changing MAXHYS
>> in the ata driver, testing that your machine boots, and then committing the change
>> to source control.  Some drivers will have non-obvious restrictions based on
>> the number of SG elements allowed in a particular command format.  MPT
>> comes to mind (its multi message SG code seems to be broken when I tried
>> testing large MAXPHYS on it), but I bet that there are others.
> 
> As you should remember, we have made it in such way, that all unchecked
> drivers keep using DFLTPHYS, which is not going to be changed ever. So
> there is no problem. I would more worry about non-CAM storages and above
> stuff, like some rare GEOM classes.

And that's why I say that everything needs to be audited.  Are there CAM drivers
that default to being silent on cpi->maxio, but still look at DFLTPHYS and MAXPHYS?
Are there non-CAM drivers that look at MAXPHYS, or that silently assume that
MAXPHYS will never be more than 128k?

Scott