svn commit: r196777 - head/sys/dev/ahci

[Scott Long]

On Thu, 3 Sep 2009, Ivan Voras wrote:
> 2009/9/3 Scott Long <scottl at samsco.org>:
>> On Thu, 3 Sep 2009, Alexander Motin wrote:
>
>>> It would be nice if every level would do it's own job.
>>
>> It's the job of the driver to handle the limitations of the hardware, yes.
>> Again, if you want to experiment with pushing this functionality into GEOM,
>> be my guest.  But until then, consider following my advice.
>
> Speaking as an user who goes "huh, well" every time he sees a RAID
> card with a GB of cache talking to the OS in 64 kB chunks, eventually
> removing this limitation seems a Nice Thing to Have.
>
> I don't know how things look at the driver side, but GEOM by itself
> has no problems passing around requests of at least "long" in length.
> Specifically, it cares about struct bio, where bio_bcount is a "long"
> and bio_length and bio_completed are off_t. So, ssize_t looks ok as a
> high boundary.
>
> There was a time (apparently much of it was a bug in reporting and is
> now fixed :( ) when MAXPHYS could be manually redefined to be 256K or
> more and iostat would nicely state the higher value. I think the
> concern raised at the topic was that it doesn't play nice with
> bufcache, and I think the specific problem was possible out-of-memory
> situations. Now that kernel limits on AMD64 are much increased (to
> values not longer fitting in uint32_t) I wonder if the problem is so
> serious?
>

The problem is lack of kernel address space, not lack of RAM, but that's 
just semantics in this discussion.  I've tested with increasing MAXPHYS in 
increments to 1M.  Performance increases logrithmically, and effectively 
hits a max at 512K for the variety of controllers that I tested.  The gain 
from 64K to 128K is huge, the gain from 128K to 256K is ok, the gain from 
256k to 512k is measurable but less significant, and the gain from 512k to 
1m is almost not measurable.

I have simple patches to increase MAXPHYS.  The introduction of the the 
maxio paramter in the CAM SIM interface is there in preparation for this. 
However, a _LOT_LOT_LOT_ of drivers in the tree falsely assume that 
MAXPHYS and DEFLTPHYS are 128k and 64k respectively, and size their data 
structures accordingly.  Changing these values will cause the drivers to 
fail in bad ways.  So an audit needs to be done.  Also, MAXPHYS is abused 
by the swapper in the struct-buf, so that needs to be reviewed as well.

Even though kernel address space is less restricted on 64bit platforms, 
it's still not free and limitless.  Large I/O's requires more work in the 
VM to assign address space, and in turn causes more lock contention.  I 
haven't done any practical measurements of this on common workloads, but I 
can anecdotally say that I see increased lock contention from it in 
locking profiles.  If FreeBSD wants to seriously increase MAXPHYS, this 
needs to be looked at and either proven to not be important, or fixed.

Scott