Very low disk performance on 5.x

[Bruce Evans]

On Sun, 8 May 2005, Scott Long wrote:

> Changing MAXPHYS is very dangerous, unfortunately.  The root of the
> problem is that kernel virtual memory (KVA) gets assigned to each I/O
> buffer as it passes through the kernel. If we allow too much I/O through
> at once then we have the very real possibility of exhausting the kernel
> address space and causing a deadlock and/or panic.  That is why MAXPHYS
> is set so low.  Your DD test is unlikely to trigger a problem, but try
> doing a bunch of DD's is parallel and you likely will.
>
> The solution is to re-engineer the way that I/O buffers pass through
> the kernel and only assign KVA when needed (for doing software parity
> calculations, for example).  That way we could make MAXPHYS be any
> arbitrarily large number and not worry about exhausting KVA.  I believe
> that there is some work in progress in this area, but it's a large
> project since nearly every single storage driver would need to be
> changed.  Another possibility is to recognise that amd64 doesn't have
> the same KVA restrictions as i386 and thus can be treated differently.
> However, doing the KVA work is still attractive since it'll yeild some
> performance benefits too.

How would it yield anything except complexity and negative performance
benefits on machines that don't need it?  The most efficient way to map
kva for disk buffers is to do it only once at boot time using a simple
sparse mapping technique as in 4.4BSD-Lite1.  Simple sparse mapping needs
lots of kva, and i386's don't have enough for machines with more than
about 16MB of RAM, but machines with a 64-bit address space can have
2^32 times as much kva as i386's.

Bruce