Partial cacheline flush problems on ARM and MIPS

[Mark Tinguely]

On Mon, Aug 27, 2012 at 2:31 PM, John Baldwin <jhb at freebsd.org> wrote:
> On Monday, August 27, 2012 2:53:46 pm Konstantin Belousov wrote:
>> On Sun, Aug 26, 2012 at 05:13:31PM -0600, Warner Losh wrote:
>> >
>> > On Aug 26, 2012, at 12:25 PM, Ian Lepore wrote:
>> > > In this regard, it's the busdma implementation that's broken, because it
>> > > should bounce those IOs through a DMA-safe buffer.  There's absolutely
>> > > no rule that I've ever heard of in FreeBSD that says IO can only take
>> > > place using memory allocated from busdma.
>> >
>> > That's partially true. Since BUSDMA grew up in the storage area, you
>> > must allocate the memory from busdma, or it must be page aligned has
>> > been the de-facto rule here. The mbuf and uio variants of load were
>> > invented to cope with common cases of mbufs and user I/O to properly
>> > flag things.
>>
>> I once looked at x86 bus_dmamap_load_uio(), and I was unable to
>> understand how to use it with usermode uio. I think this is a good
>> moment to ask. Most existing users use UIO_SYSSPACE, but several crypto
>> drivers might allow the UIO_USERSPACE for them.
>>
>> For UIO_USERSPACE, if the page is not resident, the pmap_extract() call from
>> _bus_dmamap_load_buffer() returns 0. So the i/o happens to the page
>> located at 0, which contains real mode IVT and other BIOS sensitive tables.
>>
>> Worse, if the page is resident, but it is mapped at the region which
>> requires COW on write, then DMA will be performed to the wrong page
>> which is typically shared with other innocent users. to the COW area
>> which was not yet copied,
>>
>> Am I missing some trick there ?
>
> No.  The caller is required to wire the pages first in some manner.
> In general bus_dmamap_load_uio() isn't a good idea.  I do believe the
> crypto drivers are careful to wire the buffer first.  I think requiring
> the caller to wire is the only sane way that can be used.
>
> Also, doing DMA to a stack variable is absolutely horrible for a related
> reason since presumably the thread will block while it waits for the DMA
> to complete, and a sleeping thread can be swapped out (including having
> it's stack swapped out).
>
> --
> John Baldwin

The POST commands for UIO DMA may be a bit hairy because the address
space may not be the same as the caller. Someone once told me they got
around that (in a Sparc enviroment?) by shadow mapping the address to
KVA.

--Mark TInguely.