How to use the DMA Engine in FreeBSD?

[O'Connor, Daniel]

> On 18 Dec 2018, at 05:29, Rajesh Kumar <rajfbsd at gmail.com> wrote:
> Any thoughts (or) suggestions  here?

You don't need to "assign DMA channels" and a PCIe device doesn't "master the bus" because it's all point to point.

"DMA channels" are a legacy of ISA cards, although the terminology may be reused for things like ioat.

For PCI[e] devices which support bus mastering (ie virtually all of them) the device itself issues the transfer. How that is actually done depends on the device - usually you program registers with a scatter/gather (SG) list, or with the physical address of an SG list in system memory and then the device will do the job.

The bus_dma* routines in the kernel are a tool box to let a driver tell the kernel what kind of DMA a card is going to do and to help with things like mapping it as necessary, cache coherency etc etc.

Systems with ioat are pretty rare I believe, the vast majority of DMA in a modern PC is done by PCI cards (or things which look like PCI cards to the OS).

> On Mon, Dec 17, 2018 at 4:54 PM Rajesh Kumar <rajfbsd at gmail.com> wrote:
> 
>> Hi,
>> 
>> Thanks everyone for your inputs. I assume, "PCI busmaster"ing refers to a
>> peripheral device being made as the owner of the bus by the DMA engine and
>> do data transfer.  Please correct me if I am wrong. But, will this bus_dma
>> API uses the DMA engine in hardware (does it support PCI busmastering)?
>> 
>> As I understand, DMA engine will expose DMA channels, which can be used by
>> the client drivers to queue their DMA requests (without CPU intervention)
>> and completions are given through the callback functions. If this is
>> considered the real DMA operation, I don't see options to request DMA
>> channels using bus_dma API's, and the callbacks in bus_dma API is meant to
>> receive just the DMA map details (doesn't indicate a DMA completion).
>> ioat(4) driver seems to be doing what I described above, but that seems to
>> be Intel specific. Can it be used with any DMA controllers?
>> 
>> So, I am wondering what needs to be done in FreeBSD to do the actual DMA
>> involving DMA engine in a more generic way?
>> 
>> Thanks,
>> Rajesh.
>> 
>> 
>> On Sat, Dec 15, 2018 at 6:30 AM Conrad Meyer <cem at freebsd.org> wrote:
>> 
>>> On Fri, Dec 14, 2018 at 8:17 AM Alan Somers <asomers at freebsd.org> wrote:
>>>>> For some Intel based server hardware, there is the "ioat" driver,
>>> which
>>>>> allows for user code to schedule DMA operations.  See ioat(4) for
>>>>> details, including a pointer to the test program.
>>> 
>>> This isn't true (or at least, only for testing).  It's only usable by
>>> the kernel at the moment.
>>> 
>>>> * In what context are callbacks called?  Are they called from a signal
>>>> handler, or in a separate thread, or something else?
>>> 
>>> Directly from the ithread, mostly.  Callers can't take sleep locks or
>>> do very much besides set a flag and wakeup, or kick off a callout or
>>> taskqueue.
>>> 
>>>> * Why isn't ioat.h installed?
>>> 
>>> Why would it be?  It's a kernel interface.
>>> 
>>>> * Are "interrupts" synonymous with callbacks?
>>> 
>>> I don't understand the question.  Interrupts are synonymous with
>>> interrupts.  Like, MSI-X.
>>> 
>>>> * Do you have a rough idea for about the minimum buffer size that
>>>> makes sense to use with ioat?
>>> 
>>> What makes sense will depend on your use case.  It's not necessarily
>>> faster than CPU memcpy, but it may be worth some slowdown to offload
>>> latency-insensitive bulk copies.  (It may make a lot of sense for
>>> asynchronous page zero, if someone wants to bring that back.)  It's
>>> worth benchmarking your specific model.
>>> 
>>> We use it for >= 8kB copies on Broadwell, although that's largely an
>>> artifact of our use case (write sizes are exactly 8 bytes, 512 bytes,
>>> or multiples of 8kB).  IIRC, it shows up as a gen3 PCIe device with
>>> some small number of lanes on Broadwell, although I may be mistaken;
>>> and it may not communicate with RAM over the PCIe bus anyway, given
>>> the tight integration with the CPU.
>>> 
>>> Best,
>>> Conrad
>>> _______________________________________________
>>> freebsd-drivers at freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-drivers
>>> To unsubscribe, send any mail to "freebsd-drivers-unsubscribe at freebsd.org
>>> "
>>> 
>> 
> _______________________________________________
> freebsd-hackers at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org"

--
Daniel O'Connor
"The nice thing about standards is that there
are so many of them to choose from."
 -- Andrew Tanenbaum