How to use the DMA Engine in FreeBSD?

Rajesh Kumar rajfbsd at gmail.com
Wed Dec 19 09:45:59 UTC 2018


Thanks for your explanation Daniel. I will see how to initiate the DMA
transfer in my case after the bus_dma API's has done its part.

On Tue, Dec 18, 2018 at 6:01 PM O'Connor, Daniel <darius at dons.net.au> wrote:

>
>
> > On 18 Dec 2018, at 05:29, Rajesh Kumar <rajfbsd at gmail.com> wrote:
> > Any thoughts (or) suggestions  here?
>
> You don't need to "assign DMA channels" and a PCIe device doesn't "master
> the bus" because it's all point to point.
>
> "DMA channels" are a legacy of ISA cards, although the terminology may be
> reused for things like ioat.
>
> For PCI[e] devices which support bus mastering (ie virtually all of them)
> the device itself issues the transfer. How that is actually done depends on
> the device - usually you program registers with a scatter/gather (SG) list,
> or with the physical address of an SG list in system memory and then the
> device will do the job.
>
> The bus_dma* routines in the kernel are a tool box to let a driver tell
> the kernel what kind of DMA a card is going to do and to help with things
> like mapping it as necessary, cache coherency etc etc.
>
> Systems with ioat are pretty rare I believe, the vast majority of DMA in a
> modern PC is done by PCI cards (or things which look like PCI cards to the
> OS).
>
> > On Mon, Dec 17, 2018 at 4:54 PM Rajesh Kumar <rajfbsd at gmail.com> wrote:
> >
> >> Hi,
> >>
> >> Thanks everyone for your inputs. I assume, "PCI busmaster"ing refers to
> a
> >> peripheral device being made as the owner of the bus by the DMA engine
> and
> >> do data transfer.  Please correct me if I am wrong. But, will this
> bus_dma
> >> API uses the DMA engine in hardware (does it support PCI busmastering)?
> >>
> >> As I understand, DMA engine will expose DMA channels, which can be used
> by
> >> the client drivers to queue their DMA requests (without CPU
> intervention)
> >> and completions are given through the callback functions. If this is
> >> considered the real DMA operation, I don't see options to request DMA
> >> channels using bus_dma API's, and the callbacks in bus_dma API is meant
> to
> >> receive just the DMA map details (doesn't indicate a DMA completion).
> >> ioat(4) driver seems to be doing what I described above, but that seems
> to
> >> be Intel specific. Can it be used with any DMA controllers?
> >>
> >> So, I am wondering what needs to be done in FreeBSD to do the actual DMA
> >> involving DMA engine in a more generic way?
> >>
> >> Thanks,
> >> Rajesh.
> >>
> >>
> >> On Sat, Dec 15, 2018 at 6:30 AM Conrad Meyer <cem at freebsd.org> wrote:
> >>
> >>> On Fri, Dec 14, 2018 at 8:17 AM Alan Somers <asomers at freebsd.org>
> wrote:
> >>>>> For some Intel based server hardware, there is the "ioat" driver,
> >>> which
> >>>>> allows for user code to schedule DMA operations.  See ioat(4) for
> >>>>> details, including a pointer to the test program.
> >>>
> >>> This isn't true (or at least, only for testing).  It's only usable by
> >>> the kernel at the moment.
> >>>
> >>>> * In what context are callbacks called?  Are they called from a signal
> >>>> handler, or in a separate thread, or something else?
> >>>
> >>> Directly from the ithread, mostly.  Callers can't take sleep locks or
> >>> do very much besides set a flag and wakeup, or kick off a callout or
> >>> taskqueue.
> >>>
> >>>> * Why isn't ioat.h installed?
> >>>
> >>> Why would it be?  It's a kernel interface.
> >>>
> >>>> * Are "interrupts" synonymous with callbacks?
> >>>
> >>> I don't understand the question.  Interrupts are synonymous with
> >>> interrupts.  Like, MSI-X.
> >>>
> >>>> * Do you have a rough idea for about the minimum buffer size that
> >>>> makes sense to use with ioat?
> >>>
> >>> What makes sense will depend on your use case.  It's not necessarily
> >>> faster than CPU memcpy, but it may be worth some slowdown to offload
> >>> latency-insensitive bulk copies.  (It may make a lot of sense for
> >>> asynchronous page zero, if someone wants to bring that back.)  It's
> >>> worth benchmarking your specific model.
> >>>
> >>> We use it for >= 8kB copies on Broadwell, although that's largely an
> >>> artifact of our use case (write sizes are exactly 8 bytes, 512 bytes,
> >>> or multiples of 8kB).  IIRC, it shows up as a gen3 PCIe device with
> >>> some small number of lanes on Broadwell, although I may be mistaken;
> >>> and it may not communicate with RAM over the PCIe bus anyway, given
> >>> the tight integration with the CPU.
> >>>
> >>> Best,
> >>> Conrad
> >>> _______________________________________________
> >>> freebsd-drivers at freebsd.org mailing list
> >>> https://lists.freebsd.org/mailman/listinfo/freebsd-drivers
> >>> To unsubscribe, send any mail to "
> freebsd-drivers-unsubscribe at freebsd.org
> >>> "
> >>>
> >>
> > _______________________________________________
> > freebsd-hackers at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> > To unsubscribe, send any mail to "
> freebsd-hackers-unsubscribe at freebsd.org"
>
> --
> Daniel O'Connor
> "The nice thing about standards is that there
> are so many of them to choose from."
>  -- Andrew Tanenbaum
>
>
>


More information about the freebsd-hackers mailing list