How to map device addresses into user space

[John Baldwin]

On Monday, January 07, 2013 08:52:01 AM Dr. Rolf Jansen wrote:
> Am 03.01.2013 um 14:45 schrieb Dr. Rolf Jansen:
> > I am building a loadable kernel module for FreeBSD 9.1-RELEASE x86_64 for
> > a PCI Data Acquisition board from National Instruments.
> > 
> > I need to map the Base Address Registers into user space memory, in order
> > to pass the BAR's to the National Instruments Drivers Development Kit
> > (NI-DDK). The DDK is a complex set of C++ classes running in user space,
> > that read/write directly from/into the BAR's.
> > 
> > The FreeBSD bus_space_* functions are useless in this respect, because
> > the DDK isn't designed that way, I need the BAR addresses mapped into
> > user space.
> > 
> > Having the measurement done by the kernel module is not an option either,
> > because it is math intensive, and kernel modules are build without SSE
> > and with soft float.
> > 
> > I got tiny kernel modules/extensions only providing the mapped addresses
> > of the PCI BAR's running together with the Measurement Routines using
> > the NI-DDK on Darwin (Mac OS X) and Linux.
> > 
> > So, how can I map device addresses into user space on FreeBSD?
> 
> Many Thanks to everybody, who responded.
> 
> I got it working. I wrote a PCI device driver, which in addition to
> open/close/read/write responds also to ioctl and mmap requests from user
> space. In its pci_attach routine, it assembles a custom address_space
> struct containing the addresses of the two BAR's of my NI PCI-DAQ board
> and two huge allocated memory regions for DMA.
> 
>    static struct address_space space;
>    ...
>    ...
>    int              bar0_id,  bar1_id;
>    struct resource *bar0res, *bar1res;
> 
>    bar0_id = PCIR_BAR(0);
>    bar1_id = PCIR_BAR(1);
> 
>    // these resources must be teared down in pci_detach
>    bar0res = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &bar0id,
> RF_ACTIVE); bar1res = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &bar1id,
> RF_ACTIVE);
> 
>    // assemble the address_space struct, i.e.
>    // virtual addr., size, and offset of its members
>    space.bar[0].virtual = rman_get_bushandle(bar0res);
>    space.bar[1].virtual = rman_get_bushandle(bar1res);
>    space.dma[0].virtual = malloc(dmaAIBufSize, M_MYDAQ, M_WAITOK);
>    space.dma[1].virtual = malloc(dmaAOBufSize, M_MYDAQ, M_WAITOK);
> 
>    space.bar[0].size = rman_get_end(bar0res) - rman_get_start(bar0res) + 1;
>    space.bar[1].size = rman_get_end(bar1res) - rman_get_start(bar1res) + 1;
>    space.dma[0].size = dmaAIBufSize;
>    space.dma[1].size = dmaAOBufSize;
> 
>    // this is the crucial part -- The Offsets!
>    // care must be taken, that the offsets are not negative,
>    // therefore, find out the minimum and take this as the base address
>    space.base = MIN(MIN(space.bar[0].virt, space.bar[1].virt),
>                     MIN(space.dma[0].virt, space.dma[1].virt));
>    space.bar[0].offs = space.bar[0].virt - space.base;
>    space.bar[1].offs = space.bar[1].virt - space.base;
>    space.dma[0].offs = space.dma[0].virt - space.base;
>    space.dma[1].offs = space.dma[1].virt - space.base;
> 
> 
> By a call to the ioctl handler of my PCI device driver, this readily
> assembled address_space struct is transferred from the driver to my user
> space measurement controller, and here it must actually be completed by
> subsequent calls to mmap of my device driver:
> 
>    space.bar[0].physical = mmap(0, space.bar[0].size, PROT_READ|PROT_WRITE,
> MAP_SHARED, fd, space.bar[0].offset); space.bar[1].physical = mmap(0,
> space.bar[1].size, PROT_READ|PROT_WRITE, MAP_SHARED, fd,
> space.bar[1].offset); space.dma[0].physical = mmap(0, space.dma[0].size,
> PROT_READ|PROT_WRITE, MAP_SHARED, fd, space.dma[0].offset);
> space.dma[1].physical = mmap(0, space.dma[1].size, PROT_READ|PROT_WRITE,
> MAP_SHARED, fd, space.dma[1].offset);
> 
> 
> Because of the coherent offsets, the mmap handler of my PCI device driver
> can be kept quite simple:
> 
> static int mydaq_mmap(struct cdev *dev, vm_ooffset_t offset, vm_paddr_t
> *paddr, int nprot, vm_memattr_t *memattr) {
>    *paddr = vtophys(space.base + offset);
>    return 0;
> }

Note that if your BARs are not contiguous you can allow mapping of things in 
the middle.  This is a potentital security hole if your driver can be opened 
by non-root.  A more robust approach is to define a virtual address space for 
your device.  This is also a lot easier to program against in userland.
For example, you could define offset 0 as mapping to BAR 0 and the next chunk
of virtual address space (the offset) map to BAR 1, etc.:

static int
mydaq_mmap(...)
{
	struct mydaq_softc *sc;  // you should be storing things like the pointers
                             // to your BARs in a softc

	sc = dev->si_drv1;
	if (offset <= rman_get_size(sc->bar0res))
		return (rman_get_start(sc->bar0res) + offset);
	offset -= rman_get_size(sc->bar0res);
	if (offset <= rman_get_size(sc->bar1res))
		return (rman_get_start(sc->bar1res) + offset);
	offset -= rman_get_size(sc->bar1res);
	if (offset <= dmaAIBufSize)
		return (vtophys(sc->dma[0]) + offset);
	offset -= dmaAIBufSize;
	if (offset <= dmaAOBufSize)
		return (vtophys(sc->dma[1]) + offset);
	
	/* Invalid offset */
	return (-1);
}
	
Then you can use fixed offsets in userland for your mmap requests.  Also, 
rman_get_bushandle() is not appropriate to get a virtual address, that is an 
implementation detail you should not count on.  You could use 
rman_get_virtual(), but that is not the best practice.  Using rman_get_start() 
to get a PA directly is a better approach.  On some embedded platforms with 
translating bridges it would not work as Warner noted, but it should work fine 
for most platforms that FreeBSD supports.

-- 
John Baldwin