svn commit: r266481 - head/sys/x86/x86

[Scott Long]

Author: scottl
Date: Tue May 20 22:43:17 2014
New Revision: 266481
URL: http://svnweb.freebsd.org/changeset/base/266481

Log:
  Old PCIe implementations cannot allow a DMA transfer to cross a 4GB
  boundary.  This was addressed several years ago by creating a parent
  tag hierarchy for the root buses that set the boundary restriction
  for appropriate buses and allowed child deviced to inherit it.
  Somewhere along the way, this restriction was turned into a case for
  marking the tag as a candidate for needing bounce buffers, instead
  of just splitting the segment along the boundary line.  This flag
  also causes all maps associated with this tag to be non-NULL, which
  in turn causes bus_dmamap_sync() to take the slow path of function
  pointer indirection to discover that there's no bouncing work to
  do.  The end result is a lot of pages set aside in bounce pools
  that will never be used, and a slow path for data buffers in nearly
  every DMA-capable PCIe device.  For example, our workload at Netflix
  was spending nearly 1% of all CPU time going through this slow path.
  
  Fix this problem by being more selective about when to set the
  COULD_BOUNCE flag.  Only set it when the boundary restriction
  exists and the consumer cannot do more than a single DMA segment
  at once.  This fixes the case of dynamic buffers (mbufs, bio's)
  but doesn't address static buffers allocated from bus_dmamem_alloc().
  That case will be addressed in the future.
  
  For those interested, this was discovered thanks to Dtrace Flame
  Graphs.
  
  Discussed with: jhb, kib
  Obtained from:	Netflix, Inc.
  MFC after:	3 days

Modified:
  head/sys/x86/x86/busdma_bounce.c

Modified: head/sys/x86/x86/busdma_bounce.c
==============================================================================
--- head/sys/x86/x86/busdma_bounce.c	Tue May 20 22:11:52 2014	(r266480)
+++ head/sys/x86/x86/busdma_bounce.c	Tue May 20 22:43:17 2014	(r266481)
@@ -172,12 +172,35 @@ bounce_bus_dma_tag_create(bus_dma_tag_t 
 	newtag->map_count = 0;
 	newtag->segments = NULL;
 
+	/*
+	 * Bouncing might be needed if there's a filter.
+	 * XXX Filters are likely broken as there's no way to
+	 *     guarantee that bounce pages will also satisfy the
+	 *     filter requirement.
+	 */
 	if (parent != NULL && ((newtag->common.filter != NULL) ||
 	    ((parent->common.flags & BUS_DMA_COULD_BOUNCE) != 0)))
 		newtag->common.flags |= BUS_DMA_COULD_BOUNCE;
 
-	if (newtag->common.lowaddr < ptoa((vm_paddr_t)Maxmem) ||
-	    newtag->common.alignment > 1)
+	/*
+	 * Bouncing might be needed if there's an upper memory
+	 * restriction.
+	 */
+	if (newtag->common.lowaddr < ptoa((vm_paddr_t)Maxmem))
+		newtag->common.flags |= BUS_DMA_COULD_BOUNCE;
+
+	/*
+	 * Bouncing might be needed if there's an alignment
+	 * restriction that can't be satisfied by breaking up
+	 * the segment.
+	 * XXX Need to consider non-natural alignment.
+	 * XXX Static allocations that tie to bus_dmamem_alloc()
+	 *     will likely pass this test and be penalized with
+	 *     the COULD_BOUNCE flag.  Should probably have
+	 *     bus_dmamem_alloc() clear this flag.
+	 */
+	if ((newtag->common.nsegments <= 1) &&
+	    (newtag->common.alignment > 1))
 		newtag->common.flags |= BUS_DMA_COULD_BOUNCE;
 
 	if (((newtag->common.flags & BUS_DMA_COULD_BOUNCE) != 0) &&