DMA bounce buffers

[Scott Long]

Sepherosa Ziehau wrote:
> On Mon, Jan 26, 2009 at 9:05 PM, Barney Cordoba
> <barney_cordoba at yahoo.com> wrote:
>> I have a problem with the ixgbe driver, where bus_dmamap_load_mbuf_sg() is
>> failing. The underlying failure is reserve_bounce_pages(). It seems that
>> there aren't any bounce pages allocated. What could be the cause of this? Is there some tunable?
> 
> I don't think the hardware supported ixgbe(4) will need bounce pages
> on RX/TX buffers (according to data sheet, the hardware supports full
> 64bits address space accessing and no alignment constraints on RX/TX
> buffer start address).  However, the driver's buffer busdma is kinda
> misconfigured: in ixgbe_allocate_{transmit,receive}_buffers(), the
> buffers' alignment is set to PAGE_SIZE (?!), this means that almost
> all TX/RX operation will require bouncing.  The best case is poor
> performance due to additional and unnecessary memory copy in
> bus_dmamap_sync(); the worst case is completely unfunction NIC due to
> RX ring could not be filled during ifnet.if_init().
> 

Yes, no high-speed NIC should be using bounce buffers.  I haven't looked 
at the ixgbe data sheets, but I'd be highly surprised if this hardware
actually needed buffer alignment like that.  If it does, then the better
thing to do is to either use 4k mbuf clusters (and make sure that the 
system doesn't try to treat them as jumbo frames), or have the rx ring
allocator throw away and reallocate any clusters that aren't already on
4k boundaries (normal clusters are 2k, so it'll have a nominal 50% 
chance of getting what it needs).  Neither of these solutions is ideal,
though.

Scott