RFC: FreeBSD I/OAT driver
pdeuskar at freebsd.org
Wed Aug 30 20:41:28 UTC 2006
Andrew Gallatin [gallatin at cs.duke.edu] wrote:
> Jack Vogel writes:
> > We are making our development driver for the I/OAT engine available for
> > download, experimentation, and comment available at:
> > http://sourceforge.net/project/showfiles.php?group_id=42302&package_id=202220
> > This includes a core driver for the dma hardware and a set of stack changes
> > to allow use of the engine on the receive side of the stack.
> > There are certainly rough edges and limitations in this code, but we have run
> > it internally and seen some great results.
> > I would like to see this get into CURRENT, so anything Prafulla and I can do
> > to help or answer questions, send us email.
> Excellent! Can you share some of these results? I would love to try
> it, but I don't have FreeBSD on any machine with I/OAT hardware.
Dual core Woodcrest
Netperf Receive Test - 64k IO size
6.1-RELEASE SMP Kernel
MTU 1500 bytes
Num Ports Thr(Native) Thr(I/OAT) CPU (Native) CPU (I/OAT)
(Mbps) (Mbps) (%) (%)
1 943 943 14 11
2 1886 1886 46 22
4 1945 2531 84 54
It scales fairly linearly as number of ports increase.
Haven't run more than 4 port test with FreeBSD though.
> I've taken a very quick look at it. Maybe I'm just being dense,
> but I don't like the name "dma_" being in the global namespace.
> Maybe things (like dma_*_list should be called at least
> dmaengine_*_list, etc.
Yeah - probably ioatdma_ would be more clear.
> There are some style(9) defects which I'm sure others who are more
> proficient at style(9) than I am will point out (// comments, function
> names not starting in column 0, etc).
> How deep would you expect so->dma_wait_queue to get? Would it make
> sense to keep a pointer to the last item so that insertion is O(1),
> rather than O(N)?
The size is dependant on Application IO size and MSS.
So you are right it might be useful to keep a pointer to last item.
At higher IO sizes it can get quite deep for 1500 byte MTU.
> Would it be possible to have a sysctl tunable threshold, below which
> the system does a normal uiomove? A normal copyout() will certainly
> be faster at some point..
Yes - it doesn't make sense to use the DMA engine for small packets.
Also you get more benefit if you overlap IO with computation.
> Thanks for the great work!
Thank you for your help earlier.
More information about the freebsd-current