MMC/SDIO stack under CAM

[Adrian Chadd]

hi,

Let me just reiterate some .. well, experience doing this stuff at QCA.

You really, absolutely don't want too much overhead in the MMC/SDIO
path between whatever is issuing things and the network driver.

There was significant performance work done at QCA on a local MMC/SDIO
driver and bus to get extremely low latency and CPU utilisation when
pushing around small transactions. The current CAM locking model is
not geared towards getting to high transaction rates.

You may think this is a very architecturally pretty solution and it
indeed may be. But if it doesn't perform as well as the existing local
hacks that vendors have done, no company deploying this hardware is
going to want to use it. They'll end up realising there's this massive
CAM storage layer in between and either have to sit down to rip it up
and replace it with something lightweight, or they'll say "screw it"
and go back to the vendor supplied hacked up Linux solution.

So I highly recommend you profile things - and profile things with
lots of small transactions. If the CAM overhead is more than a tiny,
tiny fraction of CPU at 25,000 pps, your solution won't scale. :-)



-a


On 24 February 2014 06:26, Ilya Bakulin <ilya at bakulin.de> wrote:
> On Sun, Feb 16, 2014 at 11:05:35AM -0700, Warner Losh wrote:
>> that's cool! I'd thought of writing a mmcsim that I could use to develop the stack on x86, but time pressures of my job precluded that (though in hind sight, it would have saved a lot of time).
>
> I assume that by "mmcsim" you mean not MMC SIM for CAM, but MMC SIMulator :-)
> Yeah so essentially it should be able to return some "fake" data
> to the upper layer, maybe based on configuration (ie 1 GB SD, 16 GB SDHC, ...).
>
>> > For SDIO card, the situation will be different. Essentially,
>> > we should allow a device driver to send arbitrary messages to the card.
>> > So the device driver will attach directly to the scbus.
>> >
>> > Please let me know your thoughts about it.
>> > I really want SDIO make its way into the kernel, because there
>> > is an increasing number of ARM boards on the market that have integrated
>> > SDIO WLAN on them and we cannot support them fully.
>>
>> Generally, I like this idea...  I'd be very interested in some of the specifics to make the direct attachment work with SDIO. That's the one area I either don't think I understand your proposal well enough, or I do and I'm concerned about it. :) So maybe write a bit more about the details of the SDIO cards and how they's interact with CAM in this scenario...
>>
>> The notion of moving MMC/SD into the CAM stack has been around since I started working on MMC. At the time, CAM was too SCSI centric to do it, but Alexander Motin's work has generally improved that situation, so it may be possible now to do it and shake out the few dark corners of CAM where this isn't the case. I also think it should help performance a lot since we're currently single threaded to the card (but that's also the nature of the bus), which introduces some round-trip latencies between too many threads that CAM may help us avoid.
>>
>> Warner
>
> After talking with Alexander about CAM, here is the updated proposal:
>
> * All card communication logic should be placed in MMC XPT layer, with the new CCB type
>   that describes MMC request + some data needed by the system.
>   So almost everything from sys/dev/mmc/mmc.c goes there.
> * da(4) is a driver for SCSI disks. It makes more sense to adapt mmcsd(4) to use CAM.
>   So mmcsd(4) will create CCBs and pass them to CAM layer, getting async notifications
>   via standard CAM callback mechanism.
> * It is possible to write drivers for SDIO devices exactly like adopted mmcsd(4).
>   Every such driver would be then built as CAM peripheral driver, which means it will be able
>   to send/receive CCBs.
>   To do that, every driver has to call PERIPHDRIVER_DECLARE() macro, subscribing to CAM "new device found" events.
>
> The ongoing work for MMC CAM is here:
> https://github.com/kibab/freebsd/compare/master...mmccam
> Not so much stuff here yet.
>
> PS: In theory, ssince there is already an existing interface for passing CAM CCBs between userland and kernel,
> it WOULD be posssible to create device drivers entirely in userland... Like microkernel OSes do.
> But our network stack doesn'have such feature I guess...
>
> --
> Ilya