mbuf changes

[Robert N. M. Watson]

On 5 Oct 2010, at 17:01, Karim Fodil-Lemelin wrote:

> I will share some of the experience I had doing embed mtags. Hopefully its relevant :)
> 
> The idea of carrying a certain amount of mbuf tags within the mbuf structure is somewhat similar but much cleaner, imo, then Linux's skbuff char cb[40 - 48] (it was 40bytes in 2.4.x ...). Now this idea is not new although as you know the devil is in the details...

Hi Karim:

This sounds like very interesting work, and something we should figure out how to generalize for FreeBSD. I had also been pondering something along these lines in order to improve MAC label performance when using ubiquitously labeled policies. Since MAC often stores references to externally managed data from mbuf labels (i.e., deep structures, rather than just bytes of data) it's especially important to make sure we get the tear-down stuff right...

Robert

> 
> What we did for BSD is create a container in the mbuf and extend the API with functions we (pompously) called m_tag_fast_alloc() and m_tag_fast_free(). This means the standard m_tag_alloc() is still supported across the system and the old behavior is unchanged (list of allocated struct attached to the packet header). Whats different is the availability of a 'fast' call that directly uses the container within the mbuf, effectively avoiding those malloc and cache misses. I'll explain later how we effectively support calling m_tag_delete on a 'fast' tag.
> 
> The trick to save CPU cycles was also to quickly revert back to the standard tag mechanism if some component in the system is manipulating the tag list by deleting elements. Effectively, the m_tag_fast_free is a NOP and fast tags are not deleted once allocated (unless m_free is called on the mbuf of course). When m_tag_delete is called the container simply becomes 'fast tag' invalid for further additions. This is not flexible but has the merit of reducing the overall number of operations given that almost no components are deleting tags without deleting the mbuf (loopback does but its a special case).
> 
> One last thing we did is perform various operational tests to come up with the most statistically optimized container size. Now this is much easier to do on a proprietary system then for a general purpose OS but its certainly possible.
> 
> Finally, we did see speed increase for our application and if someone is interested I could provide a patch although I would have to rewrite it without the proprietary bits in it.
> 
> Best regards,
> 
> Karim.