libpcap perf improvement? latest ideas?

[sbahra at kerneled.org]

> On Thu, 5 Oct 2006, Jochen Kaiser wrote:

[...]

>> after reading a german master thesis [1] (dated 12/2004) about pcap  
>>  performance (with comparison of linux and freebsd) I searched   
>> freebsd resources for pcap improvements. Unofortunately I did not   
>> find any improvements like PF_RING and/or efforts for reducing the   
>> number of copy operations from device to user space.

Hi Jochen,

I suggest you take a look at http://www.kerneled.org/?p=15
What is implemented here is reasonably light-weight but still uses a  
ring buffer model. I agree that a reference model can be used to  
reduce the number of copies done currently for BPF. I personally  
haven't seen work there and I have yet the time unfortunately to  
shut-up and just code such a thing.

>> Maybe I think too simple because I don't know how SMP fine locks   
>> are influencing this (maybe it is very complex to improve that when  
>>  you want to avoid side effects.).

Basically, the reason for bad performance here in Linux in the  
PF_PACKET sense is high system call overhead. BPF does collective  
buffering, so system call over-head is less. Applications doing packet  
analysis will also find a lot of required information in the BPF  
header as well (such as timestamps). In PF_PACKET you are forced to do  
a system call per-acquisition and another system for receiving the  
time-stamp of the last packet read for example.

Other research papers have also shown that BPF still out-weights  
PF_PACKET and other zero-copy PF_* solutions with small packets as  
well. The kerneled.org post helps to explain this in a general sense.  
Basically, zero-copy isn't the end-it-all solution but for larger  
packets it makes great sense. The application I've been developing for  
a while now makes use of both depending on average packet size.

On Thu, 6 Oct 2006, Robert Watson wrote:
> Quite a bit of work has been done on zero-copy for BPF, but none of it
> really commitable.  Christian Peron (CC'd) and I have been talking
> about doing something that is commitable, but some of the details (such
> as memory ownership) are still very much up in the air.

Is there any archive or summary online noting the approaches you guys  
would like to take with BPF? I am really interested in this, and when  
my time permits would love to write some code. I am also interested in  
the other work you noted that is not committable. Could we be provided  
some URLs please? :-)

> PF_RING takes an interesting approach, and one we should look at, but we'd
> also like to keep all the benefits of BPF rather than discard them,  
> so need to
> consider how best to apply elements of the approach in our context.
> I'd like to see something like this happen for FreeBSD 7.0, with a
> possible backport if it goes really well. :-)

The problem with PF_RING is it's static ring buffer model and concept  
of marking slots. This doesn't make it so feasible for real-world  
applications. I proposed a model that allows for dynamic ring buffer  
size and signaling for soft and hard-limits to allow application  
buffering to handle potential drops (or request specific allocation  
strategies from the kernel). A Professor at my university also had an  
idea of providing packet acquisition prioritization (push specific  
packets to the beginning of the packet copy queue, which isn't  
currently a concept in BPF) as a BPF language extension. An mmap  
interface to BPF will be cool but unless we also provide stubs to  
automize packet management from user-space it could be doomed to be  
like PF_RING. :-P

[...]

Regards.
--
Samy Al Bahra