network statistics in SMP

[Robert Watson]

On Sun, 20 Dec 2009, Harti Brandt wrote:

> RNMW>> I will try to come up with a patch for the kernel/user interface in the
> RNMW>> mean time. This will be for 9.x only, obviously.
> RNMW>
> RNMW>Sounds good -- and the kernel stats capture can "grow into" the full
> RNMW>report format as it matures.
> RNMW>
> RNMW>> Doesn't this help for output only? For the input statistics there still
> RNMW>> will be per-ifnet statistics.
> RNMW>
> RNMW>Most ifnet-layer stats should really be per-queue, both for input and
> RNMW>output, which may help.
>
> As far as I can see currently the driver just calls if_input which is the 
> interface dependend input function. There seems to be no driver-independent 
> abstraction of input queues. (The hatm driver I wrote several years ago has 
> to input queues in hardware corresponding to 4 (or 8?) interrupt queues, but 
> somewhere in the driver you put all of this through the single if_input 
> hook). Or is there something I'm missing?

You're not missing anything, it's the code for what I describe that's missing 
:-).  Adding an ifnet-layer abstration for input and output queues (if only to 
hold stats in a cross-driver way) is something that's come up at the last 
devsummit or two, and something we need to sort out for 9.0.

> I was already thinking about how to fit the vimage stuff into the SNMP 
> model. The simplest way is to run one SNMP daemon per vimage. Next comes 
> having one daemon that has one context per vimage. Bsnmpd does its own 
> mapping of system ifnet indexes to SNMP interface indexes, because the 
> allocation of system ifnet indexes does not fit to the RFC requirements. 
> This means it will detect when an interface is moved away from a vimage and 
> comes back later. If the kernel statistics are stable over these movements, 
> there is no need to declare a counter discontinuity via SNMP. On the other 
> hand these operations are probably seldom enough ...

For a system with thousands of virtual network stacks, it would be nice to 
avoid requiring one process/vimage, but given the way we currently link 
processes to vimages, arranging that is currently awkward.  We're also well 
past the point where a 16-bit integer can describe what is required out of our 
interface system; perhaps we just bite the bullet and roll to 32-bit if 
indexes, but also give each interface a uuid (or the like) that is stable over 
vimage moves.

Robert N M Watson
Computer Laboratory
University of Cambridge