per-cpu counters (Re: svn commit: r190967 - head/sys/netinet)

[Robert Watson]

On Sun, 12 Apr 2009, Robert Watson wrote:

>> As an example, an approach i was considering is replace the existing 
>> counters with an index in a global_counters[MAXCPU][] array so the counter 
>> update will become something like this:

FYI, one other reason to put it all behind per-subsystem macros is that it 
allows us to easily prototype another statistics implementation, the no-op, on 
a per-subsystem basis.

Robert N M Watson
Computer Laboratory
University of Cambridge

>> 
>> -	pimstat.pims_rcv_registers_wrongiif++;
>> +	global_counters[PCPU_GET(cpuid)]
>> +		[pimstat.pims_rcv_registers_wrongiif]++;
>> 
>> When you allocate an entry you also need to allocate an entry in the 
>> global_counters, but entries have fixed size, and the modules that you 
>> build will not depend on the MAXCPU value.
>> 
>> All of this is still compatible with the macros you are adding, but I 
>> wonder why we should have many PIMSTAT_INC, IPFW_INC, NATD_INC and so on 
>> instead of more generic COUNTER64_INC() or COUNTER32_INC()
>
> I have a project along these lines in progress, and will post the proposal to 
> arch@ once I've finished prototyping it.  In particular, it provides common 
> implementations of "reset" and "report" in order to expose a single userspace 
> version of the structure via sysctl.
>
> The reason to use per-subsystem macros is that there's another dimmension 
> here: virtualization.  Some counters are per-CPU, others are per-VIMAGE, and 
> many are per-CPU-per-VIMAGE.  Since virtualization is a work-in-progress, and 
> they haven't yet virtualized all the counter blocks, I figured it was best to 
> push all the counter updates behind subsystem-specific macros, and then 
> virtualization would touch only the macro definitions, not the implementation 
> in every place.  In the sample implementation you show above, changes have to 
> be made to every instance of the counter when virtualization is 
> added/removed/changed for PIM, making per-CPU changes hard to MFC.
>
> Robert N M Watson
> Computer Laboratory
> University of Cambridge
>