network statistics in SMP

Julian Elischer julian at
Tue Dec 15 09:07:56 PST 2009

John Baldwin wrote:
> On Tuesday 15 December 2009 4:38:04 am Harti Brandt wrote:
>> Hi all,
>> I'm working on our network statistics (in the context of SNMP) and wonder, 
>> to what extend we want them to be correct. I've re-read part of the past 
>> discussions about 64-bit counters on 32-bit archs and got the impression, 
>> that there are users that would like to have almost correct statistics 
>> (for accounting, for example). If this is the case I wonder whether the 
>> way we do the statistics today is correct.
>> Basically all statistics are incremented or added to simply by a += b oder 
>> a++. As I understand, this worked fine in the old days, where you had 
>> spl*() calls at the right places. Nowadays when everything is SMP 
>> shouldn't we use at least atomic operations for this? Also I read that on 
>> architectures where cache coherency is not implemented in hardware even 
>> this does not help (I found a mail from jhb why for the mutex 
>> implementation this is not a problem, but I don't understand what to do 
>> for the += and ++ operations). I failed to find a way, though, to 
>> influence the caching policy (is there a function one can call to 
>> change the policy?).
> Atomic ops will always work for reliable statistics.  However, I believe 
> Robert is working on using per-CPU statistics for TCP, UDP, etc. similar to 
> what we do now for many of the 'cnt' stats (context switches, etc.).  For 
> 'cnt' each CPU has its own count of stats that are updated using non-atomic 
> ops (since they are CPU local).  sysctl handlers then sum up the various per-
> CPU counts to report global counts to userland.

the trouble is that PCPU and VNET collide.  you then need to have
Per-CPU, per VNET counters. which would be yet a different pool of 
linker set symbols..


More information about the freebsd-arch mailing list