[HEADS UP!] IPFW Ideas: possible SoC 2008 candidate

Robert Watson rwatson at FreeBSD.org
Thu Mar 27 02:15:23 PDT 2008

On Wed, 26 Mar 2008, Julian Elischer wrote:

> it wouldn't.. you'd add them together before presenting them. but every time 
> a packet changes a counter that is shared, there is a chance that it is 
> being altered by another processor, so if you have fine grained locking in 
> ipfw, you really should use atomic adds, which are slow, or accept possibl 
> collisions (which might be ok) but still cause a lot of cross cpu TLB 
> flushing.

In malloc(9) and uma(9), we maintain per-CPU stats, coalescing only for 
presentation, relying on soft critical sections rather than locks to pretect 
consistency.  What's worth remembering, however, is that recent multicore 
machines have significantly optimized the cost of atomic operations on cache 
lines held for write by the current CPU, and so the cost of locking has 
dramatically fallen in the last few years.  This re-emphasizes the importance 
of careful cacheline management for per-CPU data structures (particularly, 
don't put data written by multiple CPUs in the same cacheline if you want the 
benefits of per-CPU access).

Where read-write locking is the best model, Stephan's recent work on rmlocks 
looks quite promising.  In my micro-benchmarks, on recent hardware it performs 
extremely well on SMP for read locks, but still requires optimization for 
UP-compiled kernels.  For stats and writable structures, such as per-CPU 
caches, rmlocks aren't very helpful, but when compared with replicating 
infrequently written data structures across many CPUs, rwlocks/rmlocks offer a 
much simpler and less error-prone programming model.  We need to see more 
optimization and measurement done on rmlocks for 8.x, and the lack of full 
priority propagation for rwlocks has to be kept in mind.

Robert N M Watson
Computer Laboratory
University of Cambridge

More information about the freebsd-hackers mailing list