FreeBSD 4.8 SMP performance

[Paul Koch]

Last week we did a fairly major installation of our network 
monitoring software at a customers' site on a couple of 
identical quad processor machines running FreeBSD-4.8.
One as a production machine, the other as a warm spare.

The machines were unfortunately "hand me downs" from the 
weenies server group:
 - Compaq server of some description
 - quad 700Mhz Intel Xeon 2M cache CPUs
 - 4 gigabytes of ram
 - usual arrangement of Compaq/Adaptec SCSI and raid 
    controllers
 - 1 * raid 1 disk setup
 - 1 * raid 5 disk setup
 - single and dual port Intel NICs

We originally intended to benchmark a 4.8 and 5.1 box side
by side to see what the SMP performance gains were, but ran
out of time due to SMP kernel and dual port Intel NIC problem.
When running 5.1 with SMP, the dual port NICs would lock up
with "fxp0 device timeout" messages. Thinking this was a 5.x 
related problem, we fell back to 4.8 SMP, eventually finding 
that the same problem also existed in it.  A quick change of NIC
fixed our immediate issue.  Time ran short so we unfortunately
never got back to 5.1 SMP or to investigate the dual port NIC
problems.

Having never run a SMP box before, we were surprised when 
running 'top' on the 5.1 machine to see a process state of "Giant".
Thought this was suppose to be fine grain SMP :)

Our product is a real-time network performance monitoring 
application, only available on FreeBSD of course. Statscout 
NPM is typically deployed as a "blanket monitoring" solution. 
That is, monitor every port on every switch/router/repeater each
minute (ie. ping/snmp) for things like bytes, frames, errors, 
discards and status.

This particular customer has are rather large Cisco network, 
consisting of over 5000 switches/routers and ~100,000 network
interfaces. Our software was configured to poll some devices at
60 second intervals and others at 120 seconds, collecting 10 
SNMP OIDs per interface per poll. That's approximately one 
million SNMP OIDs to collect.

The customer provided a list of device names/IP addresses and
then it took just under one hour to SNMP walk all 5000 devices
and populate our configuration. Once running and with a bit of 
tuning to our software, the machine was cruising along collecting, 
processing and saving over 500,000 SNMP OIDs per minute 
(8333/sec), within ~20,000 SNMP requests, and ~4000 pings 
per minute, using less than 50% of available CPU. We were very
impressed with how smoothly and responsive the SMP machine
ran doing lots of simultaneous disk/network activity, while also
servicing user requested real-time reports.

It was a pity that the network wasn't big enough to push the 
machine to its limits !


We noticed several things on 4.8:

 - Some of our busy processes jumped around 'a lot' between
   the different processors. It would be nice if processor affinity 
   was implemented to get the best out of those 2M CPU caches.

 - "systat -vm" would chew up ~10 percent on one CPU when
   running a SMP kernel. Don't know what was going on there.

 - The dual port Intel fxp lockup mentioned above.

	Paul.
-- 
Paul Koch (CTO Statscout Pty Ltd)
http://www.statscout.com
Brisbane, Australia