FreeBSD 4.8 SMP performance

[Cagle, John (ISS-Houston)]

Actually, there's no reason to avoid using the multi-port NICs, as long
as you don't use full table APIC mode (which doesn't work yet in
FreeBSD).  I know that the kernel maintainers are working on this, so it
should be fixed RSN.

Also, there's no real trickery involved in the OS selection menu.  It
simply tries to set the BIOS defaults appropriate for the Operating
System you tell it you want to run.  For instance, if you select "Linux"
it will use APIC "Full Table Mapped" mode, which should work for
FreeBSD.

The reason it worked when you weren't running SMP is that it probably
wasn't using the APICs.

-----Original Message-----
From: Paul Koch [mailto:paul.koch at statscout.com] 
Sent: Monday, June 23, 2003 7:47 PM
To: Cagle, John (ISS-Houston)
Cc: freebsd-smp at freebsd.org
Subject: Re: FreeBSD 4.8 SMP performance


Ah! thanks, I did a bit of reading on this and it sounds like we need to
keep clear of these types of cards for the moment. We didn't plan on
using the multiport NICs, it's just what came in the box.  Is this only
an issue when running SMP ? because the NICs appeared to work fine when
initially running a 
non-SMP kernel.

OS selection of "Linux": hmm, didn't know about that. More reading to be
done here. I assume this does is some vendor specific trickery setting
the hardware up.

	Paul.

On Mon, 23 Jun 2003 11:55 pm, Cagle, John (ISS-Houston) wrote:
> Paul,
>
> There's a known issue with APIC IRQ routing when using Multi-Port NICs

> in PCI slots.  The existing code doesn't know how to swizzle the IRQs 
> through the PCI-to-PCI bridges that are on the NIC.  If you want to 
> use Multi-port NICs, you'll need to make sure you're not running in 
> APIC Full Table mode.  Double check that your OS Selection (in ROM 
> Based Setup) is "Linux".
>
> Of course, that may not be the problem you're experiencing, but it's 
> my best guess.
>
> Regards,
> John
> --------------------------------
> John Cagle     john.cagle at hp.com
> Principal Member Technical Staff
>    Industry Standard Servers
>     Hewlett-Packard Company
>
> -----Original Message-----
> From: Paul Koch [mailto:paul.koch at statscout.com]
> Sent: Monday, June 23, 2003 1:13 AM
> To: freebsd-smp at freebsd.org
> Subject: FreeBSD 4.8 SMP performance
>
>
> Last week we did a fairly major installation of our network monitoring

> software at a customers' site on a couple of identical quad processor 
> machines running FreeBSD-4.8. One as a production machine, the other 
> as a warm spare.
>
> The machines were unfortunately "hand me downs" from the weenies 
> server group:
>  - Compaq server of some description
>  - quad 700Mhz Intel Xeon 2M cache CPUs
>  - 4 gigabytes of ram
>  - usual arrangement of Compaq/Adaptec SCSI and raid
>     controllers
>  - 1 * raid 1 disk setup
>  - 1 * raid 5 disk setup
>  - single and dual port Intel NICs
>
> We originally intended to benchmark a 4.8 and 5.1 box side
> by side to see what the SMP performance gains were, but ran out of 
> time due to SMP kernel and dual port Intel NIC problem. When running 
> 5.1 with SMP, the dual port NICs would lock up with "fxp0 device 
> timeout" messages. Thinking this was a 5.x related problem, we fell 
> back to 4.8 SMP, eventually finding that the same problem also existed

> in it.  A quick change of NIC fixed our immediate issue.  Time ran 
> short so we unfortunately never got back to 5.1 SMP or to investigate 
> the dual port NIC problems.
>
> Having never run a SMP box before, we were surprised when running 
> 'top' on the 5.1 machine to see a process state of "Giant". Thought 
> this was suppose to be fine grain SMP :)
>
> Our product is a real-time network performance monitoring application,

> only available on FreeBSD of course. Statscout NPM is typically 
> deployed as a "blanket monitoring" solution. That is, monitor every 
> port on every switch/router/repeater each minute (ie. ping/snmp) for 
> things like bytes, frames, errors, discards and status.
>
> This particular customer has are rather large Cisco network, 
> consisting of over 5000 switches/routers and ~100,000 network 
> interfaces. Our software was configured to poll some devices at 60 
> second intervals and others at 120 seconds, collecting 10 SNMP OIDs 
> per interface per poll. That's approximately one million SNMP OIDs to 
> collect.
>
> The customer provided a list of device names/IP addresses and then it 
> took just under one hour to SNMP walk all 5000 devices and populate 
> our configuration. Once running and with a bit of tuning to our 
> software, the machine was cruising along collecting, processing and 
> saving over 500,000 SNMP OIDs per minute (8333/sec), within ~20,000 
> SNMP requests, and ~4000 pings per minute, using less than 50% of 
> available CPU. We were very impressed with how smoothly and responsive

> the SMP machine ran doing lots of simultaneous disk/network activity, 
> while also servicing user requested real-time reports.
>
> It was a pity that the network wasn't big enough to push the machine 
> to its limits !
>
>
> We noticed several things on 4.8:
>
>  - Some of our busy processes jumped around 'a lot' between
>    the different processors. It would be nice if processor affinity
>    was implemented to get the best out of those 2M CPU caches.
>
>  - "systat -vm" would chew up ~10 percent on one CPU when
>    running a SMP kernel. Don't know what was going on there.
>
>  - The dual port Intel fxp lockup mentioned above.
>
> 	Paul.