Confused about HyperThreading and Performance

[John Baldwin]

On 12-Nov-2003 Daniel Ellard wrote:
> 
> Can someone point me at some non-marketing documentation about
> hyperthreading on the latest Intel chips?  I'm seeing some strange
> performance measurements and I would like to figure out what they
> mean.
> 
> My systems have the following CPU:
> 
> CPU:  Intel(R) Xeon(TM) CPU 1.80GHz (1799.81-MHz 686-class CPU) Origin = "GenuineIntel" Id =
> 0xf27 Stepping = 7
>        
> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUS
> H,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
> 
> They are running FreeBSD 4.8-RELEASE-p13.
> 
> I'm running an application that pumps a lot of data quickly over a
> gigabit NIC (an Alteon AceNIC 1000baseT Gigabit Ethernet, using the ti
> driver).  The network uses normal ethernet frames (MTU=1500) instead
> of jumbo frames, so there are lots of interrupts.  In this case, the
> system is mostly reading data from the network, in case that makes a
> difference.  The application is multi-threaded, using pthreads.
> 
> When I run a kernel using the default configuration (no SMP, no APIC),
> the application takes an average of 8.45 seconds (wall-clock time) to
> run, and this number is consistent from run to run.  When I run a
> kernel built with either SMP or SMP+APIC, it takes an average of 13.25
> seconds, and this number is also quite consistent.  (It's not the
> Alteon; using the Intel Pro/1000 XT server adapter with the em driver
> gives a similar difference, although the intel NIC is slower.)
> 
> Interestingly, the CPU utilization for the default kernel is about
> 75-80%, while for the APIC kernel it is over 95%.  My guess would be
> that something is spinning on a lock in the APIC kernel, but that's
> just a hunch.
> 
> My questions:
> 
> 1.  Any idea why the performance plummets from 8.45 seconds to 13.25?
> 
> 2.  Is the situation any better in FreeBSD 5.1 (or 4.9)?  I know in
>       4.9 APIC is turned on by default, but if this is the result
>       than I won't be upgrading to 4.9 any time soon!

APIC is not on by default for 4.9, it will be for 5.2.  4.9 does have
HTT on by default when you build an SMP kernel though.  Did you include
'options HTT' in your 4.8 SMP kernel?  If not, you aren't actually
using your second CPU.  Also, as someone else mentioned, setting
'machdep.cpu_idle_hlt=1' can be useful on some HTT systems.  However,
p4's have a problem with their interrupt routing that can leave the
second CPU halted for a long time if you do that.

-- 

John Baldwin <jhb at FreeBSD.org>  <><  http://www.FreeBSD.org/~jhb/
"Power Users Use the Power to Serve!"  -  http://www.FreeBSD.org/