serious slowdown in 5.3-BETA1 using SCHED_4BSD and SMP

[Oliver Lehmann]

Hello,

I've noticed a serious speeddown by using SMP. When I run the same system
with only one CPU, the system works quite well. But after sticking in the
second CPU, the system slows down noticeable. The whole system responds
quite laggy.
That started to happen after I updated FreeBSD from 5.2.1 to 5.3-BETA1. On
5.2.1 I had all kernel debugging options enabled. With that kernel-config
5.3 was not useable in any way. For example playing 128kbit mp3s over nfs
was nearly impossible because it stoped playing every 10-30 seconds for
2-5 seconds....
I did a "ln -s aj /etc/malloc.conf" and disabled all the [DGK]DB, WITNESS*
and INVARIANT* options in my Kernel-config file. IT tokk me 9:52h(!!) to
compile that kernel (make kernel).


Now, booted with that kernel (NUDEL.bak) - SMP feels still much more
slower then 5.2.1 felt.


To show the speeddowns, I built now the attached kernel "NUDEL":

##########################################################################
#### on System: nudel.dmesg
  Booted kernel: "NUDEL.bak" (SCHED_4BSD)
  1 CPU :	 3927.42 real      3279.99 user       454.95 sys
  2 CPUs:	51546.74 real     44414.65 user      5151.46 sys (14hrs!!!)


  Booted kernel: "NUDEL" (SCHED_ULE)
  1 CPU :	 3943.72 real      3284.04 user       459.58 sys
  2 CPUs:	 3864.97 real      3287.69 user       454.52 sys


##########################################################################
#### on System: curry.dmesg
  Booted kernel: "CURRY" (SCHED_4BSD)
  1 CPU :	4376.41 real      3652.60 user       435.60 sys



I built with:

rm -rf /usr/obj/i386-5.3/usr/src/sys/NUDEL && \
/usr/bin/time env MAKEOBJDIRPREFIX=/usr/obj/i386-5.3 make kernel


My make.conf:

KERNCONF?=NUDEL
# MAKEOBJDIRPREFIX=/usr/obj/i386-5.2.1
WRKDIRPREFIX=/usr/obj/${MACHINE}-${OSREL}
#CPUTYPE=p2
# -- use.perl generated deltas -- #
# Created: Fri Mar 21 00:28:26 2003
# Setting to use base perl from ports:
PERL_VER=5.6.1
PERL_VERSION=5.6.1
PERL_ARCH=mach
NOPERL=yo
NO_PERL=yo
NO_PERL_WRAPPER=yo

I've attached the vmstat -i output during make

For example - a simple "make buildworld" with 2CPUs/SCHED_4BSD for (and
on) 5.2.1-RC2 took only 16916.70sec (I'm sorry, but I've no kernel
building times archived)


!!!And it's not just the kernel build - the whole system with running
2 CPUS + SCHED_4BSD is _slow_. I just took the kernel build as an
example!!!

Something seems to be verry broken with SCHED_4BSD+SMP

-- 
 Oliver Lehmann
  http://www.pofo.de/
  http://wishlist.ans-netz.de/
-- 
 Oliver Lehmann
  http://www.pofo.de/
  http://wishlist.ans-netz.de/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: NUDEL
Type: application/octet-stream
Size: 10416 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20040829/fcc0afdf/NUDEL.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: NUDEL.bak
Type: application/octet-stream
Size: 9215 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20040829/fcc0afdf/NUDEL-0001.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: CURRY
Type: application/octet-stream
Size: 9572 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20040829/fcc0afdf/CURRY.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nudel_dmesg.out
Type: application/octet-stream
Size: 4615 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20040829/fcc0afdf/nudel_dmesg.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: curry_dmesg.out
Type: application/octet-stream
Size: 3949 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20040829/fcc0afdf/curry_dmesg.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vmstat_NUDEL.bak_1CPU
Type: application/octet-stream
Size: 677 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20040829/fcc0afdf/vmstat_NUDEL.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vmstat_NUDEL.bak_2CPU
Type: application/octet-stream
Size: 677 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20040829/fcc0afdf/vmstat_NUDEL-0001.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vmstat_NUDEL_2CPU
Type: application/octet-stream
Size: 677 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20040829/fcc0afdf/vmstat_NUDEL_2CPU.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vmstat_NUDEL_1CPU
Type: application/octet-stream
Size: 677 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20040829/fcc0afdf/vmstat_NUDEL_1CPU.obj