High system in %system load .

Igor Lyapin igor.lyapin at gmail.com
Tue Nov 18 07:02:58 PST 2008


Hello
Got strange problem with high system "%system load" and very slow user level
programs (apache+php+mysql) behavior
gstat shows 1.5-4% hard disk busy load but system shows about 20-30% load
while user load is max 5%.
vmstat shows from 2 to 35 process  in "b" state.
Now use 7.0-RELEASE-p5 , but the same problem was with 7.0-RELEASE.
And have no idea what to do with this.



-------------------------------------------------------------------
#uname -a
FreeBSD i.x00.net 7.0-RELEASE-p5 FreeBSD 7.0-RELEASE-p5 #0: Mon Nov 10
19:49:07 MSK 2008     root at i.x00.net:/usr/obj/usr/src/sys/KERNEL-4BSD  amd64
-------------------------------------------------------------------
#top

last pid: 47964;  load averages:  1.26,  1.62,  1.75                 up
0+19:17:13  17:11:06
287 processes: 10 running, 277 sleeping
CPU states:  2.2% user,  0.0% nice, 28.3% system,  0.2% interrupt, 69.3%
idle
Mem: 1286M Active, 1729M Inact, 478M Wired, 131M Cache, 214M Buf, 302M Free
Swap: 8192M Total, 8192M Free
-------------------------------------------------------------------
#vmstat 5

procs      memory      page                    disks     faults      cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr ad4 ad6   in   sy   cs us
sy id
 1 24 0 4357672 419648 10598   1   0   0 10648  34   0   0  558 25212 9072
6 15 80
 1 29 0 4365116 414892  3060   0   0   0  2780   0   0   0  975 12270 11501
3 29 68
 1 25 0 4392488 408596 14142   0   0   0 13903   0   0   0  526 16696 10389
6 30 64
-------------------------------------------------------------------
#ps ax | grep " [DLW] "
45585  ??  D      0:04.68 /usr/local/sbin/httpd
45975  ??  D      0:04.92 /usr/local/sbin/httpd
45981  ??  D      0:29.91 /usr/local/sbin/httpd
45995  ??  D      0:23.99 /usr/local/sbin/httpd
46019  ??  D      0:26.76 /usr/local/sbin/httpd
46160  ??  D      0:18.89 /usr/local/sbin/httpd
46287  ??  D      0:19.88 /usr/local/sbin/httpd
46303  ??  D      0:05.14 /usr/local/sbin/httpd
46307  ??  D      0:29.15 /usr/local/sbin/httpd
46398  ??  D      0:09.18 /usr/local/sbin/httpd
46617  ??  D      0:02.00 /usr/local/sbin/httpd
46767  ??  D      0:05.26 /usr/local/sbin/httpd
46771  ??  D      0:02.07 /usr/local/sbin/httpd
46803  ??  D      0:30.51 /usr/local/sbin/httpd
46807  ??  D      0:01.34 /usr/local/sbin/httpd
46889  ??  D      0:01.35 /usr/local/sbin/httpd
47659  ??  D      0:01.54 /usr/local/sbin/httpd
47711  ??  D      0:00.92 /usr/local/sbin/httpd
47849  ??  D      0:01.00 /usr/local/sbin/httpd
47877  ??  D      0:00.40 /usr/local/sbin/httpd
-------------------------------------------------------------------
#dmesg
All buffers synced.
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-RELEASE-p5 #0: Mon Nov 10 19:49:07 MSK 2008
    root at i.x00.net:/usr/obj/usr/src/sys/KERNEL-4BSD
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz (2400.10-MHz K8-class
CPU)
  Origin = "GenuineIntel"  Id = 0x6fb  Stepping = 11

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0xe3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  Cores per package: 4
usable memory = 4283449344 (4085 MB)
avail memory  = 4134977536 (3943 MB)
ACPI APIC Table: <022108 APIC2247>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
hptrr: HPT RocketRAID controller driver v1.1 (Nov 10 2008 19:48:39)
acpi0: <022108 RSDT2247> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, eff00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 928092806000928
device_attach: est0 attach returned 6
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 928092806000928
device_attach: est1 attach returned 6
p4tcc1: <CPU Frequency Thermal Control> on cpu1
cpu2: <ACPI CPU> on acpi0
est2: <Enhanced SpeedStep Frequency Control> on cpu2
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 928092806000928
device_attach: est2 attach returned 6
p4tcc2: <CPU Frequency Thermal Control> on cpu2
cpu3: <ACPI CPU> on acpi0
est3: <Enhanced SpeedStep Frequency Control> on cpu3
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 928092806000928
device_attach: est3 attach returned 6
p4tcc3: <CPU Frequency Thermal Control> on cpu3
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci5: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci3: <ACPI PCI bus> on pcib3
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x4201> mem
0xfeaf0000-0xfeafffff irq 16 at device 0.0 on pci3
miibus0: <MII bus> on bge0
brgphy0: <BCM5750 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
bge0: Ethernet address: 00:1f:c6:48:c8:9b
bge0: [ITHREAD]
pcib4: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci4: <ACPI PCI bus> on pcib4
bge1: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x4201> mem
0xfebf0000-0xfebfffff irq 17 at device 0.0 on pci4
miibus1: <MII bus> on bge1
brgphy1: <BCM5750 10/100/1000baseTX PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
bge1: Ethernet address: 00:1f:c6:48:ca:b7
bge1: [ITHREAD]
pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib5
vgapci0: <VGA-compatible display> port 0xec00-0xec7f mem
0xf8000000-0xfbffffff,0xfe9c0000-0xfe9fffff at device 4.0 on pci1
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH7 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
atapci1: <Intel ICH7 SATA300 controller> port
0xdc00-0xdc07,0xd880-0xd883,0xd800-0xd807,0xd480-0xd483,0xd400-0xd40f mem
0xfe8ffc00-0xfe8fffff irq 19 at device 31.2 on pci0
atapci1: [ITHREAD]
atapci1: AHCI called from vendor specific driver
atapci1: AHCI Version 01.10 controller with 4 ports detected
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
ata4: <ATA channel 2> on atapci1
ata4: [ITHREAD]
ata5: <ATA channel 3> on atapci1
ata5: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 8250 or not responding
sio0: [FILTER]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding disabled,
default to accept, logging limited to 5 packets/entry by default
hptrr: no controller detected.
acd0: DVDR <MATSHITADVD-RAM UJ-860S/1.00> at ata0-master UDMA33
ad4: 476940MB <Seagate ST3500320NS SN05> at ata2-master SATA300
ad6: 476940MB <Seagate ST3500320NS SN05> at ata3-master SATA300
ad10: 476940MB <Seagate ST3500320NS SN05> at ata5-master SATA300
ar0: 476937MB <Intel MatrixRAID RAID1> status: READY
ar0: disk0 READY (master) using ad4 at ata2-master
ar0: disk1 READY (mirror) using ad6 at ata3-master
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Trying to mount root from ufs:/dev/ar0s1a
bge0: link state changed to UP


-------------------------------------------------------------------
 cat ./KERNEL-4BSD | grep -v "^#"

cpu             HAMMER
ident           KERNEL-4BSD


options         SCHED_4BSD              # 4BSD scheduler
options         PREEMPTION              # Enable kernel thread preemption
options         INET                    # InterNETworking
options         INET6                   # IPv6 communications protocols
options         SCTP                    # Stream Control Transmission
Protocol
options         FFS                     # Berkeley Fast Filesystem
options         SOFTUPDATES             # Enable FFS soft updates support
options         UFS_ACL                 # Support for access control lists
options         UFS_DIRHASH             # Improve performance on big
directories
options         UFS_GJOURNAL            # Enable gjournal-based UFS
journaling
options         MD_ROOT                 # MD is a potential root device
options         MSDOSFS                 # MSDOS Filesystem
options         CD9660                  # ISO 9660 Filesystem
options         PROCFS                  # Process filesystem (requires
PSEUDOFS)
options         PSEUDOFS                # Pseudo-filesystem framework
options         GEOM_PART_GPT           # GUID Partition Tables.
options         GEOM_LABEL              # Provides labelization
options         COMPAT_43TTY            # BSD 4.3 TTY compat [KEEP THIS!]
options         COMPAT_IA32             # Compatible with i386 binaries
options         COMPAT_FREEBSD4         # Compatible with FreeBSD4
options         COMPAT_FREEBSD5         # Compatible with FreeBSD5
options         COMPAT_FREEBSD6         # Compatible with FreeBSD6
options         SCSI_DELAY=5000         # Delay (in ms) before probing SCSI
options         KTRACE                  # ktrace(1) support
options         SYSVSHM                 # SYSV-style shared memory
options         SYSVMSG                 # SYSV-style message queues
options         SYSVSEM                 # SYSV-style semaphores
options         _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time
extensions
options         KBD_INSTALL_CDEV        # install a CDEV entry in /dev
options         ADAPTIVE_GIANT          # Giant mutex is adaptive.
options         STOP_NMI                # Stop CPUS using NMI instead of IPI
options         AUDIT                   # Security event auditing


options         SMP                     # Symmetric MultiProcessor Kernel

options         IPFIREWALL
options         IPFIREWALL_VERBOSE
options         IPFIREWALL_VERBOSE_LIMIT=5
options         IPFIREWALL_DEFAULT_TO_ACCEPT

options         QUOTA

device          cpufreq

device          acpi
device          pci

device          fdc

device          ata
device          atadisk         # ATA disk drives
device          ataraid         # ATA RAID drives
device          atapicd         # ATAPI CDROM drives
options         ATA_STATIC_ID   # Static device numbering

device          ahc             # AHA2940 and onboard AIC7xxx devices
options         AHC_REG_PRETTY_PRINT    # Print register bitfields in debug
                                        # output.  Adds ~128k to driver.
device          ahd             # AHA39320/29320 and onboard AIC79xx devices
options         AHD_REG_PRETTY_PRINT    # Print register bitfields in debug
                                        # output.  Adds ~215k to driver.
device          amd             # AMD 53C974 (Tekram DC-390(T))
device          hptiop          # Highpoint RocketRaid 3xxx series
device          isp             # Qlogic family
device          mpt             # LSI-Logic MPT-Fusion
device          sym             # NCR/Symbios Logic (newer chipsets + those
of `ncr')
device          trm             # Tekram DC395U/UW/F DC315U adapters

device          adv             # Advansys SCSI adapters
device          adw             # Advansys wide SCSI adapters
device          aic             # Adaptec 15[012]x SCSI adapters,
AIC-6[23]60.
device          bt              # Buslogic/Mylex MultiMaster SCSI adapters


device          scbus           # SCSI bus (required for SCSI)
device          ch              # SCSI media changers
device          da              # Direct Access (disks)
device          sa              # Sequential Access (tape etc)
device          cd              # CD
device          pass            # Passthrough device (direct SCSI access)
device          ses             # SCSI Environmental Services (and SAF-TE)

device          amr             # AMI MegaRAID
device          arcmsr          # Areca SATA II RAID
device          ciss            # Compaq Smart RAID 5*
device          dpt             # DPT Smartcache III, IV - See NOTES for
options
device          hptmv           # Highpoint RocketRAID 182x
device          hptrr           # Highpoint RocketRAID 17xx, 22xx, 23xx,
25xx
device          iir             # Intel Integrated RAID
device          ips             # IBM (Adaptec) ServeRAID
device          mly             # Mylex AcceleRAID/eXtremeRAID
device          twa             # 3ware 9000 series PATA/SATA RAID

device          aac             # Adaptec FSA RAID
device          aacp            # SCSI passthrough for aac (requires CAM)
device          ida             # Compaq Smart RAID
device          mfi             # LSI MegaRAID SAS
device          mlx             # Mylex DAC960 family
device          twe             # 3ware ATA RAID

device          atkbdc          # AT keyboard controller
device          atkbd           # AT keyboard
device          psm             # PS/2 mouse

device          kbdmux          # keyboard multiplexer

device          vga             # VGA video card driver

device          splash          # Splash screen and screen saver support

device          sc

device          agp             # support several AGP chipsets


device          sio             # 8250, 16[45]50 based serial ports
device          uart            # Generic UART driver



device          de              # DEC/Intel DC21x4x (``Tulip'')
device          em              # Intel PRO/1000 adapter Gigabit Ethernet
Card
device          ixgb            # Intel PRO/10GbE Ethernet Card
device          le              # AMD Am7900 LANCE and Am79C9xx PCnet
device          txp             # 3Com 3cR990 (``Typhoon'')
device          vx              # 3Com 3c590, 3c595 (``Vortex'')

device          miibus          # MII bus support
device          bce             # Broadcom BCM5706/BCM5708 Gigabit Ethernet
device          bfe             # Broadcom BCM440x 10/100 Ethernet
device          bge             # Broadcom BCM570xx Gigabit Ethernet
device          dc              # DEC/Intel 21143 and various workalikes
device          fxp             # Intel EtherExpress PRO/100B (82557, 82558)
device          lge             # Level 1 LXT1001 gigabit Ethernet
device          msk             # Marvell/SysKonnect Yukon II Gigabit
Ethernet
device          nfe             # nVidia nForce MCP on-board Ethernet
device          nge             # NatSemi DP83820 gigabit Ethernet
device          pcn             # AMD Am79C97x PCI 10/100 (precedence over
'le')
device          re              # RealTek 8139C+/8169/8169S/8110S
device          rl              # RealTek 8129/8139
device          sf              # Adaptec AIC-6915 (``Starfire'')
device          sis             # Silicon Integrated Systems SiS 900/SiS
7016
device          sk              # SysKonnect SK-984x & SK-982x gigabit
Ethernet
device          ste             # Sundance ST201 (D-Link DFE-550TX)
device          ti              # Alteon Networks Tigon I/II gigabit
Ethernet
device          tl              # Texas Instruments ThunderLAN
device          tx              # SMC EtherPower II (83c170 ``EPIC'')
device          vge             # VIA VT612x gigabit Ethernet
device          vr              # VIA Rhine, Rhine II
device          wb              # Winbond W89C840F
device          xl              # 3Com 3c90x (``Boomerang'', ``Cyclone'')



device          loop            # Network loopback
device          random          # Entropy device
device          ether           # Ethernet support
device          sl              # Kernel SLIP
device          ppp             # Kernel PPP
device          tun             # Packet tunnel.
device          pty             # Pseudo-ttys (telnet etc)
device          md              # Memory "disks"
device          gif             # IPv6 and IPv4 tunneling
device          faith           # IPv6-to-IPv4 relaying (translation)
device          firmware        # firmware assist module

device          bpf             # Berkeley packet filter

device          uhci            # UHCI PCI->USB interface
device          ohci            # OHCI PCI->USB interface
device          ehci            # EHCI PCI->USB interface (USB 2.0)
device          usb             # USB Bus (required)
device          ugen            # Generic
device          uhid            # "Human Interface Devices"
device          ukbd            # Keyboard
device          umass           # Disks/Mass storage - Requires scbus and da
device          ums             # Mouse


-- 
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material.  Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited.   If you received
this in error, please contact the sender and delete the material from any
computer.


More information about the freebsd-stable mailing list