Problem with process parallelization
Zin'kov Oleg
noc at bg.net.ua
Sun Sep 7 12:11:48 UTC 2008
Hello, freebsd-net mailing list.
We have server such configurtion:
- 2 quadcore AMD Opteron processors;
- 4 GB RAM;
- NIC Intel Pro/1000 PT, Dual Port Server Adapter.
###########################################################
Problem:
in some moments of time, at the growth of the network activity, one of
the processors is fully loaded at 100%.
###########################################################
Kernel configuration:
FreeBSD atlantis.bg.net.ua 7.0-STABLE FreeBSD 7.0-STABLE #1: Tue Apr 1
15:06:30 EEST 2008
root at atlantis.bg.net.ua:/usr/obj/usr/src/sys/ATLANTIS amd64
/etc/sysctl.conf:
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
kern.ipc.somaxconn=16384
net.inet.ip.fastforwarding=1
net.inet.ip.maxfragpackets=2000
net.inet.ip.intr_queue_maxlen=1000
net.inet.ip.dummynet.hash_size=2048
net.inet.tcp.recvspace=65536
net.inet.udp.recvspace=65536
net.inet.raw.recvspace=32768
net.local.stream.recvspace=32768
net.local.dgram.recvspace=32768
net.local.stream.sendspace=32768
net.inet.tcp.sendspace=65536
net.inet.icmp.icmplim=500
dev.em.0.rx_int_delay=500
dev.em.0.tx_int_delay=500
dev.em.0.rx_abs_int_delay=800
dev.em.0.tx_abs_int_delay=800
dev.em.1.rx_int_delay=500
dev.em.1.tx_int_delay=500
dev.em.1.rx_abs_int_delay=800
dev.em.1.tx_abs_int_delay=800
net.link.ether.inet.max_age=600
/boot/loader.conf:
hw.em.rxd=4096
hw.em.txd=4096
/etc/rc.firewall:
82 pipes like theese:
pipe 387 ip from any to 193.227.x.x in recv vlan10
pipe 388 ip from 193.227.x.x to any out xmit vlan10
#########################################
Kernel:
cpu HAMMER
ident ATLANTIS
# To statically compile in device wiring instead of /boot/device.hints
#hints "GENERIC.hints" # Default places to look for
devices.
makeoptions DEBUG=-g # Build kernel with gdb(1) debug
symbols
options SCHED_ULE # 4BSD scheduler
options PREEMPTION # Enable kernel thread preemption
options INET # InterNETworking
#options SCTP # Stream Control Transmission
Protocol
options FFS # Berkeley Fast Filesystem options
SOFTUPDATES # Enable FFS soft updates support options
UFS_ACL # Support for access control lists options
UFS_DIRHASH # Improve performance on big directories
options PROCFS # Process filesystem (requires
PSEUDOFS)
options PSEUDOFS # Pseudo-filesystem framework
options GEOM_PART_GPT # GUID Partition Tables.
options GEOM_LABEL # Provides labelization
options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP THIS!]
options COMPAT_IA32 # Compatible with i386 binaries
options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options
COMPAT_FREEBSD5 # Compatible with FreeBSD5 options
COMPAT_FREEBSD6 # Compatible with FreeBSD6 options KTRACE
# ktrace(1) support
options SYSVSHM # SYSV-style shared memory options
SYSVMSG # SYSV-style message queues options
SYSVSEM # SYSV-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time
extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options ADAPTIVE_GIANT # Giant mutex is adaptive. options
STOP_NMI # Stop CPUS using NMI instead of IPI
options AUDIT # Security event auditing
# Make an SMP-capable kernel by default
options SMP # Symmetric MultiProcessor Kernel
# Bus support.
device acpi
device pci
# ATA and ATAPI devices
device ata
device atadisk # ATA disk drives
options ATA_STATIC_ID # Static device numbering
# RAID controllers
device twe # 3ware ATA RAID
# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device vga # VGA video card driver
device splash # Splash screen and screen saver support
# syscons is the default console driver, resembling an SCO console device
sc
### COM
device sio
# PCI Ethernet NICs.
device em # Intel PRO/1000 adapter Gigabit Ethernet
Card
# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these
NICs! device miibus # MII bus support
device bge # Broadcom BCM570xx Gigabit Ethernet
device fxp # Intel EtherExpress PRO/100B (82557,
82558)
# Pseudo devices.
device loop # Network loopback
device random # Entropy device
device ether # Ethernet support
device pty # Pseudo-ttys (telnet etc)
device vlan
# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device bpf # Berkeley packet filter
## Custom options
# NetGraph
options NETGRAPH
options NETGRAPH_ONE2MANY
options NETGRAPH_NETFLOW
options NETGRAPH_CISCO
options NETGRAPH_ETHER
options NETGRAPH_KSOCKET
options NETGRAPH_SOCKET
options NETGRAPH_TEE
options IPFIREWALL
options IPFIREWALL_VERBOSE
options IPFIREWALL_FORWARD
options IPFIREWALL_VERBOSE_LIMIT=1000
options IPFIREWALL_DEFAULT_TO_ACCEPT
options DUMMYNET
options HZ=1000
options DEVICE_POLLING
#####################################################
Interfaces:
- em0
- em1
- bge0
- bge1
- vlan (61 virtual interfaces)
#####################################################
top -S
last pid: 9673; load averages: 1.94, 1.75, 1.57
up 0+19:17:21
19:45:01
77 processes: 11 running, 49 sleeping, 17 waiting
CPU states: 0.0% user, 0.0% nice, 22.6% system, 0.3% interrupt, 77.0%
idle Mem: 198M Active, 410M Inact, 455M Wired, 228K Cache, 214M Buf, 2874M
Free Swap: 4096M Total, 4096M Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 1 171 ki31 0K 16K CPU7 7 19.0H 100.00% idle:
cpu7
16 root 1 171 ki31 0K 16K CPU2 2 18.9H 100.00% idle:
cpu2
17 root 1 171 ki31 0K 16K RUN 1 18.8H 100.00% idle:
cpu1
13 root 1 171 ki31 0K 16K CPU5 5 18.8H 100.00% idle:
cpu5
18 root 1 171 ki31 0K 16K CPU0 0 916:13 100.00% idle:
cpu0
12 root 1 171 ki31 0K 16K CPU6 6 18.8H 99.85% idle:
cpu6
35 root 1 -68 - 0K 16K CPU4 4 466:17 96.00% em1
taskq
34 root 1 -68 - 0K 16K CPU3 3 482:01 90.38% em0
taskq
15 root 1 171 ki31 0K 16K RUN 3 655:20 13.38% idle:
cpu3
14 root 1 171 ki31 0K 16K RUN 4 671:52 3.08% idle:
cpu4
##############################################
19:45[p0]root at atlantis#~>netstat -w 1 -I em0
input (em0) output
packets errs bytes packets errs bytes colls
57381 0 36442155 68726 0 69126050 0
56817 0 37480502 67656 0 66053093 0
57847 0 39532712 68603 0 67037042 0
56908 0 37197022 68924 0 68660108 0
57107 0 37643382 68398 0 68113937 0
56847 0 35944754 68394 0 67896267 0
58754 0 39763361 68966 0 70029090 0
58343 0 38301796 69635 0 69948678 0
^C
19:46[p0]root at atlantis#~>netstat -w 1 -I em1
input (em1) output
packets errs bytes packets errs bytes colls
67944 0 68877031 55376 0 36252905 0
65943 0 66722222 54575 0 37710643 0
64639 0 67149621 53298 0 35423539 0
63988 0 65035759 51787 0 35402337 0
63849 0 65968513 50727 0 31683425 0
64301 0 66684912 50193 0 30917339 0
###################################################################
How can we solve this problem and parallelize em1:taskq kernel processes
between all 8 processors?
--
ISP BGNet
288-03-53
246-68-98
Zin'kov Oleg
System administrator
--
ISP BGNet
288-03-53
246-68-98
Zin'kov Oleg
System administrator
More information about the freebsd-net
mailing list