Problem with process parallelization
Dmitriy
_pppp at mail.ru
Mon Sep 8 12:41:35 UTC 2008
> Hello, freebsd-net mailing list.
>
> We have server such configurtion:
> - 2 quadcore AMD Opteron processors;
> - 4 GB RAM;
> - NIC Intel Pro/1000 PT, Dual Port Server Adapter.
>
> ###########################################################
>
> Problem:
>
> in some moments of time, at the growth of the network activity, one of
> the processors is fully loaded at 100%.
>
> ###########################################################
>
> Kernel configuration:
>
> FreeBSD atlantis.bg.net.ua 7.0-STABLE FreeBSD 7.0-STABLE #1: Tue Apr 1
> 15:06:30 EEST 2008
> root at atlantis.bg.net.ua:/usr/obj/usr/src/sys/ATLANTIS amd64
>
> /etc/sysctl.conf:
>
> net.inet.tcp.blackhole=2
> net.inet.udp.blackhole=1
> kern.ipc.somaxconn=16384
> net.inet.ip.fastforwarding=1
> net.inet.ip.maxfragpackets=2000
> net.inet.ip.intr_queue_maxlen=1000
> net.inet.ip.dummynet.hash_size=2048
> net.inet.tcp.recvspace=65536
> net.inet.udp.recvspace=65536
> net.inet.raw.recvspace=32768
> net.local.stream.recvspace=32768
> net.local.dgram.recvspace=32768
> net.local.stream.sendspace=32768
> net.inet.tcp.sendspace=65536
> net.inet.icmp.icmplim=500
> dev.em.0.rx_int_delay=500
> dev.em.0.tx_int_delay=500
> dev.em.0.rx_abs_int_delay=800
> dev.em.0.tx_abs_int_delay=800
> dev.em.1.rx_int_delay=500
> dev.em.1.tx_int_delay=500
> dev.em.1.rx_abs_int_delay=800
> dev.em.1.tx_abs_int_delay=800
> net.link.ether.inet.max_age=600
>
> /boot/loader.conf:
>
> hw.em.rxd=4096
> hw.em.txd=4096
>
> /etc/rc.firewall:
>
> 82 pipes like theese:
>
> pipe 387 ip from any to 193.227.x.x in recv vlan10
> pipe 388 ip from 193.227.x.x to any out xmit vlan10
>
>
> #########################################
> Kernel:
>
>
> cpu HAMMER
> ident ATLANTIS
>
> # To statically compile in device wiring instead of /boot/device.hints
> #hints "GENERIC.hints" # Default places to look for
> devices.
>
> makeoptions DEBUG=-g # Build kernel with gdb(1) debug
> symbols
>
> options SCHED_ULE # 4BSD scheduler
> options PREEMPTION # Enable kernel thread preemption
> options INET # InterNETworking
> #options SCTP # Stream Control Transmission
> Protocol
> options FFS # Berkeley Fast Filesystem options
> SOFTUPDATES # Enable FFS soft updates support options
> UFS_ACL # Support for access control lists options
> UFS_DIRHASH # Improve performance on big directories
> options PROCFS # Process filesystem (requires
> PSEUDOFS)
> options PSEUDOFS # Pseudo-filesystem framework
> options GEOM_PART_GPT # GUID Partition Tables.
> options GEOM_LABEL # Provides labelization
> options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP THIS!]
> options COMPAT_IA32 # Compatible with i386 binaries
> options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options
> COMPAT_FREEBSD5 # Compatible with FreeBSD5 options
> COMPAT_FREEBSD6 # Compatible with FreeBSD6 options KTRACE
> # ktrace(1) support
> options SYSVSHM # SYSV-style shared memory options
> SYSVMSG # SYSV-style message queues options
> SYSVSEM # SYSV-style semaphores
> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time
> extensions
> options KBD_INSTALL_CDEV # install a CDEV entry in /dev
> options ADAPTIVE_GIANT # Giant mutex is adaptive. options
> STOP_NMI # Stop CPUS using NMI instead of IPI
> options AUDIT # Security event auditing
>
> # Make an SMP-capable kernel by default
> options SMP # Symmetric MultiProcessor Kernel
>
> # Bus support.
> device acpi
> device pci
>
> # ATA and ATAPI devices
> device ata
>
> device atadisk # ATA disk drives
> options ATA_STATIC_ID # Static device numbering
>
> # RAID controllers
> device twe # 3ware ATA RAID
>
> # atkbdc0 controls both the keyboard and the PS/2 mouse
> device atkbdc # AT keyboard controller
> device atkbd # AT keyboard
>
> device vga # VGA video card driver
>
> device splash # Splash screen and screen saver support
>
> # syscons is the default console driver, resembling an SCO console device
> sc
>
> ### COM
> device sio
>
> # PCI Ethernet NICs.
> device em # Intel PRO/1000 adapter Gigabit Ethernet
> Card
>
> # PCI Ethernet NICs that use the common MII bus controller code.
> # NOTE: Be sure to keep the 'device miibus' line in order to use these
> NICs! device miibus # MII bus support
> device bge # Broadcom BCM570xx Gigabit Ethernet
> device fxp # Intel EtherExpress PRO/100B (82557,
> 82558)
>
> # Pseudo devices.
> device loop # Network loopback
> device random # Entropy device
> device ether # Ethernet support
> device pty # Pseudo-ttys (telnet etc)
> device vlan
>
> # The `bpf' device enables the Berkeley Packet Filter.
> # Be aware of the administrative consequences of enabling this!
> # Note that 'bpf' is required for DHCP.
> device bpf # Berkeley packet filter
>
> ## Custom options
> # NetGraph
> options NETGRAPH
> options NETGRAPH_ONE2MANY
> options NETGRAPH_NETFLOW
> options NETGRAPH_CISCO
> options NETGRAPH_ETHER
> options NETGRAPH_KSOCKET
> options NETGRAPH_SOCKET
> options NETGRAPH_TEE
>
> options IPFIREWALL
> options IPFIREWALL_VERBOSE
> options IPFIREWALL_FORWARD
> options IPFIREWALL_VERBOSE_LIMIT=1000
> options IPFIREWALL_DEFAULT_TO_ACCEPT
> options DUMMYNET
> options HZ=1000
> options DEVICE_POLLING
> #####################################################
>
> Interfaces:
> - em0
> - em1
> - bge0
> - bge1
> - vlan (61 virtual interfaces)
>
> #####################################################
> top -S
>
> last pid: 9673; load averages: 1.94, 1.75, 1.57
> up 0+19:17:21
> 19:45:01
> 77 processes: 11 running, 49 sleeping, 17 waiting
> CPU states: 0.0% user, 0.0% nice, 22.6% system, 0.3% interrupt, 77.0%
> idle Mem: 198M Active, 410M Inact, 455M Wired, 228K Cache, 214M Buf, 2874M
> Free Swap: 4096M Total, 4096M Free
>
> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
> 11 root 1 171 ki31 0K 16K CPU7 7 19.0H 100.00% idle:
> cpu7
> 16 root 1 171 ki31 0K 16K CPU2 2 18.9H 100.00% idle:
> cpu2
> 17 root 1 171 ki31 0K 16K RUN 1 18.8H 100.00% idle:
> cpu1
> 13 root 1 171 ki31 0K 16K CPU5 5 18.8H 100.00% idle:
> cpu5
> 18 root 1 171 ki31 0K 16K CPU0 0 916:13 100.00% idle:
> cpu0
> 12 root 1 171 ki31 0K 16K CPU6 6 18.8H 99.85% idle:
> cpu6
> 35 root 1 -68 - 0K 16K CPU4 4 466:17 96.00% em1
> taskq
> 34 root 1 -68 - 0K 16K CPU3 3 482:01 90.38% em0
> taskq
> 15 root 1 171 ki31 0K 16K RUN 3 655:20 13.38% idle:
> cpu3
> 14 root 1 171 ki31 0K 16K RUN 4 671:52 3.08% idle:
> cpu4
>
>
> ##############################################
> 19:45[p0]root at atlantis#~>netstat -w 1 -I em0
> input (em0) output
> packets errs bytes packets errs bytes colls
> 57381 0 36442155 68726 0 69126050 0
> 56817 0 37480502 67656 0 66053093 0
> 57847 0 39532712 68603 0 67037042 0
> 56908 0 37197022 68924 0 68660108 0
> 57107 0 37643382 68398 0 68113937 0
> 56847 0 35944754 68394 0 67896267 0
> 58754 0 39763361 68966 0 70029090 0
> 58343 0 38301796 69635 0 69948678 0
> ^C
> 19:46[p0]root at atlantis#~>netstat -w 1 -I em1
> input (em1) output
> packets errs bytes packets errs bytes colls
> 67944 0 68877031 55376 0 36252905 0
> 65943 0 66722222 54575 0 37710643 0
> 64639 0 67149621 53298 0 35423539 0
> 63988 0 65035759 51787 0 35402337 0
> 63849 0 65968513 50727 0 31683425 0
> 64301 0 66684912 50193 0 30917339 0
>
>
>
> ###################################################################
>
>
> How can we solve this problem and parallelize em1:taskq kernel processes
> between all 8 processors?
# sysctl net.isr.direct=0
would add one more kernel thread to handle your network traffic.
Regards, Dmitriy.
>
>
> --
> ISP BGNet
> 288-03-53
> 246-68-98
>
> Zin'kov Oleg
> System administrator
>
>
>
>
>
>
>
>
>
>
>
>
> --
> ISP BGNet
> 288-03-53
> 246-68-98
>
> Zin'kov Oleg
> System administrator
>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>
More information about the freebsd-net
mailing list