re0 + high load CPU, some tests
Коньков Евгений
kes-kes at yandex.ru
Fri Dec 30 19:27:52 UTC 2011
if this is interesting for anyone.
looks strange:
irq256 - WAIT and
ng_queue - sleep
are they wait each other?
last pid: 70764; load averages: 69.14, 48.87, 26.25 up 9+03:32:40 21:01:21
197 processes: 101 running, 82 sleeping, 14 waiting
CPU: 14.2% user, 0.0% nice, 44.7% system, 41.2% interrupt, 0.0% idle
Mem: 192M Active, 343M Inact, 195M Wired, 112M Buf, 1259M Free
Swap: 512M Total, 512M Free
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND
12 root -76 - 0K 112K WAIT 19.2H 52.00% {irq256: re0}
13 root 0 - 0K 8K sleep 609:11 37.06% ng_queue
69225 root 38 0 13420K 5748K select 2:49 2.10% snmpd
70764 root 41 -10 11184K 3984K RUN 0:00 2.05% perl5.10.1
7749 bind 37 0 62980K 49004K uwait 26:40 0.73% {named}
70762 root 36 0 9952K 2168K RUN 0:00 0.10% top
3772 root 36 -10 103M 87944K wait 12:06 0.05% {mpd5}
3772 root 36 -10 103M 87944K RUN 0:00 0.05% {mpd5}
# vmstat -i
interrupt total rate
irq14: ata0 1430344 1
irq20: hpet0 3255183935 4118
irq23: uhci0 ehci0 85 0
irq256: re0 2672832531 3381
Total 5929446895 7501
1 users Load 79.81 59.43 33.65 Dec 30 21:03
Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER
Tot Share Tot Share Free in out in out
Act 283856 6264 721084 7948 1289560 count
All 384748 8944 2879988 18292 pages
Proc: Interrupts
r p d s w Csw Trp Sys Int Sof Flt 54 cow 15828 total
102 33 50k 590 4377 15k 3991 242 116 zfod 3 ata0 14
9 ozfod 4120 hpet0 20
39.8%Sys 50.1%Intr 10.1%User 0.0%Nice 0.0%Idle 7%ozfod uhci0 ehci
| | | | | | | | | | | daefr 11705 re0 256
====================+++++++++++++++++++++++++>>>>> 34 prcfr
33 dtbuf 206 totfr
Namei Name-cache Dir-cache 111725 desvn react
Calls hits % hits % 64006 numvn pdwak
2151 2151 100 27893 frevn pdpgs
intrn
Disks ad1 da0 pass0 199448 wire
KB/t 14.75 0.00 0.00 208820 act
tps 3 0 0 338932 inact
MB/s 0.04 0.00 0.00 cache
%busy 1 0 0 1289560 free
# uname -a
FreeBSD bor 9.0-CURRENT FreeBSD 9.0-CURRENT #1: Wed Feb 23 09:39:22 UTC 2011 @:/usr/obj/usr/src/sys/KES_KERN_v9 i386
# vmstat -z
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 128, 0, 92, 28, 92, 0, 0
UMA Zones: 224, 0, 92, 10, 92, 0, 0
UMA Slabs: 284, 0, 1267, 21, 13143, 0, 0
UMA RCntSlabs: 544, 0, 727, 1, 727, 0, 0
UMA Hash: 128, 0, 2, 28, 3, 0, 0
16 Bucket: 76, 0, 38, 12, 38, 0, 0
32 Bucket: 140, 0, 53, 3, 54, 0, 0
64 Bucket: 268, 0, 89, 9, 93, 90, 0
128 Bucket: 524, 0, 861, 0, 6885, 572, 0
VM OBJECT: 136, 0, 36908, 560, 8428863, 0, 0
MAP: 140, 0, 7, 49, 7, 0, 0
KMAP ENTRY: 72, 57505, 31, 234, 105121, 0, 0
MAP ENTRY: 72, 0, 1085, 982,19434967, 0, 0
DP fakepg: 72, 0, 0, 0, 0, 0, 0
SG fakepg: 72, 0, 0, 0, 0, 0, 0
mt_zone: 2060, 0, 268, 11, 268, 0, 0
16: 16, 0, 4707, 571,57886292, 0, 0
32: 32, 0, 11879, 777,706645306, 0, 0
64: 64, 0, 6726, 767,33742858, 0, 0
128: 128, 0, 43378, 66482,61277757, 0, 0
256: 256, 0, 2808, 372,684799727, 0, 0
512: 512, 0, 652, 236, 1592927, 0, 0
1024: 1024, 0, 532, 40, 605103, 0, 0
2048: 2048, 0, 269, 201, 5976, 0, 0
4096: 4096, 0, 156, 27, 608758, 0, 0
Files: 56, 0, 193, 276,48647341, 0, 0
TURNSTILE: 72, 0, 261, 39, 261, 0, 0
umtx pi: 52, 0, 0, 0, 0, 0, 0
MAC labels: 20, 0, 0, 0, 0, 0, 0
PROC: 688, 0, 60, 30, 306557, 0, 0
THREAD: 728, 0, 136, 124, 64332, 0, 0
SLEEPQUEUE: 44, 0, 261, 93, 261, 0, 0
VMSPACE: 232, 0, 40, 45, 306544, 0, 0
cpuset: 40, 0, 2, 182, 2, 0, 0
audit_record: 816, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 0, 259, 1149,1685234486, 0, 0
mbuf: 256, 0, 5, 642,3090339409, 0, 0
mbuf_cluster: 2048, 25600, 1408, 6, 1408, 0, 0
mbuf_jumbo_page: 4096, 12800, 0, 20, 153798, 0, 0
mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0
NetGraph items: 36, 4130, 3, 174,540902149, 0, 0
NetGraph data items: 36, 531, 0, 531,1277775559, 0, 0
g_bio: 140, 0, 0, 3444, 5725546, 0, 0
ttyinq: 152, 0, 300, 12, 975, 0, 0
ttyoutq: 256, 0, 157, 8, 508, 0, 0
ata_request: 204, 0, 0, 855, 1431305, 0, 0
ata_composite: 180, 0, 0, 0, 0, 0, 0
VNODE: 272, 0, 64018, 46, 1519139, 0, 0
VNODEPOLL: 60, 0, 1, 125, 1, 0, 0
S VFS Cache: 72, 0, 66679, 154, 1573251, 0, 0
L VFS Cache: 292, 0, 595, 198, 20943, 0, 0
NAMEI: 1024, 0, 0, 12,38878226, 0, 0
NFSMOUNT: 532, 0, 0, 0, 0, 0, 0
NFSNODE: 460, 0, 0, 0, 0, 0, 0
DIRHASH: 1024, 0, 482, 14, 482, 0, 0
pipe: 392, 0, 11, 29, 217359, 0, 0
Mountpoints: 648, 0, 6, 6, 6, 0, 0
ksiginfo: 80, 0, 80, 976, 41491, 0, 0
itimer: 220, 0, 0, 0, 0, 0, 0
KNOTE: 72, 0, 8, 151,13777392, 0, 0
socket: 416, 25605, 92, 304,24848447, 0, 0
unpcb: 172, 25622, 27, 42, 523052, 0, 0
ipq: 32, 904, 0, 226, 1272, 0, 0
udp_inpcb: 220, 25614, 6, 138,24157973, 0, 0
udpcb: 8, 25781, 6, 400,24157973, 0, 0
tcp_inpcb: 220, 25614, 37, 935, 134055, 0, 0
tcpcb: 692, 25600, 35, 150, 134055, 0, 0
tcptw: 52, 5184, 2, 1006, 21168, 0, 0
syncache: 112, 15365, 0, 70, 87664, 0, 0
hostcache: 76, 15400, 148, 102, 1559, 0, 0
tcpreass: 20, 1690, 0, 169, 6987, 0, 0
sackhole: 20, 0, 0, 169, 351, 0, 0
sctp_ep: 880, 25600, 0, 0, 0, 0, 0
sctp_asoc: 1524, 40000, 0, 0, 0, 0, 0
sctp_laddr: 24, 80040, 0, 145, 45, 0, 0
sctp_raddr: 464, 80000, 0, 0, 0, 0, 0
sctp_chunk: 96, 400000, 0, 0, 0, 0, 0
sctp_readq: 76, 400000, 0, 0, 0, 0, 0
sctp_stream_msg_out: 72, 400044, 0, 0, 0, 0, 0
sctp_asconf: 24, 400055, 0, 0, 0, 0, 0
sctp_asconf_ack: 24, 400055, 0, 0, 0, 0, 0
ripcb: 220, 25614, 15, 129, 18472, 0, 0
rtentry: 108, 0, 9021, 231, 44539, 0, 0
IPFW dynamic rule: 108, 0, 0, 0, 0, 0, 0
divcb: 220, 25614, 0, 0, 0, 0, 0
selfd: 28, 0, 276, 232,434047205, 0, 0
ip4flow: 40, 25668, 1332, 8972,16391139, 0, 0
ip6flow: 64, 25636, 0, 174, 20, 0, 0
SWAPMETA: 276, 121576, 0, 0, 0, 0, 0
FFS inode: 116, 0, 63979, 74, 1519019, 0, 0
FFS1 dinode: 128, 0, 0, 0, 0, 0, 0
FFS2 dinode: 256, 0, 63979, 71, 1519019, 0, 0
# sysctl -a net.isr
net.isr.numthreads: 1
net.isr.maxprot: 16
net.isr.defaultqlimit: 256
net.isr.maxqlimit: 10240
net.isr.bindthreads: 0
net.isr.maxthreads: 1
net.isr.direct: 1
net.isr.direct_force: 1
#sysctl -a | grep smp | less
kern.timecounter.smp_tsc: 0
kern.smp.forward_signal_enabled: 1
kern.smp.topology: 0
kern.smp.cpus: 1
kern.smp.disabled: 3
kern.smp.active: 0
kern.smp.maxcpus: 32
kern.smp.maxid: 3
this system can process about 19000 interrupts from re0, just disable
other CPUs to fast achive overload.
renice -20 mpd5
wait 10min
/usr/local/etc/mpd5 restart
after 1mins ago mpd5 is restarted and everithing is stabilised to normal work
PS. does "41.2% interrupt" in top output show really CPU load from 're'
interrupts or in this value are some internal kernel interupts too?
suggestions:
when CPU can not devastate interrupts buffer re0 starts to flood by interrupts
mpd5 lock interupts system when CPU is 0% idle;
tuning on this machine:
kern.ipc.maxsockbuf=8388608
net.graph.recvspace=524288
net.inet.ip.fw.one_pass=0
net.inet.ip.dummynet.io_fast=1
kern.smp.disabled=3
--
С уважением,
Коньков mailto:kes-kes at yandex.ru
More information about the freebsd-questions
mailing list