Processes in block state in vmstat.
Nikolay Pavlov
quetzal at zone3000.net
Mon Jul 10 09:55:44 UTC 2006
On Friday, 7 July 2006 at 16:40:26 +0300, Nikolay Pavlov wrote:
> Hi, folks.
> I want to add some additional information about problem related
> processes in block state. I see it on my server Dual CPU: Intel(R)
> Xeon(TM) CPU 2.40GHz (2399.33-MHz 686-class CPU) with amrd device.
It seems my issue could be related this PR
http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/94139
i have submitted follow-up report with backtraces.
>
> FreeBSD 6.1-RELEASE-p1
>
> I see some httpd daemons in D state:
>
> procs memory page disk faults cpu
> r b w avm fre flt re pi po fr sr am0 in sy cs us sy
> id
> 0 527 0 2173832 94028 49 0 0 0 2722 2658 0 9219 4140 17350 1 7 92
> 0 528 0 2172836 93636 66 0 0 0 218 0 6 504 512 1221 0 2 98
> 0 523 0 2171828 93360 56 0 0 0 188 0 6 551 298 1286 0 2 98
> 0 523 0 2171972 92612 29 0 0 0 189 0 6 520 395 1250 0 2 98
> 30 494 0 2170976 91836 92 0 0 0 333 0 11 448 250 970 0 2 98
> 0 518 0 2171156 90580 43 1 0 0 314 0 10 640 758 1676 0 3 97
> 0 510 0 2170244 89452 61 0 0 0 402 0 14 615 644 1667 0 3 97
> 1 512 0 2170420 89292 1 0 0 0 0 0 0 493 307 1133 0 2 98
> 0 517 0 2169560 86868 110 0 0 0 735 0 27 623 852 1584 0 3 96
> 0 514 0 2168552 87176 53 0 0 0 198 0 0 487 200 1078 0 1 99
> 0 517 0 2178752 84040 1523 0 0 0 1261 0 0 512 2660 1320 2 3 96
>
> vmstat -s
>
> 3129790854 cpu context switches
> 1663101011 device interrupts
> 5504551 software interrupts
> 9321603 traps
> 746764672 system calls
> 54 kernel threads created
> 66887 fork() calls
> 9669 vfork() calls
> 0 rfork() calls
> 4784 swap pager pageins
> 5635 swap pager pages paged in
> 1902 swap pager pageouts
> 3590 swap pager pages paged out
> 24668 vnode pager pageins
> 64050 vnode pager pages paged in
> 0 vnode pager pageouts
> 0 vnode pager pages paged out
> 28140 page daemon wakeups
> 479450699 pages examined by the page daemon
> 75270 pages reactivated
> 2461680 copy-on-write faults
> 3805 copy-on-write optimized faults
> 4531914 zero fill pages zeroed
> 2732630 zero fill pages prezeroed
> 18095 intransit blocking page faults
> 8922821 total VM faults taken
> 0 pages affected by kernel thread creation
> 8018826 pages affected by fork()
> 1107169 pages affected by vfork()
> 0 pages affected by rfork()
> 491039188 pages freed
> 393 pages freed by daemon
> 5744590 pages freed by exiting processes
> 201969 pages active
> 184930 pages inactive
> 17190 pages in VM cache
> 107624 pages wired down
> 750 pages free
> 4096 bytes per page
> 30979017 total name lookups
> cache hits (82% pos + 1% neg) system 1% per-directory
> deletions 0%, falsehits 0%, toolong 0%
>
>
> vmstat -z
>
> ITEM SIZE LIMIT USED FREE REQUESTS
>
> UMA Kegs: 140, 0, 83, 13, 83
> UMA Zones: 480, 0, 83, 5, 83
> UMA Slabs: 64, 0, 2537, 177, 225611
> UMA RCntSlabs: 104, 0, 32769, 13, 2949442
> UMA Hash: 128, 0, 3, 27, 6
> 16 Bucket: 76, 0, 27, 23, 73
> 32 Bucket: 140, 0, 23, 33, 64
> 64 Bucket: 268, 0, 17, 39, 100
> 128 Bucket: 524, 0, 223, 71, 1036
> VM OBJECT: 132, 0, 30655, 2318, 1833813
> MAP: 192, 0, 7, 33, 7
> KMAP ENTRY: 68, 65520, 285, 11251, 78210624
> MAP ENTRY: 68, 0, 215821, 8011, 3882740
> PV ENTRY: 24, 2155135, 1391603, 119877, 42095182
> DP fakepg: 72, 0, 0, 53, 6
> mt_zone: 1024, 0, 174, 126, 174
> 16: 16, 0, 2158, 684, 578060
> 32: 32, 0, 3343, 386, 87211
> 64: 64, 0, 7323, 819, 1449967
> 128: 128, 0, 3507, 333, 519528
> 256: 256, 0, 6226, 509, 181376
> 512: 512, 0, 64, 96, 77544
> 1024: 1024, 0, 55, 73, 38283
> 2048: 2048, 0, 141, 27, 25379
> 4096: 4096, 0, 1990, 135, 84846
> Files: 72, 0, 2924, 1475, 1665330
> PROC: 524, 0, 1936, 220, 76612
> THREAD: 372, 0, 2156, 14, 2156
> KSEGRP: 88, 0, 2156, 84, 2156
> UPCALL: 44, 0, 0, 0, 0
> VMSPACE: 300, 0, 1886, 337, 76512
> mbuf_packet: 256, 0, 66135, 0, 1979970489
> mbuf: 256, 0, 7, 1253, 3511131461
> mbuf_cluster: 2048, 65536, 65538, 0, 1966671403
> mbuf_jumbo_pagesize: 4096, 0, 0, 0, 0
> mbuf_jumbo_9k: 9216, 0, 0, 0, 0
> mbuf_jumbo_16k: 16384, 0, 0, 0, 0
> ACL UMA zone: 388, 0, 0, 0, 0
> g_bio: 132, 0, 0, 87, 79264695
> ata_request: 204, 0, 0, 0, 0
> ata_composite: 196, 0, 0, 0, 0
> VNODE: 272, 0, 27508, 7072, 4868108
> VNODEPOLL: 76, 0, 1, 49, 1
> S VFS Cache: 68, 0, 27595, 2869, 4582987
> L VFS Cache: 291, 0, 1708, 203, 314868
> NAMEI: 1024, 0, 3, 129, 12912949
> DIRHASH: 1024, 0, 495, 41, 4186
> NFSMOUNT: 480, 0, 2, 14, 3
> NFSNODE: 460, 0, 13, 27, 1525
> PIPE: 408, 0, 28, 35, 31594
> KNOTE: 68, 0, 0, 168, 5762
> socket: 356, 16434, 2327, 1699, 277319
> unpcb: 140, 65548, 42, 70, 12935
> ipq: 32, 2147, 0, 0, 2235
> udpcb: 180, 16434, 5, 127, 5531
> inpcb: 180, 16434, 2259, 1393, 258840
> tcpcb: 464, 16424, 2258, 1454, 258840
> tcptw: 48, 3354, 1, 389, 38232
> syncache: 100, 15366, 17, 334, 322283
> hostcache: 76, 15400, 246, 304, 6938
> tcpreass: 20, 4225, 1, 168, 4936
> sackhole: 20, 0, 8, 499, 23975962
> ripcb: 180, 16434, 0, 44, 11
> rtentry: 132, 0, 5, 53, 54
> pfsrctrpl: 100, 0, 0, 0, 0
> pfrulepl: 604, 0, 0, 0, 0
> pfstatepl: 260, 10005, 0, 0, 0
> pfaltqpl: 128, 0, 0, 0, 0
> pfpooladdrpl: 68, 0, 0, 0, 0
> pfrktable: 1240, 0, 0, 0, 0
> pfrkentry: 156, 0, 0, 0, 0
> pfrkentry2: 156, 0, 0, 0, 0
> pffrent: 16, 5075, 0, 0, 0
> pffrag: 48, 0, 0, 0, 0
> pffrcache: 48, 10062, 0, 0, 0
> pffrcent: 12, 50141, 0, 0, 0
> pfstatescrub: 28, 0, 0, 0, 0
> pfiaddrpl: 92, 0, 0, 0, 0
> pfospfen: 108, 0, 0, 0, 0
> pfosfp: 28, 0, 0, 0, 0
> IPFW dynamic rule zone: 108, 0, 0, 0, 0
> SWAPMETA: 276, 121576, 36, 20, 1613
> FFS inode: 132, 0, 27459, 1280, 4866112
> FFS1 dinode: 128, 0, 0, 0, 0
> FFS2 dinode: 256, 0, 27459, 1206, 4866112
>
> Intresting thing that everytime i use "vmstat 1" problem gone
> and i save my uptime.
>
> Server is still under load and i see only one process in D state
>
> 85815 ?? D 0:00.00 /usr/local/sbin/megarc -ldInfo -LAll -a0
>
> Also i have found that it's much easy to trigger this state for httpd
> when i use low (default) value for MaxSpareServers (I have about
> 300 ~ 500 connections all the time)
>
>
> --
> =========================================================================
> = Best regards, Nikolay Pavlov. <<<------------------------------------ =
> =========================================================================
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
--
=========================================================================
= Best regards, Nikolay Pavlov. <<<------------------------------------ =
=========================================================================
More information about the freebsd-stable
mailing list