Very strange kevent problem possibly to do with vinum
Kevin Day
toasty at dragondata.com
Fri Dec 10 06:16:31 PST 2004
On Dec 10, 2004, at 5:01 AM, Igor Sysoev wrote:
> On Wed, 8 Dec 2004, Kevin Day wrote:
>
>> I have a really really strange kevent problem(i think anyway) that has
>> really stumped me.
>
> What does "systat -vm" show on these machines ?
>
>
From one of the servers that is having problems:
1 users Load 0.35 0.22 0.18 Dec 10 08:12
Mem:KB REAL VIRTUAL VN PAGER SWAP
PAGER
Tot Share Tot Share Free in out in
out
Act 25908 2528 95248 4180 75624 count
All 2045316 4704 1717416 7796 pages
320 zfod
Interrupts
Proc:r p d s w Csw Trp Sys Int Sof Flt cow 4672
total
1 75 12864 76812224 8384 192 448 150848 wire
1: atkb
32104 act
6: fdc0
50.0%Sys 0.0%Intr 0.0%User 0.0%Nice 50.0%Idl 1792248 inact 128
8: rtc
| | | | | | | | | | 72628 cache
13: npx
========================= 2996 free
14: ata
daefr 2816
28: bge
Namei Name-cache Dir-cache prcfr 1600
29: bge
Calls hits % hits % react
30: ahc
52 52 100 pdwak
31: ahc
pdpgs 128
0: clk
Disks da0 da1 da2 da3 da4 pass0 pass1 intrn
KB/t 0.00 0.00 0.00 0.00 0.00 0.00 0.00 93456 buf
tps 0 0 0 0 0 0 0 7 dirtybuf
MB/s 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100000 desiredvnodes
% busy 0 0 0 0 0 0 0 90243 numvnodes
79704 freevnodes
(currently sending around 80mbps)
From one that is working correctly:
1 users Load 0.50 0.60 0.65 Dec 10 08:08
Mem:KB REAL VIRTUAL VN PAGER SWAP
PAGER
Tot Share Tot Share Free in out in
out
Act 38132 2628 66944 4364 92244 count
All 2047208 3628 3255500 6044 pages
Interrupts
Proc:r p d s w Csw Trp Sys Int Sof Flt cow 8414
total
1 24 39782 2 115720205 183 291140 wire
6: fdc0
53080 act 128
8: rtc
4.0%Sys 9.3%Intr 0.3%User 0.0%Nice 86.4%Idl 1614860 inact
13: npx
| | | | | | | | | | 89248 cache
stray 1
==+++++ 2996 free
14: ata
daefr 2083
28: bge
Namei Name-cache Dir-cache prcfr 6064
29: bge
Calls hits % hits % react 39
30: aac
186 186 100 pdwak 100
0: clk
zfod pdpgs
Disks aacd0 fd0 ofod intrn
KB/t 84.31 0.00 %slo-z 114464 buf
tps 28 0 578 tfree 18 dirtybuf
MB/s 2.31 0.00 134371 desiredvnodes
% busy 22 0 98167 numvnodes
86885 freevnodes
(currently sending around 250mbps)
A ktrace example of what I'm talking about with kevent on one of the
slow servers:
468 thttpd 0.000015 CALL
kevent(0x4,0x8090000,0,0x80fd000,0x2b57,0xbfbfe898)
468 thttpd 0.000160 RET kevent 1
468 thttpd 0.000037 CALL gettimeofday(0xbfbfe914,0)
468 thttpd 0.000025 RET gettimeofday 0
468 thttpd 0.000023 CALL
sendfile(0x52,0x13a,0x9df1000,0,0x2011c38,0,0xbfbfe878,0)
468 thttpd 0.000057 RET sendfile -1 errno 35 Resource
temporarily unavailable
468 thttpd 0.000015 CALL
kevent(0x4,0x8090000,0,0x80fd000,0x2b57,0xbfbfe898)
468 thttpd 0.000183 RET kevent 1
468 thttpd 0.000031 CALL gettimeofday(0xbfbfe914,0)
468 thttpd 0.000020 RET gettimeofday 0
468 thttpd 0.000035 CALL
sendfile(0xcb,0x6b,0xd9f000,0,0x28ae934,0,0xbfbfe878,0)
468 thttpd 0.000058 RET sendfile -1 errno 35 Resource
temporarily unavailable
468 thttpd 0.000018 CALL
kevent(0x4,0x8090000,0,0x80fd000,0x2b57,0xbfbfe898)
468 thttpd 0.000563 RET kevent 1
468 thttpd 0.000036 CALL gettimeofday(0xbfbfe914,0)
468 thttpd 0.000018 RET gettimeofday 0
468 thttpd 0.000009 CALL
sendfile(0x52,0x13a,0x9df2000,0,0x2010c38,0,0xbfbfe878,0)
468 thttpd 0.000088 RET sendfile -1 errno 35 Resource
temporarily unavailable
468 thttpd 0.000012 CALL
kevent(0x4,0x8090000,0,0x80fd000,0x2b57,0xbfbfe898)
468 thttpd 0.000018 RET kevent 1
468 thttpd 0.000020 CALL gettimeofday(0xbfbfe914,0)
468 thttpd 0.000015 RET gettimeofday 0
468 thttpd 0.000034 CALL
sendfile(0x52,0x13a,0x9df4000,0,0x200ec38,0,0xbfbfe878,0)
468 thttpd 0.000056 RET sendfile -1 errno 35 Resource
temporarily unavailable
468 thttpd 0.000016 CALL
kevent(0x4,0x8090000,0,0x80fd000,0x2b57,0xbfbfe898)
468 thttpd 0.001385 RET kevent 1
468 thttpd 0.000041 CALL gettimeofday(0xbfbfe914,0)
468 thttpd 0.000024 RET gettimeofday 0
468 thttpd 0.000024 CALL
sendfile(0x113,0x139,0x87df000,0,0x7208034,0,0xbfbfe878,0)
468 thttpd 0.000055 RET sendfile -1 errno 35 Resource
temporarily unavailable
468 thttpd 0.000017 CALL
kevent(0x4,0x8090000,0,0x80fd000,0x2b57,0xbfbfe898)
468 thttpd 0.004689 RET kevent 1
468 thttpd 0.000033 CALL gettimeofday(0xbfbfe914,0)
468 thttpd 0.000025 RET gettimeofday 0
468 thttpd 0.000024 CALL
sendfile(0x64,0x97,0x1e000,0,0x2e6d96,0,0xbfbfe878,0)
468 thttpd 0.000025 RET sendfile -1 errno 35 Resource
temporarily unavailable
Here's a ktrace on the fast server:
91919 thttpd 0.000022 CALL
kevent(0x4,0x80f7000,0x6,0x8377000,0x10000,0xbfbfe658)
91919 thttpd 0.000668 RET kevent 1241/0x4d9
91919 thttpd 0.000035 CALL gettimeofday(0xbfbfe6d4,0)
91919 thttpd 0.000008 RET gettimeofday 0
91919 thttpd 0.000008 CALL
sendfile(0x128,0x807,0xb5a1000,0,0x13254a,0,0xbfbfe638,0)
91919 thttpd 0.000300 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000013 CALL
sendfile(0x128,0x721,0xb2a6000,0,0x42d54a,0,0xbfbfe638,0)
91919 thttpd 0.000229 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000011 CALL
sendfile(0x128,0x210,0xa978000,0,0xd5b54a,0,0xbfbfe638,0)
91919 thttpd 0.000276 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000023 CALL
sendfile(0x128,0x773,0xa781000,0,0xf5254a,0,0xbfbfe638,0)
91919 thttpd 0.000241 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000058 CALL
sendfile(0x33,0x4ec,0xa30a000,0,0x3b3dca,0,0xbfbfe638,0)
91919 thttpd 0.000285 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000014 CALL
sendfile(0x1f3,0x3c3,0xb5a5000,0,0x148efc0,0,0xbfbfe638,0)
91919 thttpd 0.000274 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000019 CALL
sendfile(0x3e4,0x448,0xa1c5000,0,0x1c3dc38,0,0xbfbfe638,0)
91919 thttpd 0.000197 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000007 CALL
sendfile(0x128,0x523,0x9daa000,0,0x192954a,0,0xbfbfe638,0)
91919 thttpd 0.000264 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000018 CALL
sendfile(0x128,0x85e,0xa7e0000,0,0xef354a,0,0xbfbfe638,0)
91919 thttpd 0.000418 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000027 CALL
sendfile(0x33,0x7b7,0x9e80000,0,0x83ddca,0,0xbfbfe638,0)
91919 thttpd 0.000299 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000022 CALL
sendfile(0x128,0xfd,0xb348000,0,0x38b54a,0,0xbfbfe638,0)
91919 thttpd 0.000231 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000006 CALL
sendfile(0xd2,0x160,0x6681000,0,0x129a6d6,0,0xbfbfe638,0)
91919 thttpd 0.000231 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000010 CALL
sendfile(0x128,0x46f,0xa15f000,0,0x157454a,0,0xbfbfe638,0)
91919 thttpd 0.000223 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000007 CALL
sendfile(0x128,0x88,0xac4d000,0,0xa8654a,0,0xbfbfe638,0)
91919 thttpd 0.000226 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000006 CALL
sendfile(0x128,0x85c,0xa538000,0,0x119b54a,0,0xbfbfe638,0)
91919 thttpd 0.000197 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000007 CALL
sendfile(0x128,0x1ac,0x51b0000,0,0x652354a,0,0xbfbfe638,0)
91919 thttpd 0.000270 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000008 CALL
sendfile(0x128,0x856,0x996e000,0,0x1d6554a,0,0xbfbfe638,0)
91919 thttpd 0.000304 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000008 CALL
sendfile(0x128,0x7a8,0xa505000,0,0x11ce54a,0,0xbfbfe638,0)
91919 thttpd 0.000218 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000008 CALL
sendfile(0x33,0x7cd,0xa51b000,0,0x1a2dca,0,0xbfbfe638,0)
91919 thttpd 0.000292 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000024 CALL
sendfile(0x128,0x1e0,0x8b64000,0,0x2b6f54a,0,0xbfbfe638,0)
91919 thttpd 0.000211 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000006 CALL
sendfile(0x128,0x907,0x9933000,0,0x1da054a,0,0xbfbfe638,0)
91919 thttpd 0.000097 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000008 CALL
sendfile(0xd2,0x2db,0x6c6f000,0,0xcac6d6,0,0xbfbfe638,0)
91919 thttpd 0.000286 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000040 CALL
sendfile(0x128,0x845,0xafab000,0,0x72854a,0,0xbfbfe638,0)
91919 thttpd 0.000273 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000020 CALL
sendfile(0x128,0x873,0x9968000,0,0x1d6b54a,0,0xbfbfe638,0)
91919 thttpd 0.000136 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000006 CALL
sendfile(0xd2,0x2ba,0x5bae000,0,0x1d6d6d6,0,0xbfbfe638,0)
91919 thttpd 0.000230 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000006 CALL
sendfile(0x33,0x1df,0xa564000,0,0x159dca,0,0xbfbfe638,0)
91919 thttpd 0.000195 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000006 CALL
sendfile(0x128,0x13,0x6abf000,0,0x4c1454a,0,0xbfbfe638,0)
91919 thttpd 0.000258 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000015 CALL
sendfile(0x9,0x2ae,0x7ff1000,0,0x30bc204,0,0xbfbfe638,0)
91919 thttpd 0.000499 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000036 CALL
sendfile(0x128,0x2ce,0x4c4e000,0,0x6a8554a,0,0xbfbfe638,0)
91919 thttpd 0.000201 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000007 CALL
sendfile(0x33,0x27e,0x5df5000,0,0x48c8dca,0,0xbfbfe638,0)
91919 thttpd 0.000193 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000007 CALL
sendfile(0x128,0x22e,0x567b000,0,0x605854a,0,0xbfbfe638,0)
91919 thttpd 0.000235 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000007 CALL
sendfile(0x128,0x77,0x9199000,0,0x253a54a,0,0xbfbfe638,0)
91919 thttpd 0.000324 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000028 CALL
sendfile(0x33,0x4d3,0x98e2000,0,0xddbdca,0,0xbfbfe638,0)
91919 thttpd 0.000241 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000007 CALL
sendfile(0x128,0x17b,0x4e27000,0,0x68ac54a,0,0xbfbfe638,0)
91919 thttpd 0.000250 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000016 CALL
sendfile(0x128,0x25b,0x48ca000,0,0x6e0954a,0,0xbfbfe638,0)
91919 thttpd 0.000380 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000024 CALL
sendfile(0x128,0x6c1,0xb0eb000,0,0x5e854a,0,0xbfbfe638,0)
91919 thttpd 0.000466 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000050 CALL
sendfile(0x128,0x620,0xa517000,0,0x11bc54a,0,0xbfbfe638,0)
91919 thttpd 0.001469 RET sendfile -1 errno 35 Resource
temporarily unavailable
91919 thttpd 0.000043 CALL
sendfile(0x33,0x43d,0xa5fd000,0,0xc0dca,0,0xbfbfe638,0)
91919 thttpd 0.005250 RET sendfile -1 errno 35 Resource
temporarily unavailable
More information about the freebsd-net
mailing list