amrd disk performance drop after running under high load
Alexey Popov
lol at chistydom.ru
Wed Oct 17 12:53:56 PDT 2007
Hi
Kris Kennaway wrote:
>>>> And few hours ago I received feed back from Andrzej Tobola, he has
>>>> the same problem on FreeBSD 7 with Promise ATA software mirror:
>>> Well, he didnt provide any evidence yet that it is the same problem,
>>> so let's not become confused by feelings :)
>> I think he is telling about 100% disk busy while processing ~5
>> transfers/sec.
>
> "% busy" as reported by gstat doesn't mean what you think it does. What
> is the I/O response time? That's the meaningful statistic for
> evaluating I/O load. Also you didnt post about this.
At the problematic time the disk felt to be very slow, processes all
were in reading disk state and vmstat proved it by the % numbers.
>>>> So I can conclude that FreeBSD has a long standing bug in VM that
>>>> could be triggered when serving large amount of static data (much
>>>> bigger than memory size) on high rates. Possibly this only applies
>>>> to large files like mp3 or video.
>>> It is possible, we have further work to do to conclude this though.
>> I forgot to mention I have pmc and kgmon profiling for good and bad
>> times. But I have not enough knowledge to interpret it right and not
>> sure if it can help.
> pmc would be useful.
Unfortunately i've lost pmc profiling results. I'll try to collect it
again later. See vmstats in attach (vmstat -z; netstat -m; vmstat -i;
vmstat -w 1 | head -11;).
Also you can see kgmon profiling results at: http://83.167.98.162/gprof/
With best regards,
Alexey Popov
-------------- next part --------------
ITEM SIZE LIMIT USED FREE REQUESTS FAILURES
UMA Kegs: 240, 0, 71, 4, 71, 0
UMA Zones: 376, 0, 71, 9, 71, 0
UMA Slabs: 128, 0, 1011, 62, 243081, 0
UMA RCntSlabs: 128, 0, 361, 1205, 363320, 0
UMA Hash: 256, 0, 4, 11, 7, 0
16 Bucket: 152, 0, 45, 30, 72, 0
32 Bucket: 280, 0, 25, 45, 69, 0
64 Bucket: 536, 0, 17, 25, 55, 53
128 Bucket: 1048, 0, 287, 88, 1200, 95423
VM OBJECT: 224, 0, 5536, 23228, 7675004, 0
MAP: 352, 0, 7, 15, 7, 0
KMAP ENTRY: 112, 90222, 283, 1037, 1207524, 0
MAP ENTRY: 112, 0, 1396, 419, 72221561, 0
PV ENTRY: 48, 2244600, 17835, 30261, 768591673, 0
DP fakepg: 120, 0, 0, 31, 10, 0
mt_zone: 1024, 0, 170, 6, 170, 0
16: 16, 0, 3578, 2470, 745206870, 0
32: 32, 0, 1273, 343, 1750850, 0
64: 64, 0, 6147, 1693, 487691440, 0
128: 128, 0, 4659, 387, 1464251, 0
256: 256, 0, 596, 2539, 7208469, 0
512: 512, 0, 608, 253, 791295, 0
1024: 1024, 0, 49, 239, 82867, 0
2048: 2048, 0, 27, 295, 115362, 0
4096: 4096, 0, 240, 278, 564659, 0
Files: 120, 0, 544, 324, 263880246, 0
TURNSTILE: 104, 0, 181, 83, 307, 0
PROC: 856, 0, 82, 82, 308409, 0
THREAD: 608, 0, 169, 11, 24468, 0
KSEGRP: 136, 0, 165, 69, 165, 0
UPCALL: 88, 0, 3, 73, 3, 0
SLEEPQUEUE: 64, 0, 181, 99, 307, 0
VMSPACE: 544, 0, 35, 77, 310929, 0
mbuf_packet: 256, 0, 368, 115, 1331807039, 0
mbuf: 256, 0, 2016, 2331, 5433003167, 0
mbuf_cluster: 2048, 32768, 483, 239, 1236143964, 0
mbuf_jumbo_pagesize: 4096, 0, 0, 0, 0, 0
mbuf_jumbo_9k: 9216, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 0, 0, 0, 0, 0
ACL UMA zone: 388, 0, 0, 0, 0, 0
g_bio: 216, 0, 4, 410, 48175991, 0
ata_request: 336, 0, 0, 22, 24, 0
ata_composite: 376, 0, 0, 0, 0, 0
VNODE: 496, 0, 28250, 21270, 911708, 0
VNODEPOLL: 152, 0, 0, 0, 0, 0
S VFS Cache: 104, 0, 29153, 9979, 1387950, 0
L VFS Cache: 327, 0, 258, 282, 9423, 0
NAMEI: 1024, 0, 0, 260, 286369405, 0
NFSMOUNT: 584, 0, 1, 6, 1, 0
NFSNODE: 664, 0, 1, 5, 126, 0
DIRHASH: 1024, 0, 278, 122, 1954, 0
PIPE: 768, 0, 35, 335, 253930, 0
KNOTE: 120, 0, 354, 235, 689363256, 0
socket: 616, 49152, 504, 264, 1311349, 0
ipq: 56, 1071, 0, 0, 135, 0
udpcb: 304, 49152, 6, 42, 185368, 0
inpcb: 304, 49152, 384, 192, 903992, 0
tcpcb: 752, 49155, 376, 179, 903992, 0
tcptw: 80, 8235, 8, 487, 211995, 0
syncache: 128, 15370, 0, 145, 890626, 0
hostcache: 136, 15372, 2620, 572, 251887, 0
tcpreass: 40, 2100, 0, 336, 265497, 0
sackhole: 32, 0, 7, 397, 30124600, 0
ripcb: 304, 49152, 0, 36, 64, 0
unpcb: 200, 49153, 40, 340, 221924, 0
rtentry: 264, 0, 6, 36, 26, 0
divcb: 304, 49152, 0, 0, 0, 0
IPFW dynamic rule: 120, 0, 212, 346, 113020, 0
SWAPMETA: 288, 116519, 280, 786, 87016, 0
Mountpoints: 792, 0, 9, 16, 13, 0
FFS inode: 192, 0, 28202, 6158, 911421, 0
FFS1 dinode: 128, 0, 0, 0, 0, 0
FFS2 dinode: 256, 0, 28202, 5713, 911421, 0
2381/2449/4830 mbufs in use (current/cache/total)
368/354/722/32768 mbuf clusters in use (current/cache/total/max)
368/112 mbuf+clusters out of packet secondary zone in use (current/cache)
0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/0 9k jumbo clusters in use (current/cache/total/max)
0/0/0/0 16k jumbo clusters in use (current/cache/total/max)
1331K/1320K/2651K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
8441913 requests for I/O initiated by sendfile
6263 calls to protocol drain routines
interrupt total rate
irq6: fdc0 8 0
irq14: ata0 47 0
irq16: uhci0 1464547796 1870
irq18: uhci2 12614009 16
irq23: ehci0 3 0
irq46: amr0 12215890 15
irq64: em0 1463513610 1869
cpu0: timer 1564021008 1997
cpu1: timer 1565552539 1999
Total 6082464910 7768
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr am0 in sy cs us sy id
0 2 0 84568 155760 130 3 3 0 298 193 0 771 3421 2251 0 5 95
0 2 0 84576 155488 18 0 0 0 0 0 9 3167 219 7860 0 2 98
0 2 0 84576 155360 0 0 0 0 0 0 2 3568 155 8485 0 1 99
0 2 0 84576 155296 0 0 0 0 0 0 1 2298 110 6218 0 0 100
0 2 0 84576 155232 0 0 0 0 0 0 1 1288 110 4568 0 0 100
0 2 0 84580 154792 1 0 0 0 0 0 10 1459 896 4830 0 1 99
0 2 0 84580 154664 0 0 0 0 0 0 2 2718 128 6911 0 1 99
0 2 0 84580 154376 0 0 0 0 4 0 8 1436 200 4834 0 0 100
0 2 0 84580 154312 0 0 0 0 0 0 1 1500 110 4938 0 0 100
-------------- next part --------------
ITEM SIZE LIMIT USED FREE REQUESTS FAILURES
UMA Kegs: 240, 0, 71, 4, 71, 0
UMA Zones: 376, 0, 71, 9, 71, 0
UMA Slabs: 128, 0, 1003, 70, 237825, 0
UMA RCntSlabs: 128, 0, 502, 2108, 357019, 0
UMA Hash: 256, 0, 4, 11, 7, 0
16 Bucket: 152, 0, 45, 30, 72, 0
32 Bucket: 280, 0, 25, 45, 69, 0
64 Bucket: 536, 0, 17, 25, 55, 53
128 Bucket: 1048, 0, 304, 80, 1200, 95423
VM OBJECT: 224, 0, 4475, 24289, 7583940, 0
MAP: 352, 0, 7, 15, 7, 0
KMAP ENTRY: 112, 90222, 327, 993, 1178293, 0
MAP ENTRY: 112, 0, 1396, 683, 71990087, 0
PV ENTRY: 48, 2244600, 16841, 31255, 764952854, 0
DP fakepg: 120, 0, 0, 31, 10, 0
mt_zone: 1024, 0, 170, 6, 170, 0
16: 16, 0, 3147, 1725, 721283017, 0
32: 32, 0, 1273, 343, 1378831, 0
64: 64, 0, 6161, 1567, 487602322, 0
128: 128, 0, 4658, 388, 1442320, 0
256: 256, 0, 609, 1836, 7119682, 0
512: 512, 0, 608, 253, 781061, 0
1024: 1024, 0, 49, 239, 81907, 0
2048: 2048, 0, 29, 249, 114521, 0
4096: 4096, 0, 239, 294, 558310, 0
Files: 120, 0, 274, 408, 250373577, 0
TURNSTILE: 104, 0, 181, 83, 307, 0
PROC: 856, 0, 82, 82, 304241, 0
THREAD: 608, 0, 169, 11, 24468, 0
KSEGRP: 136, 0, 165, 69, 165, 0
UPCALL: 88, 0, 3, 73, 3, 0
SLEEPQUEUE: 64, 0, 181, 99, 307, 0
VMSPACE: 544, 0, 35, 77, 306723, 0
mbuf_packet: 256, 0, 619, 207, 1297797817, 0
mbuf: 256, 0, 1584, 1190, 5274774672, 0
mbuf_cluster: 2048, 32768, 826, 178, 1203897447, 0
mbuf_jumbo_pagesize: 4096, 0, 0, 0, 0, 0
mbuf_jumbo_9k: 9216, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 0, 0, 0, 0, 0
ACL UMA zone: 388, 0, 0, 0, 0, 0
g_bio: 216, 0, 0, 270, 47261412, 0
ata_request: 336, 0, 0, 22, 24, 0
ata_composite: 376, 0, 0, 0, 0, 0
VNODE: 496, 0, 26256, 23264, 899305, 0
VNODEPOLL: 152, 0, 0, 0, 0, 0
S VFS Cache: 104, 0, 27155, 11977, 1367768, 0
L VFS Cache: 327, 0, 227, 313, 9350, 0
NAMEI: 1024, 0, 0, 260, 272236181, 0
NFSMOUNT: 584, 0, 1, 6, 1, 0
NFSNODE: 664, 0, 1, 5, 126, 0
DIRHASH: 1024, 0, 278, 122, 1938, 0
PIPE: 768, 0, 35, 335, 250212, 0
KNOTE: 120, 0, 93, 372, 666594974, 0
socket: 616, 49152, 212, 466, 1282315, 0
ipq: 56, 1071, 0, 0, 135, 0
udpcb: 304, 49152, 6, 42, 183757, 0
inpcb: 304, 49152, 225, 351, 877750, 0
tcpcb: 752, 49155, 166, 389, 877750, 0
tcptw: 80, 8235, 59, 436, 204414, 0
syncache: 128, 15370, 0, 145, 864539, 0
hostcache: 136, 15372, 2200, 348, 244865, 0
tcpreass: 40, 2100, 0, 336, 252564, 0
sackhole: 32, 0, 20, 384, 29347536, 0
ripcb: 304, 49152, 0, 36, 64, 0
unpcb: 200, 49153, 40, 340, 220743, 0
rtentry: 264, 0, 6, 36, 26, 0
divcb: 304, 49152, 0, 0, 0, 0
IPFW dynamic rule: 120, 0, 215, 219, 112109, 0
SWAPMETA: 288, 116519, 280, 786, 85846, 0
Mountpoints: 792, 0, 9, 16, 13, 0
FFS inode: 192, 0, 26208, 8152, 899018, 0
FFS1 dinode: 128, 0, 0, 0, 0, 0
FFS2 dinode: 256, 0, 26208, 7722, 899018, 0
2195/1405/3600 mbufs in use (current/cache/total)
619/385/1004/32768 mbuf clusters in use (current/cache/total/max)
619/127 mbuf+clusters out of packet secondary zone in use (current/cache)
0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/0 9k jumbo clusters in use (current/cache/total/max)
0/0/0/0 16k jumbo clusters in use (current/cache/total/max)
1786K/1121K/2908K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
8233198 requests for I/O initiated by sendfile
6116 calls to protocol drain routines
interrupt total rate
irq6: fdc0 8 0
irq14: ata0 47 0
irq16: uhci0 1428187319 1851
irq18: uhci2 12374352 16
irq23: ehci0 3 0
irq46: amr0 11983237 15
irq64: em0 1427141755 1850
cpu0: timer 1540896452 1997
cpu1: timer 1542377798 1999
Total 5962960971 7730
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr am0 in sy cs us sy id
0 1 0 80564 117716 131 3 3 0 298 191 0 628 3371 1994 0 5 95
0 1 0 80568 117640 5 0 0 0 2 0 10 10265 7724 20485 1 6 93
0 1 0 80568 117400 0 0 0 0 0 0 3 10589 8031 21145 0 8 92
0 1 0 80568 116888 0 0 0 0 0 0 8 11745 8362 22538 0 12 88
0 1 0 80568 116312 0 0 0 0 0 0 9 12191 10091 23571 1 11 88
0 1 0 80568 116184 0 0 0 0 0 0 2 13182 10350 25259 1 12 87
0 2 0 80568 115928 0 0 0 0 0 0 3 12896 8176 24112 0 10 90
0 1 0 80568 115736 0 0 0 0 33 0 4 9527 5090 18717 0 9 91
0 1 0 80568 115608 0 0 0 0 32 0 2 13953 11915 26066 0 11 89
More information about the freebsd-stable
mailing list