Am 26.10.2006 um 15:47 schrieb Stefan Bethke:
> Am 26.10.2006 um 15:37 schrieb Robert Watson:
>
>> On Thu, 26 Oct 2006, Stefan Bethke wrote:
>>
>>> acpica 3024 159K 20026966
>> ...
>>> db> show uma
>>> Zone Allocs Frees Used Cache
>>> 64 9990754 9986054 4700 9980755
>>
>> Looks like acpica has gone crazy performing allocation/freeing at
>> a very high rate, and that for some reason, UMA is failing to
>> properly reuse/release memory. So there are two bugs/problems
>> here: whatever is causing ACPI to behave this way, and then the
>> fact that UMA is failing to deal properly with its misbehavior.
>
> We had the machines running with ACPI disabled for a week or so,
> and we were still getting these panics, but I'll disable it again
> in the BIOS to make sure.
>
>> Alternatively, that we have a bug in the way statistics are
>> handled. If you can generate a coredump, it would be quite useful
>> to be able to run umstat (src/tools/tools/umastat in HEAD) on it.
>> The tool probably needs a bit of tweaking to run on the core dump
>> -- in particular, the first and second arguments of kvm_open()
>> need to be the name of the kernel and dumpfile, rather than NULL.
>> This would help confirm what actual state UMA is in.
>
> So far, the machines always just hang instead of dumping core; I'll
> see if I can get them to write a dump.
On two of the three boxes, the panics have not happened for a week.
On the one I disabled ACPI on, no panics were seen for a week as
well, so I enabled it again. Within an hour, the panic occurred.
call doadump just hangs, so I won't be able to run umastat. I'll try
to kick the box via the IPMI card and wait for the next panic.
Fri Nov 3 23:22:58 CET 2006
FreeBSD/i386 (discovery.tallence.de) (ttyd0)
login: -0255: *** Error: UtCallocate: Could not allocate size 30
-0255: *** Error: UtCallocate: Could not allocate size 30
-0255: *** Error: UtCallocate: Could not allocate size 30
ACPI-1304: *** Error: Method execution failed [\SWFS] (Node
0xc63ee220), AE_NO_MEMORY
ACPI-1304: *** Error: Method execution failed [\RBYT] (Node
0xc63ee1a0), AE_NO_MEMORY
ACPI-1304: *** Error: Method execution failed [\RTMP] (Node
0xc63ee380), AE_NO_MEMORY
ACPI-1304: *** Error: Method execution failed [\_TZ_.THRM._TMP]
(Node 0xc63ede00), AE_NO_MEMORY
panic: kmem_malloc(4096): kmem_map too small: 699756544 total allocated
KDB: enter: panic
[thread pid 1119 tid 100074 ]
Stopped at kdb_enter+0x30: leave
db> show uma
Zone Allocs Frees Used Cache
pfosfp 188 0 188 193
pfospfen 345 0 345 51
pfiaddrpl 0 0 0 0
pfstatescrub 0 0 0 0
pffrcent 0 0 0 0
pffrcache 0 0 0 0
pffrag 0 0 0 0
pffrent 0 0 0 0
pfrkentry2 0 0 0 0
pfrkentry 0 0 0 0
pfrktable 0 0 0 0
pfpooladdrpl 0 0 0 0
pfaltqpl 0 0 0 0
pfstatepl 873 844 29 76
pfrulepl 62 0 62 10
pfsrctrpl 1 0 1 77
FFS2 dinode 2117 193 1924 11
FFS1 dinode 0 0 0 0
FFS inode 2117 193 1924 48
Mountpoints 19 0 19 5
SWAPMETA 0 0 0 0
rtentry 10 0 10 48
unpcb 664 613 51 33
ripcb 0 0 0 0
sackhole 0 0 0 0
tcpreass 9 9 0 169
hostcache 39 0 39 61
syncache 150 150 0 78
tcptw 55 55 0 156
tcpcb 229 198 31 25
inpcb 229 198 31 35
udpcb 979 959 20 24
ipq 0 0 0 0
socket 1873 1771 102 8
KNOTE 1242 1242 0 112
PIPE 1033 1022 11 16
NFSNODE 0 0 0 0
NFSMOUNT 0 0 0 0
DIRHASH 91 0 91 5
NAMEI 42625 42625 0 20
L VFS Cache 0 0 0 0
S VFS Cache 6029 4101 1928 32
VNODEPOLL 0 0 0 0
VNODE 2211 199 2012 4
ata_composite 0 0 0 0
ata_request 12690 12690 0 380
g_bio 47901 47901 0 1247
ACL UMA zone 0 0 0 0
mbuf_jumbo_16k 0 0 0 0
mbuf_jumbo_9k 0 0 0 0
mbuf_jumbo_pagesize 0 0 0 0
mbuf_cluster 128 0 128 6
mbuf 8800 8772 28 114
mbuf_packet 47880 47812 68 60
VMSPACE 2384 2317 67 37
UPCALL 6 0 6 150
KSEGRP 129 0 129 71
THREAD 216 81 135 15
PROC 2424 2317 107 19
Files 17415 17156 259 59
4096 5808 5666 142 25
2048 554 185 369 5
1024 2266 2218 48 144
512 3396 3339 57 15
256 6678 6240 438 42
128 24937 22580 2357 73
64 9966167 9961335 4832 9947229
32 5515 2447 3068 96
16 10000705 9997868 2837 208
mt_zone 163 0 163 73
DP fakepg 0 0 0 0
PV ENTRY 1242630 1213645 28985 1320
MAP ENTRY 88676 86884 1792 0
KMAP ENTRY 171204 171178 26 142
MAP 7 0 7 33
VM OBJECT 45712 43104 2608 321
128 Bucket 58 2 56 0
64 Bucket 49 1 48 8
32 Bucket 31 0 31 25
16 Bucket 30 0 30 20
UMA Hash 6 1 5 25
UMA RCntSlabs 67 0 67 7
UMA Slabs 669 193 476 55
UMA Zones 83 0 83 7
UMA Kegs 83 0 83 13
db> show malloc
Type InUse MemUse Requests
NULLFS mount 8 1K 8
NULLFS hash 1 1K 1
NULLFS node 8 1K 8
pflog 1 1K 1
pf_if 26 6K 26
MADT Table 0 0K 0
acpipwr 0 0K 0
acpi_perf 0 0K 0
acpidev 93 3K 93
acpisem 17 2K 17
acpicmbat 0 0K 0
PCI Link 64 6K 64
acpitask 0 0K 2
acpica 3024 159K 19943808
KTRACE 100 13K 100
prison 4 8K 4
$PIR 0 0K 0
DEVFS3 562 71K 563
nexusdev 3 1K 3
MP Table 0 0K 0
memdesc 1 4K 1
legacydrv 0 0K 0
ithread 66 6K 66
I/O APIC 1 1K 1
zombie 0 0K 2317
proc-args 52 4K 5377
kqueue 0 0K 1234
kenv 113 8K 114
atkbddev 2 1K 2
sigio 1 1K 1
file desc to leader 0 0K 0
VM pgdata 2 65K 2
file desc 108 27K 2425
DEVFS2 100 4K 400
USBHC 0 0K 0
cdev 20 3K 20
USBdev 3 1K 9
UMAHash 1 1K 2
UFS mount 18 82K 18
UFS quota 0 0K 0
UFS dirhash 87 16K 87
savedino 0 0K 0
newdirblk 0 0K 0
dirrem 0 0K 0
mkdir 0 0K 0
diradd 0 0K 0
freefile 0 0K 0
freeblks 0 0K 0
freefrag 0 0K 0
allocindir 0 0K 0
indirdep 0 0K 0
allocdirect 0 0K 0
bmsafemap 0 0K 0
newblk 1 1K 1
inodedep 1 256K 1
pagedep 1 64K 1
rpcclnt 0 0K 0
p1003.1b 1 1K 1
agp 0 0K 0
NFS daemon 5 10K 5
NFSV3 srvdesc 0 0K 0
NFS srvsock 1 1K 1
nlminfo 0 0K 0
NFS lock 0 0K 0
NFS DirectIO 0 0K 0
NFS hash 0 0K 0
NFSV3 diroff 0 0K 0
NFSV3 bigfh 0 0K 0
NFS req 0 0K 0
NFS srvsock 0 0K 0
idmap 0 0K 0
NFS4 dev 0 0K 0
syncache 1 8K 1
USB 31 3K 31
hostcache 1 24K 1
ip_moptions 0 0K 0
Export Host 0 0K 0
in_multi 3 1K 3
igmp 0 0K 0
routetbl 20 2K 93
entropy 1024 64K 1024
ad_driver 3 1K 3
ata_generic 4 4K 4
vlan 2 1K 2
tun 0 0K 0
lo 1 1K 1
arpcom 3 1K 3
clone 3 12K 3
ether_multi 12 1K 14
ifaddr 30 7K 30
ifnet 6 6K 6
BPF 9 65K 10
acd_driver 1 2K 1
Export Host 0 0K 0
vnodemarker 0 0K 2420
mount 223 6K 314
vnodes 1 1K 1
VFS hash 1 256K 1
Export Host 0 0K 0
cluster_save buffer 0 0K 6
vfscache 1 512K 1
BIO buffer 224 448K 235
isadev 19 2K 19
pcb 21 5K 703
soname 13 1K 3568
mbuf_tag 0 0K 2
mbextcnt 0 0K 0
accf 0 0K 0
ptys 0 0K 0
ttys 1177 165K 2648
shm 1 12K 1
sem 4 7K 4
msg 4 25K 4
iov 0 0K 828
select 0 0K 0
ioctlops 0 0K 2054
Unitno 6 1K 178
turnstiles 151 10K 151
taskqueue 9 1K 9
stack 0 0K 0
MD sectors 0 0K 0
MD disk 0 0K 0
sleep queues 151 5K 151
sbuf 0 0K 906
rman 176 11K 544
LED 0 0K 0
kbdmux 6 9K 6
ata_pci 0 0K 0
kobj 115 230K 134
eventhandler 55 4K 55
devstat 14 29K 14
mirror_data 3 1K 625
bus-sc 82 32K 1841
bus 799 38K 4368
SWAP 2 549K 2
umtx 150 10K 150
sysctltmp 0 0K 333
sysctloid 3214 97K 3214
sysctl 0 0K 634
uidinfo 6 2K 141
plimit 25 7K 1334
ata_dma 6 1K 6
cred 50 7K 12035
subproc 235 464K 2552
proc 2 8K 2
session 46 6K 223
pgrp 46 3K 223
GEOM 156 17K 832
mtx_pool 1 8K 1
module 183 12K 183
pfs_vncache 0 0K 0
free 0 0K 0
pfs_nodes 0 0K 0
ip6ndp 0 0K 0
ip6opt 0 0K 0
temp 13 211K 6822
devbuf 1562 3592K 1564
cache 0 0K 0
lockf 21 2K 6581
DEVFS 62 2K 63
linker 50 243K 105
DEVFS_RULE 0 0K 0
DEVFS1 100 25K 102
db> call doadump
--
Stefan Bethke <stb at lassitu.de> Fon +49 170 346 0140