Am 25.10.2006 um 19:36 schrieb Robert Watson:
>
> On Wed, 25 Oct 2006, Stefan Bethke wrote:
>
>> We're consistely getting this panic even under smallish loads.
>> I've experimented with various values for VM_KMEM_SIZE_MAX (384,
>> 512, 768 and 1024 MB), but the boxes are still panicking after
>> even short periods (a few hours) just doing a buildworld, a few
>> ports compiles, or even when logging out of the console(?).
>>
>> Would logging vm.zone every few minutes help detecting a kernel
>> memory leak?
>>
>> I'm at a complete loss as to the actual cause of these. The ACPI
>> errors precede the panic below by only a few seconds, so I'd guess
>> they're a symptom, not a cause. We've tried with ACPI disabled in
>> the BIOS, but we also got these panics, so we re-enabled it.
>>
>> The hardware is a Tyan GT20 (2865) with a single-core Opteron, two
>> gigs of RAM and two or three SATA disks.
>>
>> Any ideas?
>
> Try the commands "show uma" and "show malloc" in DDB to print the
> memory allocations made by the uma(9) and malloc(9) kernel memory
> allocators. This output may be sufficient to suggest to us where a
> kernel memory leak, if any, might be taking place. Also, a stack
> trace ("trace") never hurts; if something is sitting there
> allocating a lot of memory at high speed, it could well be that the
> last call to the memory allocator is the one leaking.
Thanks, see below. I'll let it sit in the debugger, if someone has
some more ideas for what to look at.
This panic was triggered by me trying to ssh into the box; it had
been sitting idle since the last reboot around 1900 CEST. Note that
I had to break into the debugger manually. I'm assuming that the
memory shortage is so severe that dumping cannot be initiated, so the
kernel hangs...
Thanks,
Stefan
login: cblock_alloc_cblocks: M_NOWAIT malloc failed, trying M_WAITOK
panic: kmem_malloc(4096): kmem_map too small: 699756544 total allocated
Uptime: 5h4m51s
KDB: enter: Break sequence on console
[thread pid 1545 tid 100066 ]
Stopped at kdb_enter+0x30: leave
db> show uma
Zone Allocs Frees Used Cache
FFS2 dinode 529 80 449 1
FFS1 dinode 0 0 0 0
FFS inode 529 80 449 15
Mountpoints 4 0 4 8
SWAPMETA 0 0 0 0
rtentry 5 0 5 53
unpcb 784 772 12 44
ripcb 0 0 0 0
sackhole 0 0 0 0
tcpreass 0 0 0 0
hostcache 2 1 1 99
syncache 105 105 0 78
tcptw 1 1 0 156
tcpcb 117 109 8 16
inpcb 117 109 8 36
udpcb 288 276 12 32
ipq 0 0 0 0
socket 1190 1158 32 12
KNOTE 58 58 0 112
PIPE 655 654 1 17
NFSNODE 0 0 0 0
NFSMOUNT 0 0 0 0
DIRHASH 29 0 29 7
NAMEI 30174 30174 0 12
L VFS Cache 0 0 0 0
S VFS Cache 5865 5409 456 48
VNODEPOLL 0 0 0 0
VNODE 565 82 483 7
ata_composite 0 0 0 0
ata_request 63093 63093 0 76
g_bio 310921 310921 0 261
ACL UMA zone 0 0 0 0
mbuf_jumbo_16k 0 0 0 0
mbuf_jumbo_9k 0 0 0 0
mbuf_jumbo_pagesize 0 0 0 0
mbuf_cluster 384 0 384 6
mbuf 1725 1724 1 140
mbuf_packet 23439 23119 320 64
VMSPACE 1508 1478 30 22
UPCALL 0 0 0 0
KSEGRP 77 0 77 43
THREAD 77 0 77 13
PROC 1547 1478 69 8
Files 10873 10787 86 73
4096 5805 5704 101 18
2048 271 89 182 0
1024 3055 3008 47 145
512 4305 4270 35 13
256 4213 3909 304 26
128 22332 20502 1830 0
64 9997360 9992724 4636 9985362
32 4944 2141 2803 135
16 10069985 10067371 2614 228
mt_zone 158 0 158 78
DP fakepg 0 0 0 0
PV ENTRY 785101 775096 10005 1595
MAP ENTRY 59298 58673 625 159
KMAP ENTRY 170937 170911 26 142
MAP 7 0 7 33
VM OBJECT 29822 28947 875 82
128 Bucket 44 2 42 0
64 Bucket 32 1 31 11
32 Bucket 24 1 23 5
16 Bucket 28 0 28 22
UMA Hash 6 1 5 25
UMA RCntSlabs 195 0 195 27
UMA Slabs 340 41 299 55
UMA Zones 67 0 67 23
UMA Kegs 67 0 67 5
db> show malloc
Type Allocs Frees Used
MADT Table 0 0 0
acpipwr 0 0 0
acpi_perf 0 0 0
acpidev 93 0 93
acpisem 17 0 17
acpicmbat 0 0 0
PCI Link 64 0 64
acpitask 2 2 0
acpica 20052640 20049612 3028
KTRACE 100 0 100
prison 0 0 0
$PIR 0 0 0
DEVFS3 98 1 97
nexusdev 3 0 3
MP Table 0 0 0
memdesc 1 0 1
legacydrv 0 0 0
ithread 66 0 66
I/O APIC 1 0 1
zombie 1478 1478 0
proc-args 4944 4914 30
kqueue 52 52 0
kenv 114 1 113
atkbddev 2 0 2
sigio 1 0 1
file desc to leader 0 0 0
VM pgdata 2 0 2
file desc 1570 1500 70
DEVFS2 0 0 0
USBHC 0 0 0
cdev 21 0 21
USBdev 9 6 3
UMAHash 3 2 1
UFS mount 9 0 9
UFS quota 0 0 0
UFS dirhash 30 0 30
savedino 0 0 0
newdirblk 0 0 0
dirrem 0 0 0
mkdir 0 0 0
diradd 0 0 0
freefile 0 0 0
freeblks 0 0 0
freefrag 0 0 0
allocindir 0 0 0
indirdep 0 0 0
allocdirect 0 0 0
bmsafemap 0 0 0
newblk 1 0 1
inodedep 1 0 1
pagedep 1 0 1
rpcclnt 0 0 0
p1003.1b 1 0 1
agp 0 0 0
NFS daemon 5 0 5
NFSV3 srvdesc 0 0 0
NFS srvsock 1 0 1
nlminfo 0 0 0
NFS lock 0 0 0
NFS DirectIO 0 0 0
NFS hash 0 0 0
NFSV3 diroff 0 0 0
NFSV3 bigfh 0 0 0
NFS req 0 0 0
NFS srvsock 0 0 0
idmap 0 0 0
NFS4 dev 0 0 0
syncache 1 0 1
USB 31 0 31
hostcache 1 0 1
ip_moptions 0 0 0
Export Host 0 0 0
in_multi 3 0 3
igmp 0 0 0
routetbl 55 41 14
entropy 1024 0 1024
ata_dma 6 0 6
ad_driver 2 0 2
vlan 0 0 0
tun 0 0 0
lo 1 0 1
arpcom 2 0 2
clone 2 0 2
ether_multi 14 2 12
ifaddr 22 0 22
ifnet 4 0 4
BPF 3 0 3
ata_generic 3 0 3
Export Host 0 0 0
vnodemarker 4068 4068 0
mount 251 175 76
vnodes 1 0 1
VFS hash 1 0 1
Export Host 1 0 1
cluster_save buffer 0 0 0
vfscache 1 0 1
BIO buffer 56 9 47
isadev 18 0 18
pcb 57 35 22
soname 2881 2877 4
mbuf_tag 2 2 0
mbextcnt 0 0 0
accf 0 0 0
ptys 1 0 1
ttys 3070 1839 1231
shm 1 0 1
sem 4 0 4
msg 4 0 4
iov 581 581 0
select 0 0 0
ioctlops 1346 1346 0
Unitno 8 2 6
turnstiles 91 0 91
taskqueue 9 0 9
stack 0 0 0
MD sectors 0 0 0
MD disk 0 0 0
sleep queues 91 0 91
sbuf 246 246 0
rman 542 366 176
LED 0 0 0
kbdmux 6 0 6
acd_driver 1 0 1
kobj 134 19 115
eventhandler 44 0 44
devstat 8 0 8
mirror_data 0 0 0
bus-sc 1841 1759 82
bus 4342 3549 793
SWAP 2 0 2
umtx 90 0 90
sysctltmp 219 219 0
sysctloid 3189 0 3189
sysctl 557 557 0
uidinfo 141 136 5
plimit 1251 1237 14
ata_pci 0 0 0
cred 10711 10697 14
subproc 1626 1478 148
proc 2 0 2
session 227 203 24
pgrp 237 213 24
GEOM 429 331 98
mtx_pool 1 0 1
module 180 0 180
pfs_vncache 0 0 0
free 0 0 0
pfs_nodes 0 0 0
ip6ndp 0 0 0
ip6opt 0 0 0
temp 10112 10099 13
devbuf 1563 1 1562
cache 0 0 0
lockf 84 78 6
DEVFS 13 1 12
linker 54 24 30
DEVFS_RULE 0 0 0
DEVFS1 86 0 86
db> trace
Tracing pid 1545 tid 100066 td 0xc6893d80
kdb_enter(c06e1497,c6523cf8,200820e,418500,c06ff738,...) at kdb_enter
+0x30
siointr1(c6523c00,c6530100,c639a4c4,ffc0d61c,c069241c,...) at siointr1
+0xe1
siointr(c6523c00,c6893d80,ffc0d61c,0,c6893d80,...) at siointr+0x3f
intr_execute_handlers(c639a4c4,ffc0d634,ffc0d678,c068e5e3,38,...) at
intr_execute_handlers+0x10c
lapic_handle_intr(38) at lapic_handle_intr+0x3a
Xapic_isr1() at Xapic_isr1+0x33
--- interrupt, eip = 0xc0698f02, esp = 0xffc0d678, ebp = 0xffc0d678 ---
is_dumpable(9095000,0,0,0,0,...) at is_dumpable+0x22
minidumpsys(c0723a00,c052f034,ffc0d730,c052fa08,c0745c34,...) at
minidumpsys+0xde
dumpsys(c0723a00,ffc0d7bc,c050a7b4,c06d0ba4,104,...) at dumpsys+0x1c
doadump(c06d0ba4,104,c06d0c9d,c052aa40,ffc0d7a8,...) at doadump+0x4c
boot(104,c0723900,c06ddf3b,ffc0d7f4,100,...) at boot+0x514
panic(c06ddf3b,1000,29b57000,ffc0d824,a,...) at panic+0x13d
kmem_malloc(c104b0c0,1000,2,ffc0d87c,c063891b,...) at kmem_malloc+0xab
page_alloc(c1055880,1000,ffc0d86f,2,ffc0d877,...) at page_alloc+0x27
slab_zalloc(c1055880,2,c1055880,1,0,...) at slab_zalloc+0x11b
uma_zone_slab(c1055880,2,c1055880,1,c0704f60,...) at uma_zone_slab+0x106
uma_zalloc_bucket(c1055880,2,1,2f,4d,...) at uma_zalloc_bucket+0x17f
uma_zalloc_arg(c1055880,0,2,c1056b40,c69a4780,...) at uma_zalloc_arg
+0x319
malloc(80,c07029a0,2,4d,c6517400,...) at malloc+0x58
cblock_alloc_cblocks(4d,c6517400,c6517400,c6893d80,ffc0d978,...) at
cblock_alloc_cblocks+0x5d
clist_alloc_cblocks(c6517400,2000,2000,c6517400,c6517400,...) at
clist_alloc_cblocks+0x77
ttsetwater(c6517400,8,c69a2b80,ffc0d9b4,c0548da3,...) at ttsetwater+0xa1
tty_open(c6897b00,c6517400,0,c0703040,ffc0d9bc,...) at tty_open+0x7d
ptsopen(c6897b00,3,2000,c6893d80,0,...) at ptsopen+0x103
giant_open(c6897b00,3,2000,c6893d80,ffc0da18,...) at giant_open+0x71
devfs_open(ffc0da50,ffc0db0c,c058423d,c06f75e0,ffc0da50,...) at
devfs_open+0x272
VOP_OPEN_APV(c06f75e0,ffc0da50,c071be20,c66ac000,0,...) at
VOP_OPEN_APV+0x43
vn_open_cred(ffc0dbc0,ffc0dcc0,0,c6971580,6,...) at vn_open_cred+0x49d
vn_open(ffc0dbc0,ffc0dcc0,0,6,ffc0db58,...) at vn_open+0x33
kern_open(c6893d80,bfbfe050,0,3,0,...) at kern_open+0xc8
open(c6893d80,ffc0dd04,c,c6893d80,8089000,...) at open+0x36
syscall(3b,3b,3b,8077274,4,...) at syscall+0x362
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (5, FreeBSD ELF32, open), eip = 0x28300747, esp =
0xbfbfe01c, ebp = 0xbfbfe078 ---
--
Stefan Bethke <stb at lassitu.de> Fon +49 170 346 0140