amd64/128263: 2 amd64 dl380 g5 with dual quadcore xeons, 8 and 16gb ram, crash and dump mem

Tue Oct 21 08:10:01 UTC 2008

>Number:         128263
>Category:       amd64
>Synopsis:       2 amd64 dl380 g5 with dual quadcore xeons, 8 and 16gb ram, crash and dump mem
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-amd64
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Oct 21 08:10:01 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator:     Martin W
>Release:        7.0 amd64
>Organization:
Cypoint
>Environment:
FreeBSD db04 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Sun Feb 24 10:35:36 UTC 2008     root at driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
Two HP DL 380 G5's with dual quadcore xeons, 8GB ram on one and 16GB ram on the other. They have recently been crashing and dumping mem. The 16GB one has crashed 3 times the last 1 1/2 month, twice yesterday. The 8GB one had its first crash yesterday. They are db-servers which run postgresql or mysql.
They are both Running 7.0 amd64 RELEASE with GENERIC kernel, completely default installs except for some sysctl tweaks:
kern.ipc.maxsockets=16424
kern.maxfiles=65536
kern.maxfilesperproc=32768
net.inet.tcp.recvspace=32768
net.inet.tcp.sendspace=65536

Below is the output from kgdb, which looks about the same on both of them.

>How-To-Repeat:
Random, so don't know.
>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted:
 >kgdb /usr/obj/usr/src/sys/GENERIC/kernel.debug /var/crash/vmcore.0 

 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]

 GNU gdb 6.1.1 [FreeBSD]

 GDB is free software, covered by the GNU General Public License, and you are

 welcome to change it and/or distribute copies of it under certain conditions.

 Type "show copying" to see the conditions.

 There is absolutely no warranty for GDB.  Type "show warranty" for details.

 This GDB was configured as "amd64-marcel-freebsd".

 Unread portion of the kernel message buffer:

 <2>NMI ISA b0, EISA ff

 <2><RAM2 p>NMaINNr MMNNiNNItMMIIM M yIII ISeSr  AIr ISS IA  IbASIAbS Ao b0r0, I0Sb,0Ab ,0l,i k,A  Ee ,IEE  ElIybS IhAIaSS SE A0rI b,AAf 0dfwfS f 

 E,aff<

 f

 2<

 >

 R2

 <A<><<Mr2 2e2>>p

 <<><22<><22>>a2rA>> ifRI t2ARSM>yEAa  fIff

 A

 Mi<

 2p<

 <>S2

 <

 e<A2> r<> R222rflof>>uAfa Mr

 ,prr >e

  f<<p

 <

 .2<2>>2il><Rtai<2A>yk ree2rlrMor,i 2 tlyiRp>k>eaARl yh AryhaMa rirtadd wewyaapM rrarrer rei eoirf aftialrriupor,lye .ula triyek.elrr yi h,at rdyl eweierrarrrrkere ofaoiollrur,r e,y. r ,llh aliirkdikkeeelllwyyya   rhhehaaa rrfdrdawdawirlaurrweaere e .fa i fflaauiirleu.lreu.re.

 F

 ata

 FFFaaa

 ttlaFl t alt arltt atrra

 apa

 tFprp

 aFtat a  aa

 1Flaa ttall  ltrrt aapt p11prr99aa1p9::9 :n  :o nn-o1nm9np-   11 o9n:m -nmaasaosnk-sakkabmlbeal baei snitkea1n99rblrlnee utpeitr r::inu o:  tnttrenraprnno-puet prt m ansn rturwooahtpnk-ambtl-naimlper atpr a-im pwea sass kkiah wnhwihlkei bnatlaeki leelrei rn n er iekbilnetle  brmeonuirpd nuterkn

 ecle  rinpltept  tlmrtereruo aian inptreuddpm e k=eo d

 ecr2t  pn;tp

  euwilc da prr awrmpuiho=idiuheipltpl cee d 

   ti wrni ak ehr=c1ipd  u0i;= i;a pni 0 cd 2 =

  inlepl kw amioee 3dnps t=;i rhua pidnicertc c0nk i0iele oi

 enc

  priidi du inndellp o==i nnts0t 1r0

 ei3nru   kcet	s

 =t rm=moonddee iior0nnx8 :u0cxpe

 l  5;sociftnm o

 itfetrrauofcf	tfif=cdppuepf o0fnnxui

 i88  ppo0:o0iiddc  =i7nx0nftfeftref	8f2=f

 fsft a0f  =6ixrc8	:=0fcdp 8 0x7f f04f0fkxf ffp88f=; 2:fof08xi0nt70u;i0e

 rfs  faa5piff

 cdp i=tf8a	cf  2ff k i cn is

 it7sr uddc  ;tf fa8 cp 0 o==t i ki n=t7  0ep0ofxrif	nao t1 0e 8r2	 :  

  s0 xtp    faf f c k  =004i p o6nfi0fxfnc

  f1ftae=b0r :	 0 x7f0fx 1 0 pi

 nisnt:9f 0fx isdr f=u f f5 fff=0 ff00ofxtircuntct7eiro	

 ni nt=i p0oxs88

 f0f1rfn:afaf7tof0a:mirn0tu cxeptrof0bex7f3f 98f0pffoffifnif	oiff0tf

 0enntfrr	a=m e  f a bp7

  9   aoff irnfat0m0efxr

 e8	ep:=o r p=0fof	 r 0i anxf1 t 0m: efifn8e0 xt0 700xff8f:f0fxeffffrx pf	8f2f=f

 ff forf fi	 sff=f n0t exr0 010 ftf001	: 2  afc8k800 0 2xdf7a 0 0 0f

 = cf7xf8of   0d0 fxfpf:o08f18x f0 0e0=0: 01xs2 fef2f0fxgdf2f022

 i0ms

 ne1000ft

 n:ct0otsa1cftfafcfke fk	20d	rpx=e  2bfdasse 	f f 8poo0 7 ie6 g00x0finn0 ttmefefrr80f,e

  nlc 	i	    tmo	d	ef= i=t  sf  f0e0xgfb00fma0f1fs2        x21 

  6ee3 0 :=s0fx  0ftx =f1an,0 0cf0kf : fxt0f000pxfffxfafbf1f00t7:ofy	p

 	c=e,o  0lax0ifnxi1 bbamdiste e0 x

 f	s	fetfff gfm,	ef=0anxfebftrf0  t	,y	 D=P Ll ibp04	feff7fa 9ff0f a m0,sifat  ex0 

  b 1 pfr

  7 a=m e0fxr190 :br es00x

 x af01f0f,,fm fex 	ffpl	 iofmflfoi	t,= n ptg 0yx pfDefo0fifnf1f f,f PiLd0 xftnfearb

 7	a 1fe0fb,e f 3,2  p t0yfft0 e

 pr	,	fr	ee s g r=	  rra  mae   m 1 0aepDxPnL,  o 1   bl1o0ng

  1	,,	

   dp	perrpo=f 3ceDsPeLs oi=in n  0tt xe1er=0 :	0 r 0 xx1 2s,   loo0 	f1 f 0  f f:n0,gr  0 1,p,r ege fd   l fx = f=fraesagnfs  f03	 0x0f01x0f12, = 1lo

 n pir gonc0t0 ,1f1:02:e e1rgrsurpa,nts o r1  edeef

 f0flpnaarbgo3l2cx6003xff0 eess0d,8ff0ff0f	0f1=

  ,s i ogrfc0fnI rtaee2o6r03dcOfnl00e0 sr 1faf

 1cugPsp

 tp Le	n= r2ooe0c0g603a=e b0 slsemidonr,t e rre

 fIlOaPgLu s=	 =0 pitcn

 d4cut rere1e0netn0re	2ruarnrbte 	

 =s ecgbm6lu4epnt aosdeen2pter 0	 0	od ,tc ep reosIncOe0s=Psasbsl		e

  Le	 	g=mc= oxbean=d ,1  186I  O0(diP(Li

  c=udr drlel0ets0ee,n:t  	 	 l0ix=0

 cs ,m elibpceuu r2:r)pe raocscpe

 sunst0t)t r	p

 r	otcr=e isasa	1	7e g m0epp  =nxn0t(miui,t  mn1dub 	l i0	m=emrlb	eer5			 :=( i d=c ix1199pl

 e

 :fp uac1p)nui0

 3ct)r:a p ft xnn

 ub mtobnr0e-ra	mfaffxf	pa,s =nks u1a9 bl

 temeb eirn	t	 fyf0f=e pf1r9rxfef0f f

 u,pf, t  t0lytx,ip1em i b0txrt 1aybpp0ex f

 f

 0c	

 xfp1fu	f	ib		,=

 	 	=dt  	y=Dp P e2	L =0  D0P

 DLx,1U P 0pbp,Lrt  0iem

 ,psre	  :1p	r ,7	e dl=so  1n51Dg,e hPs1L 2 ,0 ml 04o1,n ,gd1  pelsr1o,f eng

 3d s2 e11 ,f,0 ,l og ndr3ega2 n f01 3,,1  2

 dpegfr r30a2o ,n0c,e s ggs orrr1a ae

 nfpn rl 1ao1cge

 ps

 spr	sro=ooc rcei nseetfsslseooarrgrr u esepfft	l= la egianngatssbe		l== er dirinun,t epIrttO ePreLunrarbup lptt=e   eed0n,n a

 baIlbcluOeerPddr,,Le  n tII= OO pPP0LLr  ==

 o c c00ue

 s

 rcsrcu	eu	nr=trr re 1e2nnt p t(p rirpodorclcoeeessscs	e:	s 	=c	p s1u=63	 ) (	1=i

  d4t 1lr(1iae dpl(:e in d:uclm pbeuec:5p r)u	c	4p

 =) tu1r

 97)t

 arp

 at prn aupnm unbmuembrbee	rr			=		= = 11 991P9

 h

 y

 sical memory: 8178 MB

 Dumping 577 MB: 562 546 530 514 498 482 466 450 434 418 402 386 370 354 338 322 306 290 274 258 242 226 210 194 178 162 146 130 114 98 82 66 50 34 18 2

 #0  doadump () at pcpu.h:194

 194	pcpu.h: No such file or directory.

 	in pcpu.h

 (kgdb) backtrace

 #0  doadump () at pcpu.h:194

 #1  0x0000000000000004 in ?? ()

 #2  0xffffffff80477699 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409

 #3  0xffffffff80477a9d in panic (fmt=0x104 <Address 0x104 out of bounds>) at /usr/src/sys/kern/kern_shutdown.c:563

 #4  0xffffffff8072ec94 in trap_fatal (frame=0xffffff00010f3000, eva=18446742974215641296) at /usr/src/sys/amd64/amd64/trap.c:724

 #5  0xffffffff8072f7e5 in trap (frame=0xffffffffab795f40) at /usr/src/sys/amd64/amd64/trap.c:526

 #6  0xffffffff8071596b in nmi_calltrap () at /usr/src/sys/amd64/amd64/exception.S:387

 #7  0xffffffff8070ff82 in acpi_cpu_c1 () at /usr/src/sys/amd64/acpica/acpi_machdep.c:68

 #8  0xffffffff801ca7ec in acpi_cpu_idle () at /usr/src/sys/dev/acpica/acpi_cpu.c:939

 #9  0xffffffff80495028 in sched_idletd (dummy=Variable "dummy" is not available.

 ) at /usr/src/sys/kern/sched_4bsd.c:1377

 #10 0xffffffff80458db3 in fork_exit (callout=0xffffffff80494f90 <sched_idletd>, arg=0x0, frame=0xffffffffac284c80) at /usr/src/sys/kern/kern_fork.c:781

 #11 0xffffffff807159de in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:415

 #12 0x0000000000000000 in ?? ()

 #13 0x0000000000000000 in ?? ()

 #14 0x0000000000000001 in ?? ()

 #15 0x0000000000000000 in ?? ()

 #16 0x0000000000000000 in ?? ()

 #17 0x0000000000000000 in ?? ()

 #18 0x0000000000000000 in ?? ()

 #19 0x0000000000000000 in ?? ()

 #20 0x0000000000000000 in ?? ()

 #21 0x0000000000000000 in ?? ()

 #22 0x0000000000000000 in ?? ()

 #23 0x0000000000000000 in ?? ()

 #24 0x0000000000000000 in ?? ()

 #25 0x0000000000000000 in ?? ()

 #26 0x0000000000000000 in ?? ()

 #27 0x0000000000000000 in ?? ()

 #28 0x0000000000000000 in ?? ()

 #29 0x0000000000000000 in ?? ()

 #30 0x0000000000000000 in ?? ()

 #31 0x0000000000000000 in ?? ()

 #32 0x0000000000000000 in ?? ()

 #33 0x0000000000000000 in ?? ()

 #34 0x0000000000000000 in ?? ()

 #35 0x0000000000000000 in ?? ()

 #36 0x0000000000cbd000 in ?? ()

 #37 0xffffffff80494f90 in critical_enter () at kern_switch.c:167

 #38 0x00000000fffffffb in ?? ()

 #39 0xffffff00010e58d0 in ?? ()

 #40 0xffffff0001105680 in ?? ()

 #41 0xffffff00010f3000 in ?? ()

 #42 0xffffffffac284b98 in ?? ()

 #43 0xffffff00010f3000 in ?? ()

 #44 0xffffffff80495359 in sched_switch (td=0x0, newtd=0xffffffff80494f90, flags=0) at /usr/src/sys/kern/sched_4bsd.c:905

 #45 0x0000000000000000 in ?? ()

 #46 0x0000000000000000 in ?? ()

 #47 0x0000000000000000 in ?? ()

 #48 0x0000000000000000 in ?? ()

 #49 0x0000000000000000 in ?? ()

 #50 0x0000000000000000 in ?? ()

 #51 0x0000000000000000 in ?? ()

 #52 0x0000000000000000 in ?? ()

 #53 0x0000000000000000 in ?? ()

 ---Type <return> to continue, or q <return> to quit---

 #54 0x0000000000000000 in ?? ()

 #55 0x0000000000000000 in ?? ()

 #56 0x0000000000000000 in ?? ()

 #57 0x0000000000000000 in ?? ()

 #58 0x0000000000000000 in ?? ()

 #59 0x0000000000000000 in ?? ()

 #60 0x0000000000000000 in ?? ()

 #61 0x0000000000000000 in ?? ()

 #62 0x0000000000000000 in ?? ()

 #63 0x0000000000000000 in ?? ()

 #64 0x0000000000000000 in ?? ()

 #65 0x0000000000000000 in ?? ()

 #66 0x0000000000000000 in ?? ()

 #67 0x0000000000000000 in ?? ()

 #68 0x0000000000000000 in ?? ()

 #69 0x0000000000000000 in ?? ()

 #70 0x0000000000000000 in ?? ()

 #71 0x0000000000000000 in ?? ()

 #72 0x0000000000000000 in ?? ()

 #73 0x0000000000000000 in ?? ()

 #74 0x0000000000000000 in ?? ()

 #75 0x0000000000000000 in ?? ()

 #76 0x0000000000000000 in ?? ()

 #77 0x0000000000000000 in ?? ()

 #78 0x0000000000000000 in ?? ()

 #79 0x0000000000000000 in ?? ()

 #80 0x0000000000000000 in ?? ()

 #81 0x0000000000000000 in ?? ()

 #82 0x0000000000000000 in ?? ()

 #83 0x0000000000000000 in ?? ()

 #84 0x0000000000000000 in ?? ()

 #85 0x0000000000000000 in ?? ()

 #86 0x0000000000000000 in ?? ()

 #87 0x0000000000000000 in ?? ()

 #88 0x0000000000000000 in ?? ()

 #89 0x0000000000000000 in ?? ()

 #90 0x0000000000000000 in ?? ()

 #91 0x0000000000000000 in ?? ()

 #92 0x0000000000000000 in ?? ()

 #93 0x0000000000000000 in ?? ()

 #94 0x0000000000000000 in ?? ()

 #95 0x0000000000000000 in ?? ()

 #96 0x0000000000000000 in ?? ()

 #97 0x0000000000000000 in ?? ()

 #98 0x0000000000000000 in ?? ()

 #99 0x0000000000000000 in ?? ()

 #100 0x0000000000000000 in ?? ()

 #101 0x0000000000000000 in ?? ()

 #102 0x0000000000000000 in ?? ()

 #103 0x0000000000000000 in ?? ()

 #104 0x0000000000000000 in ?? ()

 #105 0x0000000000000000 in ?? ()

 #106 0x0000000000000000 in ?? ()

 #107 0x0000000000000000 in ?? ()

 ---Type <return> to continue, or q <return> to quit---

 #108 0x0000000000000000 in ?? ()

 #109 0x0000000000000000 in ?? ()

 #110 0x0000000000000000 in ?? ()

 #111 0x0000000000000000 in ?? ()

 #112 0x0000000000000000 in ?? ()

 #113 0x0000000000000000 in ?? ()

 #114 0x0000000000000000 in ?? ()

 #115 0x0000000000000000 in ?? ()

 #116 0x0000000000000000 in ?? ()

 Cannot access memory at address 0xffffffffac285000