kern/90206: Server reboot after "FAILURE - out of memory in ata_raid_init_request"

[Simon]

>Number:         90206
>Category:       kern
>Synopsis:       Server reboot after "FAILURE - out of memory in ata_raid_init_request"
>Confidential:   no
>Severity:       critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Dec 10 18:30:02 GMT 2005
>Closed-Date:
>Last-Modified:
>Originator:     Simon
>Release:        6.0-RELEASE
>Organization:
KSD
>Environment:
FreeBSD bnet 6.0-RELEASE FreeBSD 6.0-RELEASE #0: Sat Dec 10 02:13:22 UTC 2005     ender at bnet:/usr/obj/usr/src/sys/BNET  i386
>Description:
A server deamon I run keeps its database in memory, and after a schedualed time flushes the dirty database files from memory to harddrives. The database files are stored as millions of tiny CDB binary files in 1 directory. I use RAID 0+1 on a promise raid card. When the server kicks off a flush and starts doing a large amount of IO, i get the following errors in dmesg:

Dec 10 13:51:39 <kern.crit> bnet kernel: DOH! ata_alloc_request failed!
Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request
Dec 10 13:51:39 <kern.crit> bnet last message repeated 4 times
Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE g- out of memory i_vn fsat_da_one(raid):_iarni0st_1dre[WquRIestTE(offset=2030
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs
Dec 10 13:51:39 <kern.crit> bnet kernel: _done()FA:aILr0UREs1 -d[ oWRIut oTEf (omeffmoseryt= 2in ata0329_rai03d_94init_request
Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request
Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory in ata_rai62d_4, inleit_rengquthest=16384)]error = 5
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203290411008, length=16384)]error = 5
Dec 10 13:51:39 <kern.crit> bnet kernel:
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_dFAoInLUe():RE - arou0st 1dof[ mWRemITory E(inoffs aet=t2a_raid0329_init0427392
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE_re(oquffseest=t203290460160, length=16384)]error = 5
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203290509312, length=16384)]error = 5
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=20329057
Dec 10 13:51:39 <kern.crit> bnet kernel: 4848, lenFAILgtURh=E - 163ou84t )]of error = 5
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203290787840, length=16384)]error = 5
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203675779072, length=16384)]error me= mory5 in ata_r
Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request
Dec 10 13:51:39 <kern.crit> bnet kernel: F
Dec 10 13:51:39 <kern.crit> bnet kernel: AILURE - out go_fv fmse_mdoornye (i)n: aart0as_1rda[iWdR_IiTnEi(to_frfesqeute=st
Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory i2n0 3a8t6a8_5r2a0i4d4_8i,n ilte_nrgetqhu=est
Dec 10 13:51:39 <kern.crit> bnet kernel: 16384)]error = 5
Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - ougt_ ovff mse_modroy nien (a)ta:_arra0isd1_di[nWiRtI_TrEe(qofufesestt=
Dec 10 13:51:39 <kern.crit> bnet kernel: 203868536832, length=F16A3I84L)U]ReErr o-r  =o u5t
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203 8o6f8 5m5e3m2o1r6y,  ilne nagttah_=raid_1i6n3i8t
Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memorry  i=n  5a
Dec 10 13:51:39 <kern.crit> bnet kernel: ta_raid_init_request
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=F2A04I0L61U2R78E2 0-8 ,o ulte nogft hm=emory 1in ata
Dec 10 13:51:39 <kern.crit> bnet kernel:

<snip>

Dec 10 13:52:02 <kern.crit> bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=12288)]error = 5
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
Dec 10 13:52:02 <kern.crit> bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=14336)]error = 5
Dec 10 13:52:02 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request
Dec 10 13:52:02 <kern.crit> bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=14336)]error = 5
Dec 10 13:52:02 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request
Dec 10 13:52:02 <kern.crit> bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=14336)]error = 5
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

<crash>

Dec 10 13:53:24 <kern.crit> bnet kernel: Copyright (c) 1992-2005 The FreeBSD Project.
Dec 10 13:53:24 <kern.crit> bnet kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Dec 10 13:53:24 <kern.crit> bnet kernel: The Regents of the University of California. All rights reserved.
Dec 10 13:53:24 <kern.crit> bnet kernel: FreeBSD 6.0-RELEASE #0: Sat Dec 10 02:13:22 UTC 2005

Kernel:
http://ender.tog.net/crash/BNET
dmesg:
http://ender.tog.net/crash/dmesg.today
loader.conf:
http://ender.tog.net/crash/loader.conf
messages:
http://ender.tog.net/crash/messages.0

This problem was also repeated on just RAID 0 array. Unfortunatally i do not have a kernel dump file at the moment due to the 
bnet# dumpon /dev/ar0
dumpon: ioctl(DIOCSKERNELDUMP): Operation not supported by device
Limitation, but hopefully will have another disk in this server to get a proper kernel dump soon. 
>How-To-Repeat:
Run a deamon that uses alot of disk IO on a raid array.         
>Fix:
              
>Release-Note:
>Audit-Trail:
>Unformatted: