kern/89660: panic due to g_malloc returning null in gv_drive_done

Stijn Hoop stijn at
Mon Nov 28 09:50:04 GMT 2005

>Number:         89660
>Category:       kern
>Synopsis:       panic due to g_malloc returning null in gv_drive_done
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Nov 28 09:50:02 GMT 2005
>Originator:     Stijn Hoop
>Release:        FreeBSD 6.0-RELEASE i386

System: FreeBSD 6.0-RELEASE #1: Sun Nov 27 14:48:26 CET 2005 stijn at i386


- This machine panics every night on the daily maintenance time, with the
  following backtrace (hand transcribed because of a lack of serial cable):

Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x0
fault code              = supervisor write, page not present
instruction pointer     = 0x20:0xc0711a41
stack pointer           = 0x28:0xe3245ccc
frame pointer           = 0x28:0xe3245cd8
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 3 (g_up)
[thread pid 3 tid 100021 ]
Stopped at      gv_drive_done+0x29:     movl    %ebx,0(%eax)
db> bt
Tracing pid 3 tid 100021 td 0xc1e8f300
gv_drive_done(c6bcc630) at gv_drive_done+0x29
biodone(c6bcc630) at biodone+0x8b
g_io_schedule_up(c1e8f300) at g_io_schedule_up+0x86
g_up_procbody(0,e3245d38) at g_up_procbody+0x6e
fork_exit(c0482c04,0,e3245d38) at fork_exit+0x70
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe3245d6c, ebp = 0 ---

  Luckily I was able to obtain a crash dump, which yielded the following
  information (after loading the geom_vinum symbols from a debug module):

#0  doadump () at pcpu.h:165
#1  0xc0431953 in db_fncall (dummy1=-1066867712, dummy2=0, dummy3=-1067549157,
    dummy4=0xe3245af8 "$[$ã\210e^À\020[$ã\024[$ã\220\a")
    at /net/freebsd/6.0-SECURITY/src/sys/ddb/db_command.c:492
#2  0xc0431758 in db_command (last_cmdp=0xc0667c04, cmd_table=0x0,
    aux_cmd_tablep=0xc0638d58, aux_cmd_tablep_end=0xc0638d5c)
    at /net/freebsd/6.0-SECURITY/src/sys/ddb/db_command.c:350
#3  0xc0431820 in db_command_loop ()
    at /net/freebsd/6.0-SECURITY/src/sys/ddb/db_command.c:458
#4  0xc043342d in db_trap (type=12, code=0)
    at /net/freebsd/6.0-SECURITY/src/sys/ddb/db_main.c:221
#5  0xc04d38ff in kdb_trap (type=12, code=0, tf=0xe3245c8c)
    at /net/freebsd/6.0-SECURITY/src/sys/kern/subr_kdb.c:473
#6  0xc05fea4c in trap_fatal (frame=0xe3245c8c, eva=0)
    at /net/freebsd/6.0-SECURITY/src/sys/i386/i386/trap.c:822
#7  0xc05fe7bb in trap_pfault (frame=0xe3245c8c, usermode=0, eva=0)
    at /net/freebsd/6.0-SECURITY/src/sys/i386/i386/trap.c:742
#8  0xc05fe3d5 in trap (frame=
      {tf_fs = 8, tf_es = 40, tf_ds = -484179928, tf_edi = 0, tf_esi = -1038670080, tf_ebp = -484156200, tf_isp = -484156232, tf_ebx = -960707024, tf_edx = 0, tf_ecx = -1041698048, tf_eax = 0, tf_trapno = 12, tf_err = 2, tf_eip = -1066329535, tf_cs = 32, tf_eflags = 590470, tf_esp = -1066329576, tf_ss = -960707024})
    at /net/freebsd/6.0-SECURITY/src/sys/i386/i386/trap.c:432
#9  0xc05f0f8a in calltrap ()
    at /net/freebsd/6.0-SECURITY/src/sys/i386/i386/exception.s:139
#10 0xc0711a41 in gv_drive_done (bp=0xc6bcc630) at geom.h:290
#11 0xc05038a7 in biodone (bp=0xc6bcc630)
    at /net/freebsd/6.0-SECURITY/src/sys/kern/vfs_bio.c:2893
#12 0xc0482a26 in g_io_schedule_up (tp=0xc1e8f300)
    at /net/freebsd/6.0-SECURITY/src/sys/geom/geom_io.c:474
#13 0xc0482c72 in g_up_procbody ()
    at /net/freebsd/6.0-SECURITY/src/sys/geom/geom_kern.c:95
#14 0xc04a5820 in fork_exit (callout=0xc0482c04 <g_up_procbody>, arg=0x0,
    frame=0xe3245d38) at /net/freebsd/6.0-SECURITY/src/sys/kern/kern_fork.c:789
#15 0xc05f0fec in fork_trampoline ()
    at /net/freebsd/6.0-SECURITY/src/sys/i386/i386/exception.s:208
(kgdb) frame 10
#10 0xc0711a41 in gv_drive_done (bp=0xc6bcc630) at geom.h:290
290     geom.h: No such file or directory.
        in geom.h
(kgdb) print/x $eax
$3 = 0x0
(kgdb) disassemble gv_drive_done+0x41
Dump of assembler code for function gv_drive_done:
0xc0711a18 <gv_drive_done+0>:   push   %ebp
0xc0711a19 <gv_drive_done+1>:   mov    %esp,%ebp
0xc0711a1b <gv_drive_done+3>:   push   %edi
0xc0711a1c <gv_drive_done+4>:   push   %esi
0xc0711a1d <gv_drive_done+5>:   push   %ebx
0xc0711a1e <gv_drive_done+6>:   mov    0x8(%ebp),%ebx
0xc0711a21 <gv_drive_done+9>:   mov    0x44(%ebx),%eax
0xc0711a24 <gv_drive_done+12>:  mov    (%eax),%eax
0xc0711a26 <gv_drive_done+14>:  mov    0x3c(%eax),%esi
0xc0711a29 <gv_drive_done+17>:  orb    $0x1,0x2(%ebx)
0xc0711a2d <gv_drive_done+21>:  push   $0x101
0xc0711a32 <gv_drive_done+26>:  push   $0xc0643ca0
0xc0711a37 <gv_drive_done+31>:  push   $0xc
0xc0711a39 <gv_drive_done+33>:  call   0xc04b0a30 <malloc>
0xc0711a3e <gv_drive_done+38>:  add    $0xc,%esp
0xc0711a41 <gv_drive_done+41>:  mov    %ebx,(%eax)
0xc0711a43 <gv_drive_done+43>:  push   $0xf6
0xc0711a48 <gv_drive_done+48>:  push   $0xc071a752
0xc0711a4d <gv_drive_done+53>:  lea    0x7c(%esi),%ebx

  which on my system corresponds to

static void
gv_drive_done(struct bio *bp)
        struct gv_drive *d;
        struct gv_bioq *bq;

        /* Put the BIO on the worker queue again. */
        d = bp->bio_from->geom->softc;
        bp->bio_cflags |= GV_BIO_DONE;
        bq = g_malloc(sizeof(*bq), M_NOWAIT | M_ZERO);  <--- g_malloc
								returns NULL
        bq->bp = bp;
        TAILQ_INSERT_TAIL(&d->bqueue, bq, queue);

  Now, I don't know if this routine is run in an interrupt context, so
  I really wouldn't know if it is possible to remove the M_NOWAIT.

  Any thoughts?


I wish I knew; I've tried running all of 'periodic daily', the backup
routines (normally run at 4:00 AM), generating disk load using
dd if={gvinum drive} of=/dev/null, all at the same time, and the box
just copes (albeit slowly). However this is the 5th day in a row that
it crashes during daily maintenance, so something else must also be
triggering it. I'll update the PR if I can reproduce it more easily but
for now it's "only" once a day...


Would love to have one.

More information about the freebsd-bugs mailing list