NULL point deref in ithread after halt?

[Robert Watson]

I've currently got an SMP Xeon box up and running for packet forwarding
tests using a pair of if_em interfaces.  I set net.isr.enable to cause the
network stack to dispatch in the ithreads rather than pushing all packets
through the netisr.  I ran intothe following page fault and panic on
shutdown:

tiger-2# reboot
Oct  9 12:30:01 tiger-2 reboot: rebooted by root
Oct  9 12:30:01 tiger-2 syslogd: exiting on signal 15
boot() called on cpu#1
Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...1 1 0 1 0 0 0 done
All buffers synced.
Uptime: 18h32m32s


 atal trap 12: page fault while inS hkuetrtnienlg  mdoodwen
dAcCpPuIi
i s=t r1a;y  aipriqc9 
od = R0e6b
roftaiunlgt. .v.i
 tual address   = 0x3c
fault code              = supervisor read, page not present
instruction pointer     = 0x8:0xc077dcbd
stack pointer           = 0x10:0xe93c45d8
frame pointer           = 0x10:0xe93c45e4
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 42 (irq31: em3)
[thread 100047]
Stopped at      bus_dmamap_create+0x11: cmpl    $0,0x3c(%esi)
db> trace
bus_dmamap_create(0,1,e93c4628) at bus_dmamap_create+0x11
em_encap(c551e800,c5e6d400) at em_encap+0x80
em_start_locked(c551e800) at em_start_locked+0x219
em_start(c551e800) at em_start+0x46
if_start(c551e800) at if_start+0x53
ether_output_frame(c551e800,c5e6d400,0,0,0) at ether_output_frame+0x20c
ether_output(c551e800,c5e6d400,e93c4964,c5a417bc,c5d21b00) at
ether_output+0x37c
ip_output(c5e6d400,0,e93c4960,1,0,0) at ip_output+0x698
ip_forward(c5e6d400,0) at ip_forward+0x2c8
ip_input(c5e6d400) at ip_input+0x48b
netisr_dispatch(2,c5e6d400,0,c5e60800,c5521000) at netisr_dispatch+0x50
ether_demux(c5521000,c5e6d400) at ether_demux+0x25d
ether_input(c5521000,c5e6d400) at ether_input+0x22d
em_process_receive_interrupts(c5521000,fffffffe,c5451a80,c5559540,4) at
em_proce
ss_receive_interrupts+0x2f9
em_intr(c5521000) at em_intr+0x10a
ithread_loop(c53e2d80,e93c4d48) at ithread_loop+0x159
fork_exit(c05e06ac,c53e2d80,e93c4d48) at fork_exit+0x75
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe93c4d7c, ebp = 0 ---

This would seem to suggest that after the system wasintended to have
shutdown, the interrupts from one of the interfaces are still firing,
resulting in calls into the network stack to forward packets, eventually
self-toasting.  We have a very carefully ordered and arranged startup of
the system, but I've never been entirely convinced we have an orderly and
arranged shutdown, and this could be a symptom of that.  Or, maybe
en_encap() and bus_dmamap_create() have a NULL pointer dereference and
this is a fine time to be forwarding packets still?

(gdb) l *em_encap+0x80
0xc04f1d1c is in em_encap (../../../dev/em/if_em.c:1216).
1211            }
1212
1213            /*
1214             * Map the packet for DMA.
1215             */
1216            if (bus_dmamap_create(adapter->txtag, BUS_DMA_NOWAIT,
&q.map)) {
1217                    adapter->no_tx_map_avail++;
1218                    return (ENOMEM);
1219            }
1220            error = bus_dmamap_load_mbuf(adapter->txtag, q.map,
(gdb) l *bus_dmamap_create+0x11
0xc0782bf1 is in bus_dmamap_create
(../../../i386/i386/busdma_machdep.c:343).
338     {
339             int error;
340
341             error = 0;
342
343             if (dmat->segments == NULL) {
344                     dmat->segments = (bus_dma_segment_t *)malloc(
345                         sizeof(bus_dma_segment_t) * dmat->nsegments,
M_DEVBUF,
346                         M_NOWAIT);
347                     if (dmat->segments == NULL)

Or perhaps most likely, one interface is tearing itself down, but we race
with an ithread that tries to use it?

Anyhow, thoughts welcome.

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert at fledge.watson.org      Principal Research Scientist, McAfee Research