vimage tool crash when deleting a jail @netisr_process_workstream_proto

[Monthadar Al Jaberi]

On Wed, Apr 25, 2012 at 11:57 PM, Marko Zec <zec at fer.hr> wrote:
> On Wednesday 25 April 2012 14:08:48 Monthadar Al Jaberi wrote:
>> Hi,
>>
>> Not sure if I should post this on virt or jail.
>>
>> I am not sure about this, but I thought it was an amd64 specific thing
>> (had posted about it some time ago), but  now I also get a panic on
>> i386. What is weird is that if I add options VNET_DEBUG to the kernel
>> config I dont get the panic! Debug output after vimage -c jid=0:
>> hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required
>> hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required
>>
>>
>> Also There seem to be a LOR after running vimage -c jid=0
>> lock order reversal:
>>  1st 0xc1037dac allprison (allprison) @ /usr/src/sys/kern/kern_jail.c:970
>>  2nd 0xc11b23f4 vnet_sysinit_sxlock (vnet_sysinit_sxlock) @
>> /usr/src/sys/net/vnet.c:615
>> KDB: stack backtrace:
>> db_trace_self_wrapper(c0e95cbb,6b636f6c,20402029,7273752f,6372732f,...)
>> at db_trace_self_wrapper+0x26
>> kdb_backtrace(c09e2ecb,c0e9974c,c1183ed0,267,e1b409f8,...) at
>> kdb_backtrace+0x2a
>> _witness_debugger(c0e9974c,c11b23f4,c0ea92c5,c7d64fc8,c0ea9418,...) at
>> _witness_debugger+0x25
>> witness_checkorder(c11b23f4,1,c0ea9418,267,0,...) at
>> witness_checkorder+0x86f _sx_slock(c11b23f4,0,c0ea9418,267,cad672e0,...) at
>> _sx_slock+0x9a
>> vnet_sysinit(cad7f000,c0ff9700,5560,cad7e028,c0fb3588,...) at
>> vnet_sysinit+0x2b vnet_alloc(cad7e028,c0e8c936,0,10,0,...) at
>> vnet_alloc+0x168
>> kern_jail_set(cad672e0,c9470d00,1,c9470d00,0,...) at kern_jail_set+0x1bb4
>> sys_jail_set(cad672e0,e1b40cec,c0edada8,c0e9a9a6,c1047f40,...) at
>> sys_jail_set+0x50
>> syscall(e1b40d28) at syscall+0x2de
>> Xint0x80_syscall() at Xint0x80_syscall+0x21
>> --- syscall (507, FreeBSD ELF32, sys_jail_set), eip = 0x280bfd5b, esp
>> = 0xbfbfe23c, ebp = 0xbfbfe328 ---
>>
>>
>>
>> This is my setup:
>> Host PC: Ubuntu 11.04 (Linux bane 2.6.38-12-generic)
>> VirtualBox: 4.1.6 r74713
>> FreeBSD gues: i386 head at 234636 (attaching kernel config)
>>
>> I compile and install /usr/src/tools/tools/vimage
>>
>> running:
>> vimage -c jid=0
>> vimage -d jid=0
>>
>> crashes the kernel (attaching core.txt.2)
>> Fatal trap 12: page fault while in kernel mode
>> cpuid = 0; apic id = 00
>> fault virtual address = 0xdeadc0e6
>> fault code            = supervisor read, page not present
>> instruction pointer   = 0x20:0xc0a78d20
>> stack pointer         = 0x28:0xc7980c48
>> frame pointer         = 0x28:0xc7980c90
>> code segment          = base 0x0, limit 0xfffff, type 0x1b
>>                       = DPL 0, pres 1, def32 1, gran 1
>> processor eflags      = interrupt enabled, resume, IOPL = 0
>> current process               = 12 (swi1: netisr 0)
>>
>> #0  doadump (textdump=0) at pcpu.h:244
>> 244   pcpu.h: No such file or directory.
>>       in pcpu.h
>> (kgdb) #0  doadump (textdump=0) at pcpu.h:244
>> #1  0xc05104b3 in db_dump (dummy=-1062761184, dummy2=0, dummy3=-1,
>>     dummy4=0xc798096c "") at /usr/src/sys/ddb/db_command.c:538
>> #2  0xc050fbd1 in db_command (last_cmdp=0xc10000dc, cmd_table=0x0,
>> dopager=1) at /usr/src/sys/ddb/db_command.c:449
>> #3  0xc050fd2a in db_command_loop () at /usr/src/sys/ddb/db_command.c:502
>> #4  0xc0511d1d in db_trap (type=12, code=0)
> at /usr/src/sys/ddb/db_main.c:231
>> #5  0xc09de976 in kdb_trap (type=12, code=0,
>> tf=0xc7980c08)
>>     at /usr/src/sys/kern/subr_kdb.c:654
>> #6  0xc0cf2eff in trap_fatal (frame=0xc7980c08, eva=3735929062)
>>     at /usr/src/sys/i386/i386/trap.c:1013
>> #7  0xc0cf32ee in trap_pfault (frame=0xc7980c08, usermode=0,
>> eva=3735929062) at /usr/src/sys/i386/i386/trap.c:936
>> #8  0xc0cf40b1 in trap (frame=0xc7980c08) at
>> /usr/src/sys/i386/i386/trap.c:546 #9  0xc0cdd8ec in calltrap () at
>> /usr/src/sys/i386/i386/exception.s:169 #10 0xc0a78d20 in swi_net
>> (arg=0xc1825880) at /usr/src/sys/net/netisr.c:805
>
> The backtrace says it's most probably this line in netisr.c:
>
>    805                 CURVNET_SET(m->m_pkthdr.rcvif->if_vnet);
>
>> #11 0xc0979d75 in
>> intr_event_execute_handlers (p=0xc7dc6598, ie=0xc7e95300) at
>> /usr/src/sys/kern/kern_intr.c:1260
>> #12 0xc097ac49 in ithread_loop (arg=0xc7e0f7a0)
>>     at /usr/src/sys/kern/kern_intr.c:1273
>> #13 0xc0976fa8 in fork_exit (callout=0xc097aba0 <ithread_loop>,
>>     arg=0xc7e0f7a0, frame=0xc7980d28) at /usr/src/sys/kern/kern_fork.c:992
>> #14 0xc0cdd994 in fork_trampoline () at
>> /usr/src/sys/i386/i386/exception.s:276 (kgdb)
>>
>>
>> It seems that it crashes on netisr_process_workstream_proto, why are
>> we even in this function? Is someone sending a packet?
>
> When netisr_process_workstream_proto() processes queued mbufs, in options
> VIMAGE builds it expects all mbufs to have m->m_pkthdr.rcvif set to a valid
> interface, so that the curvnet context could be harvested from there.
>
> Perhaps you've created a new mbuf and queued it for netisr dispatching, but
> have left m->m_pkthdr.rcvif uninitialized?

I have not done anything I am aware of, basically I called vimage the
first thing I log in freebsd. so I have not assigned it any interface
or anything :/

When I do this in single user mode I dont get this problem but I guess
that is because no interfaces are up yet :)

But I think this is weird:
VNET_ASSERT(m->m_pkthdr.rcvif != NULL,
		    ("%s:%d rcvif == NULL: m=%p", __func__, __LINE__, m));

this panic should have showed up if no one have set rcvif, but when I
check its value it is pointing to 0xdeadc0de, why?

>
> Marko



-- 
Monthadar Al Jaberi