VNET + pf => panic [Was: Panic when tearing down a VNET jail; pf mentioned in stack trace]

Sun Nov 20 11:11:17 UTC 2016

Hello,

I had another panic, this time not related to stopping a jail, but
also mentioning pf in the stack trace. Therefore the previously
mentioned correlation was not causation -- this crash must be
connected with simply combining pf and VNET, not with tearing down the
VNET interface.

I had two crashes in about one hour. I then put `set skip on vnet0:1`
in pf.conf, and have not seen a panic since then, even with relatively
heavy network traffic (continous 60Mbit/s since my last email, and an
evening of 1Gbit/s).

The second stack trace follows:

Nov 19 15:19:04 oresme kernel: Fatal trap 12: page fault while in kernel mode
Nov 19 15:19:04 oresme kernel: cpuid = 1; apic id = 01
Nov 19 15:19:04 oresme kernel: fault virtual address    = 0x400
Nov 19 15:19:04 oresme kernel: fault code               = supervisor
write data, page not present
Nov 19 15:19:04 oresme kernel: instruction pointer      =
0x20:0xffffffff8263eaa1
Nov 19 15:19:04 oresme kernel: stack pointer            =
0x28:0xfffffe085315c870
Nov 19 15:19:04 oresme kernel: frame pointer            =
0x28:0xfffffe085315c8e0
Nov 19 15:19:04 oresme kernel: code segment             = base 0x0,
limit 0xfffff, type 0x1b
Nov 19 15:19:04 oresme kernel: = DPL 0, pres 1, long 1, def32 0, gran 1
Nov 19 15:19:04 oresme kernel: processor eflags = interrupt enabled,
resume, IOPL = 0
Nov 19 15:19:04 oresme kernel: current process          = 477 (pf purge)
Nov 19 15:19:04 oresme kernel: trap number              = 12
Nov 19 15:19:04 oresme kernel: panic: page fault
Nov 19 15:19:04 oresme kernel: cpuid = 1
Nov 19 15:19:04 oresme kernel: KDB: stack backtrace:
Nov 19 15:19:04 oresme kernel: #0 0xffffffff80aa8787 at kdb_backtrace+0x67
Nov 19 15:19:04 oresme kernel: #1 0xffffffff80a5d632 at vpanic+0x182
Nov 19 15:19:04 oresme kernel: #2 0xffffffff80a5d4a3 at panic+0x43
Nov 19 15:19:04 oresme kernel: #3 0xffffffff80f3cd51 at trap_fatal+0x351
Nov 19 15:19:04 oresme kernel: #4 0xffffffff80f3cf43 at trap_pfault+0x1e3
Nov 19 15:19:04 oresme kernel: #5 0xffffffff80f3c4ec at trap+0x26c
Nov 19 15:19:04 oresme kernel: #6 0xffffffff80f1f521 at calltrap+0x8
Nov 19 15:19:04 oresme kernel: #7 0xffffffff8263e32d at
pf_purge_expired_states+0x12d
Nov 19 15:19:04 oresme kernel: #8 0xffffffff8263e1bb at pf_purge_thread+0x13b
Nov 19 15:19:04 oresme kernel: #9 0xffffffff80a13e85 at fork_exit+0x85
Nov 19 15:19:04 oresme kernel: #10 0xffffffff80f1fa5e at fork_trampoline+0xe

Do you need any other information?

Thanks!

Kamila

On Sat, Nov 19, 2016 at 3:01 PM, Kamila Součková <kamila at ksp.sk> wrote:
> Hello,
>
> (if this is not the right mailing list, please reroute this email -- I
> am not sure where to post VNET-related stuff.)
>
> I experienced a panic when stopping an iocage-managed jail with VNET.
> Some information:
>
> - The host (which is a physical machine) panicked after calling `iocage stop`.
> - The host has pf enabled and active, the jail does not.
> - The standard iocage configuration for VNET is used, i.e. the host
> part of the epair device is bridged to the local network.
> - It has only happened to me once in about 10 tries, so I assume it
> must be a race condition.
> - The stack trace is attached below.
>
> What could be the problem? How can I help debug it? (I do not know
> anything about FreeBSD internals, yet.)
>
> Thank you!
>
> Kamila
>
> -------------------------------------------
>
> trace:
>
> Nov 19 14:12:04 oresme kernel: Fatal trap 12: page fault while in kernel mode
> Nov 19 14:12:04 oresme kernel: cpuid = 5; apic id = 05
> Nov 19 14:12:04 oresme kernel: fault virtual address    = 0x420
> Nov 19 14:12:04 oresme kernel: fault code               = supervisor
> read data, page not present
> Nov 19 14:12:04 oresme kernel: instruction pointer      =
> 0x20:0xffffffff826657a9
> Nov 19 14:12:04 oresme kernel: stack pointer            =
> 0x28:0xfffffe0852d63340
> Nov 19 14:12:04 oresme kernel: frame pointer            =
> 0x28:0xfffffe0852d633b0
> Nov 19 14:12:04 oresme kernel: code segment             = base 0x0,
> limit 0xfffff, type 0x1b
> Nov 19 14:12:04 oresme kernel: = DPL 0, pres 1, long 1, def32 0, gran 1
> Nov 19 14:12:04 oresme kernel: processor eflags = interrupt enabled,
> resume, IOPL = 0
> Nov 19 14:12:04 oresme kernel: current process          = 12 (irq272:
> igb1:que 1)
> Nov 19 14:12:04 oresme kernel: trap number              = 12
> Nov 19 14:12:04 oresme kernel: panic: page fault
> Nov 19 14:12:04 oresme kernel: cpuid = 5
> Nov 19 14:12:04 oresme kernel: KDB: stack backtrace:
> Nov 19 14:12:04 oresme kernel: #0 0xffffffff80aa8787 at kdb_backtrace+0x67
> Nov 19 14:12:04 oresme kernel: #1 0xffffffff80a5d632 at vpanic+0x182
> Nov 19 14:12:04 oresme kernel: #2 0xffffffff80a5d4a3 at panic+0x43
> Nov 19 14:12:04 oresme kernel: #3 0xffffffff80f3cd51 at trap_fatal+0x351
> Nov 19 14:12:04 oresme kernel: #4 0xffffffff80f3cf43 at trap_pfault+0x1e3
> Nov 19 14:12:04 oresme kernel: #5 0xffffffff80f3c4ec at trap+0x26c
> Nov 19 14:12:04 oresme kernel: #6 0xffffffff80f1f521 at calltrap+0x8
> Nov 19 14:12:04 oresme kernel: #7 0xffffffff82641acc at pf_test+0xfdc
> Nov 19 14:12:04 oresme kernel: #8 0xffffffff8265408d at pf_check_in+0x1d
> Nov 19 14:12:04 oresme kernel: #9 0xffffffff80b820f3 at pfil_run_hooks+0x83
> Nov 19 14:12:04 oresme kernel: #10 0xffffffff80be9a5f at ip_input+0x42f
> Nov 19 14:12:04 oresme kernel: #11 0xffffffff80b8107f at
> netisr_dispatch_src+0xff
> Nov 19 14:12:04 oresme kernel: #12 0xffffffff80b6915a at ether_demux+0x13a
> Nov 19 14:12:04 oresme kernel: #13 0xffffffff80b69e75 at ether_nh_input+0x345
> Nov 19 14:12:04 oresme kernel: #14 0xffffffff80b8107f at
> netisr_dispatch_src+0xff
> Nov 19 14:12:04 oresme kernel: #15 0xffffffff80b69414 at ether_input+0x54
> Nov 19 14:12:04 oresme kernel: #16 0xffffffff80553b9c at igb_rxeof+0x7fc
> Nov 19 14:12:04 oresme kernel: #17 0xffffffff80552def at igb_msix_que+0x18f