[PATCH] ipoib: Patch for crash in icmp_error, fault trap 12

[Andreas Kempe]

Hello everyone,

We have been using IP over IB in connected mode between a Linux
machine running Void Linux and another machine running
FreeBSD 12.1 STABLE. After having initially transferred data at
expected speeds, about 5 Gbit/s, and letting the computers rest for a
while the FreeBSD machine throws transmission timeout errors. When a
new data transfer is started, the machine would complain that it
cannot send a few packets because of them being too large. After this
the kernel would panic. See example logs below:

Timing out:

> ib0: timing out; 7 sends not completed

When starting new transfers:

> ib0: packet len 32812 (> 2044) too long to send, dropping
> ib0: packet len 8248 (> 2044) too long to send, dropping

Kernel crash:

> Fatal trap 12: page fault while in kernel mode
> cpuid = 3; apic id = 03
> fault virtual address   = 0x28
> fault code              = supervisor read data, page not present
> instruction pointer     = 0x20:0xffffffff80d76edf
> stack pointer           = 0x28:0xfffffe008edbeb50
> frame pointer           = 0x28:0xfffffe008edbebb0
> code segment            = base 0x0, limit 0xfffff, type 0x1b
>                         = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags        = interrupt enabled, resume, IOPL = 0
> current process         = 0 (ipoib)
> trap number             = 12
> panic: page fault
> cpuid = 3
> time = 1578710936
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe008edbe7b0
> vpanic() at vpanic+0x17e/frame 0xfffffe008edbe810
> panic() at panic+0x43/frame 0xfffffe008edbe870
> trap_pfault() at trap_pfault/frame 0xfffffe008edbe8e0
> trap_pfault() at trap_pfault+0x4f/frame 0xfffffe008edbe950
> trap() at trap+0x288/frame 0xfffffe008edbea80
> calltrap() at calltrap+0x8/frame 0xfffffe008edbea80
> --- trap 0xc, rip = 0xffffffff80d76edf, rsp = 0xfffffe008edbeb50, rbp = 0xfffffe008edbebb0 ---
> icmp_error() at icmp_error+0x2f/frame 0xfffffe008edbebb0
> ipoib_cm_mb_reap() at ipoib_cm_mb_reap+0x154/frame 0xfffffe008edbec00
> linux_work_fn() at linux_work_fn+0xfc/frame 0xfffffe008edbec60
> taskqueue_run_locked() at taskqueue_run_locked+0x144/frame 0xfffffe008edbecc0
> taskqueue_thread_loop() at taskqueue_thread_loop+0xd3/frame 0xfffffe008edbecf0
> fork_exit() at fork_exit+0x7e/frame 0xfffffe008edbed30
> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008edbed30
> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
> KDB: enter: panic


The 0x28 access that causes the trap is caused by the error statistics
if statement at the top of icmp_error in sys/netinet/ip_icmp.c:

> if (type != ICMP_REDIRECT)
>         ICMPSTAT_INC(icps_error);

ICMPSTAT_INC needs the VIMAGE for the current thread to be set.
Its calling function, i.e. ipoib_cm_mb_reap in
sys/ofed/drivers/infiniband/ulp/ipoib/ipoib_cm.c, is scheduled in its
own thread when the MTU size is too large in ipoib_cm_send. It then
calls ipoib_cm_mb_too_long, which in turn schedules ipoib_cm_mb_reap
(both functions are located in sys/ofed/drivers/infiniband/ulp/ipoib/ipoib_cm.c).

The attached patch fixes the issue by setting the VIMAGE for the
thread in ipoib_cm_mb_reap.

We still have not investigated what causes the MTU to be perceived as
too large, but our machine stopped crashing after applying the patch.

Cordially,
Andreas Kempe
-------------- next part --------------
A non-text attachment was scrubbed...
Name: set-vnet-in-ipoib_cm_mb_reap.patch
Type: text/x-diff
Size: 588 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-infiniband/attachments/20200111/9a1e02f9/attachment.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-infiniband/attachments/20200111/9a1e02f9/attachment.sig>