Re: Fun with OFED and RDMA for NFS

On Sun, Sep 20, 2026 at 4:40 PM Rick Macklem <rick.macklem@gmail.com> wrote:
>
> Hi,
>
> I have been debugging client code for NFS over RDMA and it is
> going pretty well.
>
> However, with repeated cycles of..
> # make -j8 buildkernel
> where both the sources and obj are on the mount, I can get..
>
> Sep 12 03:28:24 mercat1 kernel: mlx5_1: WARN: dump_cqe:273:(pid
> 100228): dump error cqe
> Sep 12 03:28:24 mercat1 kernel: 00000000 00000000 00000000 00000000
> Sep 12 03:28:24 mercat1 syslogd: last message repeated 2 times
> Sep 12 03:28:24 mercat1 kernel: 00000000 08007806 25000907 1b0405d3
> Sep 12 03:28:24 mercat1 kernel: rpcrdma_send_done: failed opcode=0
> status=6                             (Status 6 is IB_WC_MW_BIND_ERR.)
> Sep 12 03:28:24 mercat1 kernel: rpcrdma_send_done: pg=0x400003163
> sgeaddr=0x0 len=0 lkey=0x0 num_sge=0 send_flags=0x0
>
> Obviously, it has been trashed, but it is not obvious why?
>
> I don't get it on every "make -j8 buildkernel" and never seem to get
> it for "make -j8 buildworld".
>
> I have a KASSERT() just before the ib_post_send() that sanity
> checks the WR and this KASSERT() is never triggered, so the WR
> seems ok when ib_post_send() is called.
>
> So, I thought I'd try an "options KASAN" kernel, to see if it might
> find something. What happened?
> - Nothing. I've now run 10 cycles of "make -j8 buildkernel" and none
>   of the above. (Without KASAN, it happens within the first 5 cycles.)
> So, I'm guessing that KASAN slows things down enough that it
> never occurs.
>
> I can's see anything in my code that would walk over the WR and
> the CQE that precedes it in the same structure is ok, since the done
> event handler gets called.
>
> Any ideas w.r.t. tracking this down further?
Turns out the answer is an upgrade to a post-Sep. 16 kernel.
The KASAN kernel didn't fail because it had KASAN, it didn't fail
because it was built with post-Sep. 16 main kernel sources.

I am now running a kernel built from those sources without KASAN
and it is not failing, either.

I'd guess one of the commits done by kib@ on Sep. 16 fixed the
problem (or at least made it much harder to reproduce).

Thanks for offers of help everyone, rick

>
> Thanks in advance for any suggestions, rick
> ps: The NICs are ConnectX4 and have firmware that is a few years old.
>       I've been paranoid w.r.t. asking Netperf admin for permission to update
>       their firmware, since I can test effectively now and if (not likely) newer
>       firmware broke them, I'd be stuck.