From nobody Wed Feb 21 06:01:07 2024 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TflyC45Ymz5BX7d; Wed, 21 Feb 2024 06:01:07 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TflyC3LYmz4m6Q; Wed, 21 Feb 2024 06:01:07 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1708495267; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=F+yLiVnR/gGdmx7wP7K1VC9BADDAlT3ZB+rss7IV09c=; b=fFIODAjHA9JI2/CHShSPj7Bw8trY8TNdR5nJBbkUoCrjc+ejD510PRJ9rB+0dbB1pge0im voZhQnRZtdulpcwl9s7xjQOy8GzWte/E9w51GnhYzVE9A/2cWo4sZdtLPzLefqXr68KhSV c8I8a389IIrJx9q5E57WDhVUWdUrwh/IBDgQ801lPbGx3i5lqbn0RbxQRNmvmhlseQWkLk caBbu4Yump9pc2cb+00/de0u9/caC2R6iGXigppPXSCLH6ww6LSUmDKSO57GWj5Ao2at7O U6TL9PRaadbhHO2sYSvfNggDs4p3+ugIMIneoNpp7xwE+vwPcoYOJ4asLhH75A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1708495267; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=F+yLiVnR/gGdmx7wP7K1VC9BADDAlT3ZB+rss7IV09c=; b=P5bGpGgk+GbHqIBoLJ9tHPfNyYbNLx+l4e0mzvxrcvzNPMw8GLDcrKSKL6zemoDaCRlwTi /cG1rwV4CDSGEz5Mm+gfp3D8SgvN/ekSsTRpCewJD11ZOQ92pAzUL9ABltCJSEs+drHweG ODzS7/GkdkQsOxCFiF5sJP/jeFcvQD1VGucgvosHZAdoaL0H3EQHQJQXsPa2hWErzVO8qr wRWs0Szyi2cZ/p6GyL8ktqUAseCqY8kRzdUmht3kio7DrOyuP4zZbO8Yorf2QCKBEI4Zkw Gyn0yh6P443FdxJ41cxjZxgPun/iYStKH+r+1TCyG084jZ0gZ2U8HkqNtKcYjA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1708495267; a=rsa-sha256; cv=none; b=QyRoGijOQ62tKKB7rO1/4u5H1HJAo6ws3frFYphOmnMvHaELTxc0VpqLg6sLTKsGo2m2eA YoeWnqBRfKe7Ch2swbL/ZLAeGpguIGSrb/oOy1TU2PeyHuZmegoHgam79PJOuZJ6D7TfNy goGyUaOBntqbGpNI6Klxt81zv2PiRClXqy5pBPqD2PXnVFDMxJ8oFrlT1olLa7U7sLCyMh zgzd4MIl2L84+7eZ68Cv/41bS6ipkKwdcssinmbI/lD11VsU4apveV6qiC07EvtpElUasK Iutk3tHB7h7Nb3Kzh7BLNl0kIe8drIf9VHw6GsvbtwDk2oczESwwQvPEBsmGCw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TflyC2PDXzrND; Wed, 21 Feb 2024 06:01:07 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 41L617ST068318; Wed, 21 Feb 2024 06:01:07 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 41L617UF068315; Wed, 21 Feb 2024 06:01:07 GMT (envelope-from git) Date: Wed, 21 Feb 2024 06:01:07 GMT Message-Id: <202402210601.41L617UF068315@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Eric Joyner Subject: git: a4510dafa804 - stable/13 - irdma(4): Upgrade to 1.2.36-k List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: erj X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: a4510dafa8045bfa521303db7f23123a47d8749f Auto-Submitted: auto-generated The branch stable/13 has been updated by erj: URL: https://cgit.FreeBSD.org/src/commit/?id=a4510dafa8045bfa521303db7f23123a47d8749f commit a4510dafa8045bfa521303db7f23123a47d8749f Author: Bartosz Sobczak AuthorDate: 2024-02-16 21:55:08 +0000 Commit: Eric Joyner CommitDate: 2024-02-21 06:00:00 +0000 irdma(4): Upgrade to 1.2.36-k Update Intel irdma driver to version 1.2.36-k. Notable changes: - Start using ib_sge directly instead of irdma_sge - Turn off flush completion generator for libirdma - Minor formatting changes Signed-off-by: Bartosz Sobczak Signed-off-by: Eric Joyner Reviewed by: erj@ Sponsored by: Intel Corporation Differential Revision: https://reviews.freebsd.org/D43567 (cherry picked from commit 5b5f7d0e77a9eee73eb5d596f43aef4e1a3674d8) --- contrib/ofed/libirdma/abi.h | 8 +- contrib/ofed/libirdma/irdma-abi.h | 11 ++ contrib/ofed/libirdma/irdma.h | 1 - contrib/ofed/libirdma/irdma_defs.h | 13 --- contrib/ofed/libirdma/irdma_uk.c | 106 ++++++++---------- contrib/ofed/libirdma/irdma_umain.c | 2 +- contrib/ofed/libirdma/irdma_umain.h | 2 - contrib/ofed/libirdma/irdma_user.h | 198 ++-------------------------------- contrib/ofed/libirdma/irdma_uverbs.c | 89 +++++---------- contrib/ofed/libirdma/osdep.h | 1 - sys/dev/irdma/fbsd_kcompat.c | 5 +- sys/dev/irdma/fbsd_kcompat.h | 25 ++--- sys/dev/irdma/icrdma.c | 2 +- sys/dev/irdma/icrdma_hw.c | 7 +- sys/dev/irdma/icrdma_hw.h | 3 +- sys/dev/irdma/irdma-abi.h | 11 ++ sys/dev/irdma/irdma.h | 2 +- sys/dev/irdma/irdma_cm.c | 47 ++++---- sys/dev/irdma/irdma_cm.h | 2 +- sys/dev/irdma/irdma_ctrl.c | 99 ++++++++++------- sys/dev/irdma/irdma_defs.h | 3 - sys/dev/irdma/irdma_hw.c | 43 +++++--- sys/dev/irdma/irdma_kcompat.c | 96 +++++++---------- sys/dev/irdma/irdma_main.h | 13 ++- sys/dev/irdma/irdma_pble.h | 2 +- sys/dev/irdma/irdma_protos.h | 2 +- sys/dev/irdma/irdma_puda.c | 8 +- sys/dev/irdma/irdma_type.h | 29 +++-- sys/dev/irdma/irdma_uda.c | 2 +- sys/dev/irdma/irdma_uda.h | 3 +- sys/dev/irdma/irdma_uk.c | 80 +++++++------- sys/dev/irdma/irdma_user.h | 35 +++--- sys/dev/irdma/irdma_utils.c | 53 +++------ sys/dev/irdma/irdma_verbs.c | 202 ++++++++--------------------------- sys/dev/irdma/irdma_verbs.h | 4 +- sys/dev/irdma/irdma_ws.c | 4 +- sys/dev/irdma/irdma_ws.h | 2 +- 37 files changed, 424 insertions(+), 791 deletions(-) diff --git a/contrib/ofed/libirdma/abi.h b/contrib/ofed/libirdma/abi.h index 1d84fbc747f6..a3e159828bec 100644 --- a/contrib/ofed/libirdma/abi.h +++ b/contrib/ofed/libirdma/abi.h @@ -1,7 +1,7 @@ /*- * SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB * - * Copyright (C) 2019 - 2022 Intel Corporation + * Copyright (C) 2019 - 2023 Intel Corporation * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -85,6 +85,7 @@ struct irdma_ucreate_qp { struct ibv_create_qp ibv_cmd; __aligned_u64 user_wqe_bufs; __aligned_u64 user_compl_ctx; + __aligned_u64 comp_mask; }; struct irdma_ucreate_qp_resp { @@ -97,6 +98,9 @@ struct irdma_ucreate_qp_resp { __u8 lsmm; __u8 rsvd; __u32 qp_caps; + __aligned_u64 comp_mask; + __u8 start_wqe_idx; + __u8 rsvd2[7]; }; struct irdma_umodify_qp_resp { @@ -137,6 +141,8 @@ struct irdma_get_context_resp { __u8 hw_rev; __u8 rsvd2; __aligned_u64 comp_mask; + __u16 min_hw_wq_size; + __u8 rsvd3[6]; }; struct irdma_ureg_mr { diff --git a/contrib/ofed/libirdma/irdma-abi.h b/contrib/ofed/libirdma/irdma-abi.h index 8c04dcce2e8b..ae805919ea55 100644 --- a/contrib/ofed/libirdma/irdma-abi.h +++ b/contrib/ofed/libirdma/irdma-abi.h @@ -54,6 +54,11 @@ enum irdma_memreg_type { enum { IRDMA_ALLOC_UCTX_USE_RAW_ATTR = 1 << 0, + IRDMA_ALLOC_UCTX_MIN_HW_WQ_SIZE = 1 << 1, +}; + +enum { + IRDMA_CREATE_QP_USE_START_WQE_IDX = 1 << 0, }; struct irdma_alloc_ucontext_req { @@ -82,6 +87,8 @@ struct irdma_alloc_ucontext_resp { __u8 hw_rev; __u8 rsvd2; __aligned_u64 comp_mask; + __u16 min_hw_wq_size; + __u8 rsvd3[6]; }; struct irdma_alloc_pd_resp { @@ -101,6 +108,7 @@ struct irdma_create_cq_req { struct irdma_create_qp_req { __aligned_u64 user_wqe_bufs; __aligned_u64 user_compl_ctx; + __aligned_u64 comp_mask; }; struct irdma_mem_reg_req { @@ -130,6 +138,9 @@ struct irdma_create_qp_resp { __u8 lsmm; __u8 rsvd; __u32 qp_caps; + __aligned_u64 comp_mask; + __u8 start_wqe_idx; + __u8 rsvd2[7]; }; struct irdma_modify_qp_resp { diff --git a/contrib/ofed/libirdma/irdma.h b/contrib/ofed/libirdma/irdma.h index 0e2a5189acd1..f4a5a4796f82 100644 --- a/contrib/ofed/libirdma/irdma.h +++ b/contrib/ofed/libirdma/irdma.h @@ -46,7 +46,6 @@ enum irdma_vers { IRDMA_GEN_RSVD = 0, IRDMA_GEN_1 = 1, IRDMA_GEN_2 = 2, - IRDMA_GEN_MAX = 2, }; struct irdma_uk_attrs { diff --git a/contrib/ofed/libirdma/irdma_defs.h b/contrib/ofed/libirdma/irdma_defs.h index da9eea472568..39d4e7772c31 100644 --- a/contrib/ofed/libirdma/irdma_defs.h +++ b/contrib/ofed/libirdma/irdma_defs.h @@ -456,19 +456,6 @@ enum irdma_qp_wqe_size { IRDMA_WQE_SIZE_256 = 256, }; -enum irdma_ws_op_type { - IRDMA_WS_OP_TYPE_NODE = 0, - IRDMA_WS_OP_TYPE_LEAF_NODE_GROUP, -}; - -enum irdma_ws_rate_limit_flags { - IRDMA_WS_RATE_LIMIT_FLAGS_VALID = 0x1, - IRDMA_WS_NO_RDMA_RATE_LIMIT = 0x2, - IRDMA_WS_LEAF_NODE_IS_PART_GROUP = 0x4, - IRDMA_WS_TREE_RATE_LIMITING = 0x8, - IRDMA_WS_PACING_CONTROL = 0x10, -}; - /** * set_64bit_val - set 64 bit value to hw wqe * @wqe_words: wqe addr to write diff --git a/contrib/ofed/libirdma/irdma_uk.c b/contrib/ofed/libirdma/irdma_uk.c index 5fa9d792745f..115c5f0a27f0 100644 --- a/contrib/ofed/libirdma/irdma_uk.c +++ b/contrib/ofed/libirdma/irdma_uk.c @@ -45,16 +45,16 @@ * @valid: The wqe valid */ static void -irdma_set_fragment(__le64 * wqe, u32 offset, struct irdma_sge *sge, +irdma_set_fragment(__le64 * wqe, u32 offset, struct ibv_sge *sge, u8 valid) { if (sge) { set_64bit_val(wqe, offset, - FIELD_PREP(IRDMAQPSQ_FRAG_TO, sge->tag_off)); + FIELD_PREP(IRDMAQPSQ_FRAG_TO, sge->addr)); set_64bit_val(wqe, offset + IRDMA_BYTE_8, FIELD_PREP(IRDMAQPSQ_VALID, valid) | - FIELD_PREP(IRDMAQPSQ_FRAG_LEN, sge->len) | - FIELD_PREP(IRDMAQPSQ_FRAG_STAG, sge->stag)); + FIELD_PREP(IRDMAQPSQ_FRAG_LEN, sge->length) | + FIELD_PREP(IRDMAQPSQ_FRAG_STAG, sge->lkey)); } else { set_64bit_val(wqe, offset, 0); set_64bit_val(wqe, offset + IRDMA_BYTE_8, @@ -71,14 +71,14 @@ irdma_set_fragment(__le64 * wqe, u32 offset, struct irdma_sge *sge, */ static void irdma_set_fragment_gen_1(__le64 * wqe, u32 offset, - struct irdma_sge *sge, u8 valid) + struct ibv_sge *sge, u8 valid) { if (sge) { set_64bit_val(wqe, offset, - FIELD_PREP(IRDMAQPSQ_FRAG_TO, sge->tag_off)); + FIELD_PREP(IRDMAQPSQ_FRAG_TO, sge->addr)); set_64bit_val(wqe, offset + IRDMA_BYTE_8, - FIELD_PREP(IRDMAQPSQ_GEN1_FRAG_LEN, sge->len) | - FIELD_PREP(IRDMAQPSQ_GEN1_FRAG_STAG, sge->stag)); + FIELD_PREP(IRDMAQPSQ_GEN1_FRAG_LEN, sge->length) | + FIELD_PREP(IRDMAQPSQ_GEN1_FRAG_STAG, sge->lkey)); } else { set_64bit_val(wqe, offset, 0); set_64bit_val(wqe, offset + IRDMA_BYTE_8, 0); @@ -209,8 +209,7 @@ irdma_qp_push_wqe(struct irdma_qp_uk *qp, __le64 * wqe, u16 quanta, if (IRDMA_RING_CURRENT_HEAD(qp->initial_ring) != IRDMA_RING_CURRENT_TAIL(qp->sq_ring) && !qp->push_mode) { - if (post_sq) - irdma_uk_qp_post_wr(qp); + irdma_uk_qp_post_wr(qp); } else { push = (__le64 *) ((uintptr_t)qp->push_wqe + (wqe_idx & 0x7) * 0x20); @@ -338,7 +337,7 @@ irdma_uk_rdma_write(struct irdma_qp_uk *qp, struct irdma_post_sq_info *info, return EINVAL; for (i = 0; i < op_info->num_lo_sges; i++) - total_size += op_info->lo_sg_list[i].len; + total_size += op_info->lo_sg_list[i].length; read_fence |= info->read_fence; @@ -357,7 +356,7 @@ irdma_uk_rdma_write(struct irdma_qp_uk *qp, struct irdma_post_sq_info *info, qp->sq_wrtrk_array[wqe_idx].signaled = info->signaled; set_64bit_val(wqe, IRDMA_BYTE_16, - FIELD_PREP(IRDMAQPSQ_FRAG_TO, op_info->rem_addr.tag_off)); + FIELD_PREP(IRDMAQPSQ_FRAG_TO, op_info->rem_addr.addr)); if (info->imm_data_valid) { set_64bit_val(wqe, IRDMA_BYTE_0, @@ -386,7 +385,7 @@ irdma_uk_rdma_write(struct irdma_qp_uk *qp, struct irdma_post_sq_info *info, ++addl_frag_cnt; } - hdr = FIELD_PREP(IRDMAQPSQ_REMSTAG, op_info->rem_addr.stag) | + hdr = FIELD_PREP(IRDMAQPSQ_REMSTAG, op_info->rem_addr.lkey) | FIELD_PREP(IRDMAQPSQ_OPCODE, info->op_type) | FIELD_PREP(IRDMAQPSQ_IMMDATAFLAG, info->imm_data_valid) | FIELD_PREP(IRDMAQPSQ_REPORTRTT, info->report_rtt) | @@ -437,7 +436,7 @@ irdma_uk_rdma_read(struct irdma_qp_uk *qp, struct irdma_post_sq_info *info, return EINVAL; for (i = 0; i < op_info->num_lo_sges; i++) - total_size += op_info->lo_sg_list[i].len; + total_size += op_info->lo_sg_list[i].length; ret_code = irdma_fragcnt_to_quanta_sq(op_info->num_lo_sges, &quanta); if (ret_code) @@ -475,8 +474,8 @@ irdma_uk_rdma_read(struct irdma_qp_uk *qp, struct irdma_post_sq_info *info, ++addl_frag_cnt; } set_64bit_val(wqe, IRDMA_BYTE_16, - FIELD_PREP(IRDMAQPSQ_FRAG_TO, op_info->rem_addr.tag_off)); - hdr = FIELD_PREP(IRDMAQPSQ_REMSTAG, op_info->rem_addr.stag) | + FIELD_PREP(IRDMAQPSQ_FRAG_TO, op_info->rem_addr.addr)); + hdr = FIELD_PREP(IRDMAQPSQ_REMSTAG, op_info->rem_addr.lkey) | FIELD_PREP(IRDMAQPSQ_REPORTRTT, (info->report_rtt ? 1 : 0)) | FIELD_PREP(IRDMAQPSQ_ADDFRAGCNT, addl_frag_cnt) | FIELD_PREP(IRDMAQPSQ_OPCODE, @@ -525,7 +524,7 @@ irdma_uk_send(struct irdma_qp_uk *qp, struct irdma_post_sq_info *info, return EINVAL; for (i = 0; i < op_info->num_sges; i++) - total_size += op_info->sg_list[i].len; + total_size += op_info->sg_list[i].length; if (info->imm_data_valid) frag_cnt = op_info->num_sges + 1; @@ -620,15 +619,15 @@ irdma_set_mw_bind_wqe_gen_1(__le64 * wqe, * @polarity: compatibility parameter */ static void -irdma_copy_inline_data_gen_1(u8 *wqe, struct irdma_sge *sge_list, +irdma_copy_inline_data_gen_1(u8 *wqe, struct ibv_sge *sge_list, u32 num_sges, u8 polarity) { u32 quanta_bytes_remaining = 16; u32 i; for (i = 0; i < num_sges; i++) { - u8 *cur_sge = (u8 *)(uintptr_t)sge_list[i].tag_off; - u32 sge_len = sge_list[i].len; + u8 *cur_sge = (u8 *)(uintptr_t)sge_list[i].addr; + u32 sge_len = sge_list[i].length; while (sge_len) { u32 bytes_copied; @@ -683,7 +682,7 @@ irdma_set_mw_bind_wqe(__le64 * wqe, * @polarity: polarity of wqe valid bit */ static void -irdma_copy_inline_data(u8 *wqe, struct irdma_sge *sge_list, +irdma_copy_inline_data(u8 *wqe, struct ibv_sge *sge_list, u32 num_sges, u8 polarity) { u8 inline_valid = polarity << IRDMA_INLINE_VALID_S; @@ -694,8 +693,8 @@ irdma_copy_inline_data(u8 *wqe, struct irdma_sge *sge_list, wqe += 8; for (i = 0; i < num_sges; i++) { - u8 *cur_sge = (u8 *)(uintptr_t)sge_list[i].tag_off; - u32 sge_len = sge_list[i].len; + u8 *cur_sge = (u8 *)(uintptr_t)sge_list[i].addr; + u32 sge_len = sge_list[i].length; while (sge_len) { u32 bytes_copied; @@ -775,7 +774,7 @@ irdma_uk_inline_rdma_write(struct irdma_qp_uk *qp, return EINVAL; for (i = 0; i < op_info->num_lo_sges; i++) - total_size += op_info->lo_sg_list[i].len; + total_size += op_info->lo_sg_list[i].length; if (unlikely(total_size > qp->max_inline_data)) return EINVAL; @@ -788,9 +787,9 @@ irdma_uk_inline_rdma_write(struct irdma_qp_uk *qp, qp->sq_wrtrk_array[wqe_idx].signaled = info->signaled; read_fence |= info->read_fence; set_64bit_val(wqe, IRDMA_BYTE_16, - FIELD_PREP(IRDMAQPSQ_FRAG_TO, op_info->rem_addr.tag_off)); + FIELD_PREP(IRDMAQPSQ_FRAG_TO, op_info->rem_addr.addr)); - hdr = FIELD_PREP(IRDMAQPSQ_REMSTAG, op_info->rem_addr.stag) | + hdr = FIELD_PREP(IRDMAQPSQ_REMSTAG, op_info->rem_addr.lkey) | FIELD_PREP(IRDMAQPSQ_OPCODE, info->op_type) | FIELD_PREP(IRDMAQPSQ_INLINEDATALEN, total_size) | FIELD_PREP(IRDMAQPSQ_REPORTRTT, info->report_rtt ? 1 : 0) | @@ -846,7 +845,7 @@ irdma_uk_inline_send(struct irdma_qp_uk *qp, return EINVAL; for (i = 0; i < op_info->num_sges; i++) - total_size += op_info->sg_list[i].len; + total_size += op_info->sg_list[i].length; if (unlikely(total_size > qp->max_inline_data)) return EINVAL; @@ -911,7 +910,7 @@ irdma_uk_stag_local_invalidate(struct irdma_qp_uk *qp, u64 hdr; u32 wqe_idx; bool local_fence = false; - struct irdma_sge sge = {0}; + struct ibv_sge sge = {0}; u16 quanta = IRDMA_QP_WQE_MIN_QUANTA; info->push_wqe = qp->push_db ? true : false; @@ -922,7 +921,7 @@ irdma_uk_stag_local_invalidate(struct irdma_qp_uk *qp, if (!wqe) return ENOSPC; - sge.stag = op_info->target_stag; + sge.lkey = op_info->target_stag; qp->wqe_ops.iw_set_fragment(wqe, IRDMA_BYTE_0, &sge, 0); set_64bit_val(wqe, IRDMA_BYTE_16, 0); @@ -1436,8 +1435,7 @@ irdma_uk_cq_poll_cmpl(struct irdma_cq_uk *cq, IRDMA_RING_MOVE_TAIL(cq->cq_ring); set_64bit_val(cq->shadow_area, IRDMA_BYTE_0, IRDMA_RING_CURRENT_HEAD(cq->cq_ring)); - memset(info, 0, - sizeof(struct irdma_cq_poll_info)); + memset(info, 0, sizeof(*info)); return irdma_uk_cq_poll_cmpl(cq, info); } } @@ -1510,7 +1508,6 @@ exit: if (pring && IRDMA_RING_MORE_WORK(*pring)) move_cq_head = false; } - if (move_cq_head) { IRDMA_RING_MOVE_HEAD_NOCHECK(cq->cq_ring); if (!IRDMA_RING_CURRENT_HEAD(cq->cq_ring)) @@ -1591,10 +1588,12 @@ irdma_get_wqe_shift(struct irdma_uk_attrs *uk_attrs, u32 sge, int irdma_get_sqdepth(struct irdma_uk_attrs *uk_attrs, u32 sq_size, u8 shift, u32 *sqdepth) { + u32 min_size = (u32)uk_attrs->min_hw_wq_size << shift; + *sqdepth = irdma_round_up_wq((sq_size << shift) + IRDMA_SQ_RSVD); - if (*sqdepth < ((u32)uk_attrs->min_hw_wq_size << shift)) - *sqdepth = uk_attrs->min_hw_wq_size << shift; + if (*sqdepth < min_size) + *sqdepth = min_size; else if (*sqdepth > uk_attrs->max_hw_wq_quanta) return EINVAL; @@ -1608,10 +1607,12 @@ irdma_get_sqdepth(struct irdma_uk_attrs *uk_attrs, u32 sq_size, u8 shift, u32 *s int irdma_get_rqdepth(struct irdma_uk_attrs *uk_attrs, u32 rq_size, u8 shift, u32 *rqdepth) { + u32 min_size = (u32)uk_attrs->min_hw_wq_size << shift; + *rqdepth = irdma_round_up_wq((rq_size << shift) + IRDMA_RQ_RSVD); - if (*rqdepth < ((u32)uk_attrs->min_hw_wq_size << shift)) - *rqdepth = uk_attrs->min_hw_wq_size << shift; + if (*rqdepth < min_size) + *rqdepth = min_size; else if (*rqdepth > uk_attrs->max_hw_rq_quanta) return EINVAL; @@ -1644,41 +1645,16 @@ irdma_setup_connection_wqes(struct irdma_qp_uk *qp, { u16 move_cnt = 1; - if (qp->uk_attrs->feature_flags & IRDMA_FEATURE_RTS_AE) + if (info->start_wqe_idx) + move_cnt = info->start_wqe_idx; + else if (qp->uk_attrs->feature_flags & IRDMA_FEATURE_RTS_AE) move_cnt = 3; - qp->conn_wqes = move_cnt; IRDMA_RING_MOVE_HEAD_BY_COUNT_NOCHECK(qp->sq_ring, move_cnt); IRDMA_RING_MOVE_TAIL_BY_COUNT(qp->sq_ring, move_cnt); IRDMA_RING_MOVE_HEAD_BY_COUNT_NOCHECK(qp->initial_ring, move_cnt); } -/** - * irdma_uk_calc_shift_wq - calculate WQE shift for both SQ and RQ - * @ukinfo: qp initialization info - * @sq_shift: Returns shift of SQ - * @rq_shift: Returns shift of RQ - */ -void -irdma_uk_calc_shift_wq(struct irdma_qp_uk_init_info *ukinfo, u8 *sq_shift, - u8 *rq_shift) -{ - bool imm_support = ukinfo->uk_attrs->hw_rev >= IRDMA_GEN_2 ? true : false; - - irdma_get_wqe_shift(ukinfo->uk_attrs, - imm_support ? ukinfo->max_sq_frag_cnt + 1 : - ukinfo->max_sq_frag_cnt, - ukinfo->max_inline_data, sq_shift); - - irdma_get_wqe_shift(ukinfo->uk_attrs, ukinfo->max_rq_frag_cnt, 0, - rq_shift); - - if (ukinfo->uk_attrs->hw_rev == IRDMA_GEN_1) { - if (ukinfo->abi_ver > 4) - *rq_shift = IRDMA_MAX_RQ_WQE_SHIFT_GEN1; - } -} - /** * irdma_uk_calc_depth_shift_sq - calculate depth and shift for SQ size. * @ukinfo: qp initialization info @@ -1786,6 +1762,8 @@ irdma_uk_qp_init(struct irdma_qp_uk *qp, struct irdma_qp_uk_init_info *info) qp->wqe_ops = iw_wqe_uk_ops_gen_1; else qp->wqe_ops = iw_wqe_uk_ops; + qp->start_wqe_idx = info->start_wqe_idx; + return ret_code; } diff --git a/contrib/ofed/libirdma/irdma_umain.c b/contrib/ofed/libirdma/irdma_umain.c index 2902dd963e18..e8d27c31a0dc 100644 --- a/contrib/ofed/libirdma/irdma_umain.c +++ b/contrib/ofed/libirdma/irdma_umain.c @@ -48,7 +48,7 @@ /** * Driver version */ -char libirdma_version[] = "1.2.17-k"; +char libirdma_version[] = "1.2.36-k"; unsigned int irdma_dbg; diff --git a/contrib/ofed/libirdma/irdma_umain.h b/contrib/ofed/libirdma/irdma_umain.h index 1fa18dd7a847..269609f8c77f 100644 --- a/contrib/ofed/libirdma/irdma_umain.h +++ b/contrib/ofed/libirdma/irdma_umain.h @@ -96,7 +96,6 @@ struct irdma_cq_buf { LIST_ENTRY(irdma_cq_buf) list; struct irdma_cq_uk cq; struct verbs_mr vmr; - size_t buf_size; }; extern pthread_mutex_t sigusr1_wait_mutex; @@ -142,7 +141,6 @@ struct irdma_uqp { struct ibv_recv_wr *pend_rx_wr; struct irdma_qp_uk qp; enum ibv_qp_type qp_type; - struct irdma_sge *recv_sges; }; /* irdma_uverbs.c */ diff --git a/contrib/ofed/libirdma/irdma_user.h b/contrib/ofed/libirdma/irdma_user.h index 0f0aa30bec91..aeb6aa9feebd 100644 --- a/contrib/ofed/libirdma/irdma_user.h +++ b/contrib/ofed/libirdma/irdma_user.h @@ -1,7 +1,7 @@ /*- * SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB * - * Copyright (c) 2015 - 2022 Intel Corporation + * Copyright (c) 2015 - 2023 Intel Corporation * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -50,7 +50,7 @@ #define irdma_access_privileges u32 #define irdma_physical_fragment u64 #define irdma_address_list u64 * -#define irdma_sgl struct irdma_sge * +#define irdma_sgl struct ibv_sge * #define IRDMA_MAX_MR_SIZE 0x200000000000ULL @@ -80,96 +80,6 @@ #define IRDMA_OP_TYPE_REC_IMM 0x3f #define IRDMA_FLUSH_MAJOR_ERR 1 -#define IRDMA_SRQFLUSH_RSVD_MAJOR_ERR 0xfffe - -/* Async Events codes */ -#define IRDMA_AE_AMP_UNALLOCATED_STAG 0x0102 -#define IRDMA_AE_AMP_INVALID_STAG 0x0103 -#define IRDMA_AE_AMP_BAD_QP 0x0104 -#define IRDMA_AE_AMP_BAD_PD 0x0105 -#define IRDMA_AE_AMP_BAD_STAG_KEY 0x0106 -#define IRDMA_AE_AMP_BAD_STAG_INDEX 0x0107 -#define IRDMA_AE_AMP_BOUNDS_VIOLATION 0x0108 -#define IRDMA_AE_AMP_RIGHTS_VIOLATION 0x0109 -#define IRDMA_AE_AMP_TO_WRAP 0x010a -#define IRDMA_AE_AMP_FASTREG_VALID_STAG 0x010c -#define IRDMA_AE_AMP_FASTREG_MW_STAG 0x010d -#define IRDMA_AE_AMP_FASTREG_INVALID_RIGHTS 0x010e -#define IRDMA_AE_AMP_FASTREG_INVALID_LENGTH 0x0110 -#define IRDMA_AE_AMP_INVALIDATE_SHARED 0x0111 -#define IRDMA_AE_AMP_INVALIDATE_NO_REMOTE_ACCESS_RIGHTS 0x0112 -#define IRDMA_AE_AMP_INVALIDATE_MR_WITH_BOUND_WINDOWS 0x0113 -#define IRDMA_AE_AMP_MWBIND_VALID_STAG 0x0114 -#define IRDMA_AE_AMP_MWBIND_OF_MR_STAG 0x0115 -#define IRDMA_AE_AMP_MWBIND_TO_ZERO_BASED_STAG 0x0116 -#define IRDMA_AE_AMP_MWBIND_TO_MW_STAG 0x0117 -#define IRDMA_AE_AMP_MWBIND_INVALID_RIGHTS 0x0118 -#define IRDMA_AE_AMP_MWBIND_INVALID_BOUNDS 0x0119 -#define IRDMA_AE_AMP_MWBIND_TO_INVALID_PARENT 0x011a -#define IRDMA_AE_AMP_MWBIND_BIND_DISABLED 0x011b -#define IRDMA_AE_PRIV_OPERATION_DENIED 0x011c -#define IRDMA_AE_AMP_INVALIDATE_TYPE1_MW 0x011d -#define IRDMA_AE_AMP_MWBIND_ZERO_BASED_TYPE1_MW 0x011e -#define IRDMA_AE_AMP_FASTREG_INVALID_PBL_HPS_CFG 0x011f -#define IRDMA_AE_AMP_MWBIND_WRONG_TYPE 0x0120 -#define IRDMA_AE_AMP_FASTREG_PBLE_MISMATCH 0x0121 -#define IRDMA_AE_UDA_XMIT_DGRAM_TOO_LONG 0x0132 -#define IRDMA_AE_UDA_XMIT_BAD_PD 0x0133 -#define IRDMA_AE_UDA_XMIT_DGRAM_TOO_SHORT 0x0134 -#define IRDMA_AE_UDA_L4LEN_INVALID 0x0135 -#define IRDMA_AE_BAD_CLOSE 0x0201 -#define IRDMA_AE_RDMAP_ROE_BAD_LLP_CLOSE 0x0202 -#define IRDMA_AE_CQ_OPERATION_ERROR 0x0203 -#define IRDMA_AE_RDMA_READ_WHILE_ORD_ZERO 0x0205 -#define IRDMA_AE_STAG_ZERO_INVALID 0x0206 -#define IRDMA_AE_IB_RREQ_AND_Q1_FULL 0x0207 -#define IRDMA_AE_IB_INVALID_REQUEST 0x0208 -#define IRDMA_AE_WQE_UNEXPECTED_OPCODE 0x020a -#define IRDMA_AE_WQE_INVALID_PARAMETER 0x020b -#define IRDMA_AE_WQE_INVALID_FRAG_DATA 0x020c -#define IRDMA_AE_IB_REMOTE_ACCESS_ERROR 0x020d -#define IRDMA_AE_IB_REMOTE_OP_ERROR 0x020e -#define IRDMA_AE_WQE_LSMM_TOO_LONG 0x0220 -#define IRDMA_AE_DDP_INVALID_MSN_GAP_IN_MSN 0x0301 -#define IRDMA_AE_DDP_UBE_DDP_MESSAGE_TOO_LONG_FOR_AVAILABLE_BUFFER 0x0303 -#define IRDMA_AE_DDP_UBE_INVALID_DDP_VERSION 0x0304 -#define IRDMA_AE_DDP_UBE_INVALID_MO 0x0305 -#define IRDMA_AE_DDP_UBE_INVALID_MSN_NO_BUFFER_AVAILABLE 0x0306 -#define IRDMA_AE_DDP_UBE_INVALID_QN 0x0307 -#define IRDMA_AE_DDP_NO_L_BIT 0x0308 -#define IRDMA_AE_RDMAP_ROE_INVALID_RDMAP_VERSION 0x0311 -#define IRDMA_AE_RDMAP_ROE_UNEXPECTED_OPCODE 0x0312 -#define IRDMA_AE_ROE_INVALID_RDMA_READ_REQUEST 0x0313 -#define IRDMA_AE_ROE_INVALID_RDMA_WRITE_OR_READ_RESP 0x0314 -#define IRDMA_AE_ROCE_RSP_LENGTH_ERROR 0x0316 -#define IRDMA_AE_ROCE_EMPTY_MCG 0x0380 -#define IRDMA_AE_ROCE_BAD_MC_IP_ADDR 0x0381 -#define IRDMA_AE_ROCE_BAD_MC_QPID 0x0382 -#define IRDMA_AE_MCG_QP_PROTOCOL_MISMATCH 0x0383 -#define IRDMA_AE_INVALID_ARP_ENTRY 0x0401 -#define IRDMA_AE_INVALID_TCP_OPTION_RCVD 0x0402 -#define IRDMA_AE_STALE_ARP_ENTRY 0x0403 -#define IRDMA_AE_INVALID_AH_ENTRY 0x0406 -#define IRDMA_AE_LLP_CLOSE_COMPLETE 0x0501 -#define IRDMA_AE_LLP_CONNECTION_RESET 0x0502 -#define IRDMA_AE_LLP_FIN_RECEIVED 0x0503 -#define IRDMA_AE_LLP_RECEIVED_MARKER_AND_LENGTH_FIELDS_DONT_MATCH 0x0504 -#define IRDMA_AE_LLP_RECEIVED_MPA_CRC_ERROR 0x0505 -#define IRDMA_AE_LLP_SEGMENT_TOO_SMALL 0x0507 -#define IRDMA_AE_LLP_SYN_RECEIVED 0x0508 -#define IRDMA_AE_LLP_TERMINATE_RECEIVED 0x0509 -#define IRDMA_AE_LLP_TOO_MANY_RETRIES 0x050a -#define IRDMA_AE_LLP_TOO_MANY_KEEPALIVE_RETRIES 0x050b -#define IRDMA_AE_LLP_DOUBT_REACHABILITY 0x050c -#define IRDMA_AE_LLP_CONNECTION_ESTABLISHED 0x050e -#define IRDMA_AE_RESOURCE_EXHAUSTION 0x0520 -#define IRDMA_AE_RESET_SENT 0x0601 -#define IRDMA_AE_TERMINATE_SENT 0x0602 -#define IRDMA_AE_RESET_NOT_SENT 0x0603 -#define IRDMA_AE_LCE_QP_CATASTROPHIC 0x0700 -#define IRDMA_AE_LCE_FUNCTION_CATASTROPHIC 0x0701 -#define IRDMA_AE_LCE_CQ_CATASTROPHIC 0x0702 -#define IRDMA_AE_QP_SUSPEND_COMPLETE 0x0900 enum irdma_device_caps_const { IRDMA_WQE_SIZE = 4, @@ -201,8 +111,7 @@ enum irdma_device_caps_const { IRDMA_MAX_OUTBOUND_MSG_SIZE = 65537, /* 64K +1 */ IRDMA_MAX_INBOUND_MSG_SIZE = 65537, - IRDMA_MAX_PUSH_PAGE_COUNT = 1024, - IRDMA_MAX_PE_ENA_VF_COUNT = 32, + IRDMA_MAX_PE_ENA_VF_COUNT = 32, IRDMA_MAX_VF_FPM_ID = 47, IRDMA_MAX_SQ_PAYLOAD_SIZE = 2145386496, IRDMA_MAX_INLINE_DATA_SIZE = 101, @@ -229,12 +138,7 @@ enum irdma_flush_opcode { FLUSH_RETRY_EXC_ERR, FLUSH_MW_BIND_ERR, FLUSH_REM_INV_REQ_ERR, -}; - -enum irdma_qp_event_type { - IRDMA_QP_EVENT_CATASTROPHIC, - IRDMA_QP_EVENT_ACCESS_ERR, - IRDMA_QP_EVENT_REQ_ERR, + FLUSH_RNR_RETRY_EXC_ERR, }; enum irdma_cmpl_status { @@ -282,12 +186,6 @@ struct irdma_cq_uk; struct irdma_qp_uk_init_info; struct irdma_cq_uk_init_info; -struct irdma_sge { - irdma_tagged_offset tag_off; - u32 len; - irdma_stag stag; -}; - struct irdma_ring { volatile u32 head; volatile u32 tail; /* effective tail */ @@ -319,13 +217,13 @@ struct irdma_post_rq_info { struct irdma_rdma_write { irdma_sgl lo_sg_list; u32 num_lo_sges; - struct irdma_sge rem_addr; + struct ibv_sge rem_addr; }; struct irdma_rdma_read { irdma_sgl lo_sg_list; u32 num_lo_sges; - struct irdma_sge rem_addr; + struct ibv_sge rem_addr; }; struct irdma_bind_window { @@ -399,11 +297,6 @@ struct irdma_cq_poll_info { } stat; }; -struct qp_err_code { - enum irdma_flush_opcode flush_code; - enum irdma_qp_event_type event_type; -}; - int irdma_uk_inline_rdma_write(struct irdma_qp_uk *qp, struct irdma_post_sq_info *info, bool post_sq); int irdma_uk_inline_send(struct irdma_qp_uk *qp, @@ -426,9 +319,9 @@ int irdma_uk_stag_local_invalidate(struct irdma_qp_uk *qp, bool post_sq); struct irdma_wqe_uk_ops { - void (*iw_copy_inline_data)(u8 *dest, struct irdma_sge *sge_list, u32 num_sges, u8 polarity); + void (*iw_copy_inline_data)(u8 *dest, struct ibv_sge *sge_list, u32 num_sges, u8 polarity); u16 (*iw_inline_data_size_to_quanta)(u32 data_size); - void (*iw_set_fragment)(__le64 *wqe, u32 offset, struct irdma_sge *sge, + void (*iw_set_fragment)(__le64 *wqe, u32 offset, struct ibv_sge *sge, u8 valid); void (*iw_set_mw_bind_wqe)(__le64 *wqe, struct irdma_bind_window *op_info); @@ -444,8 +337,6 @@ int irdma_uk_cq_init(struct irdma_cq_uk *cq, struct irdma_cq_uk_init_info *info); int irdma_uk_qp_init(struct irdma_qp_uk *qp, struct irdma_qp_uk_init_info *info); -void irdma_uk_calc_shift_wq(struct irdma_qp_uk_init_info *ukinfo, u8 *sq_shift, - u8 *rq_shift); int irdma_uk_calc_depth_shift_sq(struct irdma_qp_uk_init_info *ukinfo, u32 *sq_depth, u8 *sq_shift); int irdma_uk_calc_depth_shift_rq(struct irdma_qp_uk_init_info *ukinfo, @@ -494,6 +385,7 @@ struct irdma_qp_uk { u8 rwqe_polarity; u8 rq_wqe_size; u8 rq_wqe_size_multiplier; + u8 start_wqe_idx; bool deferred_flag:1; bool push_mode:1; /* whether the last post wqe was pushed */ bool push_dropped:1; @@ -541,6 +433,7 @@ struct irdma_qp_uk_init_info { u32 sq_depth; u32 rq_depth; u8 first_sq_wq; + u8 start_wqe_idx; u8 type; u8 sq_shift; u8 rq_shift; @@ -574,75 +467,4 @@ int irdma_get_rqdepth(struct irdma_uk_attrs *uk_attrs, u32 rq_size, u8 shift, u3 void irdma_qp_push_wqe(struct irdma_qp_uk *qp, __le64 *wqe, u16 quanta, u32 wqe_idx, bool post_sq); void irdma_clr_wqes(struct irdma_qp_uk *qp, u32 qp_wqe_idx); - -static inline struct qp_err_code irdma_ae_to_qp_err_code(u16 ae_id) -{ - struct qp_err_code qp_err = { 0 }; - - switch (ae_id) { - case IRDMA_AE_AMP_BOUNDS_VIOLATION: - case IRDMA_AE_AMP_INVALID_STAG: - case IRDMA_AE_AMP_RIGHTS_VIOLATION: - case IRDMA_AE_AMP_UNALLOCATED_STAG: - case IRDMA_AE_AMP_BAD_PD: - case IRDMA_AE_AMP_BAD_QP: - case IRDMA_AE_AMP_BAD_STAG_KEY: - case IRDMA_AE_AMP_BAD_STAG_INDEX: - case IRDMA_AE_AMP_TO_WRAP: - case IRDMA_AE_PRIV_OPERATION_DENIED: - qp_err.flush_code = FLUSH_PROT_ERR; - qp_err.event_type = IRDMA_QP_EVENT_ACCESS_ERR; - break; - case IRDMA_AE_UDA_XMIT_BAD_PD: - case IRDMA_AE_WQE_UNEXPECTED_OPCODE: - qp_err.flush_code = FLUSH_LOC_QP_OP_ERR; - qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC; - break; - case IRDMA_AE_UDA_XMIT_DGRAM_TOO_SHORT: - case IRDMA_AE_UDA_XMIT_DGRAM_TOO_LONG: - case IRDMA_AE_UDA_L4LEN_INVALID: - case IRDMA_AE_DDP_UBE_INVALID_MO: - case IRDMA_AE_DDP_UBE_DDP_MESSAGE_TOO_LONG_FOR_AVAILABLE_BUFFER: - qp_err.flush_code = FLUSH_LOC_LEN_ERR; - qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC; - break; - case IRDMA_AE_AMP_INVALIDATE_NO_REMOTE_ACCESS_RIGHTS: - case IRDMA_AE_IB_REMOTE_ACCESS_ERROR: - qp_err.flush_code = FLUSH_REM_ACCESS_ERR; - qp_err.event_type = IRDMA_QP_EVENT_ACCESS_ERR; - break; - case IRDMA_AE_AMP_MWBIND_INVALID_RIGHTS: - case IRDMA_AE_AMP_MWBIND_BIND_DISABLED: - case IRDMA_AE_AMP_MWBIND_INVALID_BOUNDS: - case IRDMA_AE_AMP_MWBIND_VALID_STAG: - qp_err.flush_code = FLUSH_MW_BIND_ERR; - qp_err.event_type = IRDMA_QP_EVENT_ACCESS_ERR; - break; - case IRDMA_AE_LLP_TOO_MANY_RETRIES: - qp_err.flush_code = FLUSH_RETRY_EXC_ERR; - qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC; - break; - case IRDMA_AE_IB_INVALID_REQUEST: - qp_err.flush_code = FLUSH_REM_INV_REQ_ERR; - qp_err.event_type = IRDMA_QP_EVENT_REQ_ERR; - break; - case IRDMA_AE_LLP_SEGMENT_TOO_SMALL: - case IRDMA_AE_LLP_RECEIVED_MPA_CRC_ERROR: - case IRDMA_AE_ROCE_RSP_LENGTH_ERROR: - case IRDMA_AE_IB_REMOTE_OP_ERROR: - qp_err.flush_code = FLUSH_REM_OP_ERR; - qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC; - break; - case IRDMA_AE_LCE_QP_CATASTROPHIC: - qp_err.flush_code = FLUSH_FATAL_ERR; - qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC; - break; - default: - qp_err.flush_code = FLUSH_GENERAL_ERR; - qp_err.event_type = IRDMA_QP_EVENT_CATASTROPHIC; - break; - } - - return qp_err; -} #endif /* IRDMA_USER_H */ diff --git a/contrib/ofed/libirdma/irdma_uverbs.c b/contrib/ofed/libirdma/irdma_uverbs.c index 962af399ed31..e52ce1cfa229 100644 --- a/contrib/ofed/libirdma/irdma_uverbs.c +++ b/contrib/ofed/libirdma/irdma_uverbs.c @@ -152,6 +152,7 @@ irdma_ualloc_pd(struct ibv_context *context) err_free: free(iwupd); + errno = err; return NULL; } @@ -163,7 +164,6 @@ err_free: int irdma_ufree_pd(struct ibv_pd *pd) { - struct irdma_uvcontext *iwvctx = container_of(pd->context, struct irdma_uvcontext, ibv_ctx); struct irdma_upd *iwupd; int ret; @@ -374,12 +374,12 @@ irdma_free_hw_buf(void *buf, size_t size) * @cqe_64byte_ena: enable 64byte cqe */ static inline int -get_cq_size(int ncqe, u8 hw_rev, bool cqe_64byte_ena) +get_cq_size(int ncqe, u8 hw_rev) { ncqe++; /* Completions with immediate require 1 extra entry */ - if (!cqe_64byte_ena && hw_rev > IRDMA_GEN_1) + if (hw_rev > IRDMA_GEN_1) ncqe *= 2; if (ncqe < IRDMA_U_MINCQ_SIZE) @@ -388,11 +388,8 @@ get_cq_size(int ncqe, u8 hw_rev, bool cqe_64byte_ena) return ncqe; } -static inline size_t get_cq_total_bytes(u32 cq_size, bool cqe_64byte_ena){ - if (cqe_64byte_ena) - return roundup(cq_size * sizeof(struct irdma_extended_cqe), IRDMA_HW_PAGE_SIZE); - else - return roundup(cq_size * sizeof(struct irdma_cqe), IRDMA_HW_PAGE_SIZE); +static inline size_t get_cq_total_bytes(u32 cq_size) { + return roundup(cq_size * sizeof(struct irdma_cqe), IRDMA_HW_PAGE_SIZE); } /** @@ -420,7 +417,6 @@ ucreate_cq(struct ibv_context *context, u32 cq_pages; int ret, ncqe; u8 hw_rev; - bool cqe_64byte_ena; iwvctx = container_of(context, struct irdma_uvcontext, ibv_ctx); uk_attrs = &iwvctx->uk_attrs; @@ -454,11 +450,10 @@ ucreate_cq(struct ibv_context *context, return NULL; } - cqe_64byte_ena = uk_attrs->feature_flags & IRDMA_FEATURE_64_BYTE_CQE ? true : false; - info.cq_size = get_cq_size(attr_ex->cqe, hw_rev, cqe_64byte_ena); + info.cq_size = get_cq_size(attr_ex->cqe, hw_rev); + total_size = get_cq_total_bytes(info.cq_size); iwucq->comp_vector = attr_ex->comp_vector; LIST_INIT(&iwucq->resize_list); - total_size = get_cq_total_bytes(info.cq_size, cqe_64byte_ena); cq_pages = total_size >> IRDMA_HW_PAGE_SHIFT; if (!(uk_attrs->feature_flags & IRDMA_FEATURE_CQ_RESIZE)) @@ -528,8 +523,6 @@ ucreate_cq(struct ibv_context *context, info.cq_id = resp.cq_id; /* Do not report the CQE's reserved for immediate and burned by HW */ iwucq->verbs_cq.cq.cqe = ncqe; - if (cqe_64byte_ena) - info.avoid_mem_cflct = true; info.cqe_alloc_db = (u32 *)((u8 *)iwvctx->db + IRDMA_DB_CQ_OFFSET); irdma_uk_cq_init(&iwucq->cq, &info); return &iwucq->verbs_cq.cq_ex; @@ -585,7 +578,7 @@ static void irdma_free_cq_buf(struct irdma_cq_buf *cq_buf) { ibv_cmd_dereg_mr(&cq_buf->vmr.ibv_mr); - irdma_free_hw_buf(cq_buf->cq.cq_base, cq_buf->buf_size); + irdma_free_hw_buf(cq_buf->cq.cq_base, get_cq_total_bytes(cq_buf->cq.cq_size)); free(cq_buf); } @@ -1322,6 +1315,8 @@ irdma_vmapped_qp(struct irdma_uqp *iwuqp, struct ibv_pd *pd, cmd.user_wqe_bufs = (__u64) ((uintptr_t)info->sq); cmd.user_compl_ctx = (__u64) (uintptr_t)&iwuqp->qp; + cmd.comp_mask |= IRDMA_CREATE_QP_USE_START_WQE_IDX; + ret = ibv_cmd_create_qp(pd, &iwuqp->ibv_qp, attr, &cmd.ibv_cmd, sizeof(cmd), &resp.ibv_resp, sizeof(struct irdma_ucreate_qp_resp)); @@ -1331,6 +1326,8 @@ irdma_vmapped_qp(struct irdma_uqp *iwuqp, struct ibv_pd *pd, info->sq_size = resp.actual_sq_size; info->rq_size = resp.actual_rq_size; info->first_sq_wq = legacy_mode ? 1 : resp.lsmm; + if (resp.comp_mask & IRDMA_CREATE_QP_USE_START_WQE_IDX) + info->start_wqe_idx = resp.start_wqe_idx; info->qp_caps = resp.qp_caps; info->qp_id = resp.qp_id; iwuqp->irdma_drv_opt = resp.irdma_drv_opt; @@ -1379,6 +1376,8 @@ irdma_ucreate_qp(struct ibv_pd *pd, if (attr->cap.max_send_sge > uk_attrs->max_hw_wq_frags || attr->cap.max_recv_sge > uk_attrs->max_hw_wq_frags || + attr->cap.max_send_wr > uk_attrs->max_hw_wq_quanta || + attr->cap.max_recv_wr > uk_attrs->max_hw_rq_quanta || attr->cap.max_inline_data > uk_attrs->max_hw_inline) { errno = EINVAL; return NULL; @@ -1430,18 +1429,12 @@ irdma_ucreate_qp(struct ibv_pd *pd, attr->cap.max_recv_wr = info.rq_size; } - iwuqp->recv_sges = calloc(attr->cap.max_recv_sge, sizeof(*iwuqp->recv_sges)); - if (!iwuqp->recv_sges) { - status = errno; /* preserve errno */ - goto err_destroy_lock; - } - info.wqe_alloc_db = (u32 *)iwvctx->db; info.legacy_mode = iwvctx->legacy_mode; info.sq_wrtrk_array = calloc(info.sq_depth, sizeof(*info.sq_wrtrk_array)); if (!info.sq_wrtrk_array) { status = errno; /* preserve errno */ - goto err_free_rsges; + goto err_destroy_lock; } info.rq_wrid_array = calloc(info.rq_depth, sizeof(*info.rq_wrid_array)); @@ -1475,8 +1468,6 @@ err_free_rq_wrid: free(info.rq_wrid_array); err_free_sq_wrtrk: free(info.sq_wrtrk_array); -err_free_rsges: - free(iwuqp->recv_sges); err_destroy_lock: pthread_spin_destroy(&iwuqp->lock); err_free_qp: @@ -1635,7 +1626,6 @@ irdma_udestroy_qp(struct ibv_qp *qp) free(iwuqp->qp.rq_wrid_array); irdma_free_hw_buf(iwuqp->qp.sq_base, iwuqp->buf_size); - free(iwuqp->recv_sges); free(iwuqp); return 0; @@ -1645,26 +1635,6 @@ err: return ret; } -/** - * irdma_copy_sg_list - copy sg list for qp - * @sg_list: copied into sg_list - * @sgl: copy from sgl - * @num_sges: count of sg entries - * @max_sges: count of max supported sg entries - */ -static void -irdma_copy_sg_list(struct irdma_sge *sg_list, struct ibv_sge *sgl, - int num_sges) -{ - int i; - *** 2464 LINES SKIPPED ***