From nobody Thu Oct 09 19:28:57 2025 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cjKgF4sVBz6C0qw; Thu, 09 Oct 2025 19:28:57 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cjKgF4BYvz3Qn1; Thu, 09 Oct 2025 19:28:57 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1760038137; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=5ViPW98wzyPAeOpBrE55XW4XSBgGceozjMWV5SX82v0=; b=C9D6GScDBjgOQ8bVHzBDK6bdj1CVjua3ErTquz3weegUa1cyY0B+iJHDcmYJ17ycEVS+TW HOGS8H3vrE1yv2pASCLo7OlXdZO+EmvSNjYnhssXISLaNG2JnAUH6loBTiXG9csZC2PAS1 qfe06VziHa8LelZox9wmBfZTBNHSkNsA7RR2bHcN68tNdc4LTOGLB5TM7TPSeSQ8A5qdTf /5ElNLMcazX8MFLE+5odC8oxrPOoG4fNzkOpBVVLlbl9YxnkmQd9rufpsblcyo0ylFtPbq sYMAxWBlYarKG+8ZypdUTZrCtG/b29zQiPi5zdSM5rl0MkQVM0JfNpUGMwpJ5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1760038137; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=5ViPW98wzyPAeOpBrE55XW4XSBgGceozjMWV5SX82v0=; b=hF6zO57YgtqoeLYVuKEBQpciydZQY9vnhhBvNmTAmAflsu4rI8k9C7yKnsvQkbR6T7kTSn 6fXbt+22O7YqePWdN+oze5/cmABYNzR1e1cBjmM2aBcq2CPDCZqsmyvudhwM54tLcjstxH PTZ/usuK9/OuKpTILctYbU2KVlnUdtupgwyOqLZ7tdpOK4VeyOa/tZS9sjThho9qvlJBas 8CfYrAoLy6Bp04AREikF7AkRqNy0EUiQQr8xBBxwpBYsDxeoZYLRKbJmCeT9JpGPCQCj33 4hIHNFCQiDtD0RLGNcbhKg80fIqajtdhqEen6opXxZZ/JlDURcB+DRk/qUFFAg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1760038137; a=rsa-sha256; cv=none; b=HfCevNJa96OQiXqEDxjwwiEIA9zQNZvOA1aAwKtyXAAa+cwvc66v+WWthmisSoQrDMwpRY KP2MW29Gp64MGYCVJB+5GctOzjxowy3SKYC1jx17o1mxd4VCbsgsWi3Ms4JLmhh+/Afigo DFfL3uJihacjtQHDJ1dZZE5gztpfGXwYNEUrfgB/ow/8YVOLQUV/tE76GEGVyuHRvi36Q7 tgdbFtAWdeNjX/YIXUl4fhPF5r37CDdFXqV8rDPIM0BsBIf3+mwJSyG9StYEBt/OPYCxIb 2/xarZaev6Am2mUo656kON4+s48lR9ZT63Umm0nCA4IKuEGemQqcE+3ynuNQ/w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4cjKgF3kQHzVbQ; Thu, 09 Oct 2025 19:28:57 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 599JSvZn047161; Thu, 9 Oct 2025 19:28:57 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 599JSv6Q047158; Thu, 9 Oct 2025 19:28:57 GMT (envelope-from git) Date: Thu, 9 Oct 2025 19:28:57 GMT Message-Id: <202510091928.599JSv6Q047158@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: John Baldwin Subject: git: 1435a9b293e2 - main - ctld: Defer initialization of NVMeoF associations List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 1435a9b293e21f8fca1f654420c5075ea7434e8f Auto-Submitted: auto-generated The branch main has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=1435a9b293e21f8fca1f654420c5075ea7434e8f commit 1435a9b293e21f8fca1f654420c5075ea7434e8f Author: John Baldwin AuthorDate: 2025-10-09 19:24:27 +0000 Commit: John Baldwin CommitDate: 2025-10-09 19:24:27 +0000 ctld: Defer initialization of NVMeoF associations Wait until all of the configuration has been parsed before creating associations for NVMe portals. This ensures that any options specified in a transport group are honored when creating associations. To enable this, add a new virtual method portal::prepare invoked when applying a configuration prior to opening a socket (or reusing an existing socket) for a portal. Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D52844 --- usr.sbin/ctld/ctld.cc | 5 ++++ usr.sbin/ctld/ctld.hh | 1 + usr.sbin/ctld/nvmf.cc | 77 +++++++++++++++++++++++---------------------------- usr.sbin/ctld/nvmf.hh | 16 ++++------- 4 files changed, 46 insertions(+), 53 deletions(-) diff --git a/usr.sbin/ctld/ctld.cc b/usr.sbin/ctld/ctld.cc index 10c12f25068e..331c029e282e 100644 --- a/usr.sbin/ctld/ctld.cc +++ b/usr.sbin/ctld/ctld.cc @@ -814,6 +814,11 @@ portal_group::open_sockets(struct conf &oldconf) } for (portal_up &portal : pg_portals) { + if (!portal->prepare()) { + cumulated_error++; + continue; + } + /* * Try to find already open portal and reuse the * listening socket. We don't care about what portal diff --git a/usr.sbin/ctld/ctld.hh b/usr.sbin/ctld/ctld.hh index cc88e6eb590e..3bf18f6a32c0 100644 --- a/usr.sbin/ctld/ctld.hh +++ b/usr.sbin/ctld/ctld.hh @@ -151,6 +151,7 @@ struct portal { p_protocol(protocol) {} virtual ~portal() = default; + virtual bool prepare() { return true; } bool reuse_socket(portal &oldp); bool init_socket(); virtual bool init_socket_options(int s __unused) { return true; } diff --git a/usr.sbin/ctld/nvmf.cc b/usr.sbin/ctld/nvmf.cc index d1240bfa4f6c..eb116903f5c1 100644 --- a/usr.sbin/ctld/nvmf.cc +++ b/usr.sbin/ctld/nvmf.cc @@ -34,11 +34,8 @@ struct nvmf_io_portal final : public nvmf_portal { nvmf_io_portal(struct portal_group *pg, const char *listen, - portal_protocol protocol, freebsd::addrinfo_up ai, - const struct nvmf_association_params &aparams, - nvmf_association_up na) : - nvmf_portal(pg, listen, protocol, std::move(ai), aparams, - std::move(na)) {} + portal_protocol protocol, freebsd::addrinfo_up ai) : + nvmf_portal(pg, listen, protocol, std::move(ai)) {} void handle_connection(freebsd::fd_up fd, const char *host, const struct sockaddr *client_sa) override; @@ -63,8 +60,6 @@ struct nvmf_transport_group final : public portal_group { override; private: - struct nvmf_association_params init_aparams(portal_protocol protocol); - static uint16_t last_port_id; }; @@ -143,48 +138,55 @@ parse_number(const nvlist_t *nvl, const char *key, uint64_t def, uint64_t minv, return def; } -struct nvmf_association_params -nvmf_transport_group::init_aparams(portal_protocol protocol) +bool +nvmf_portal::prepare() { - struct nvmf_association_params params; - memset(¶ms, 0, sizeof(params)); + memset(&p_aparams, 0, sizeof(p_aparams)); /* Options shared between discovery and I/O associations. */ - const nvlist_t *nvl = pg_options.get(); - params.tcp.header_digests = parse_bool(nvl, "HDGST", false); - params.tcp.data_digests = parse_bool(nvl, "DDGST", false); - uint64_t value = parse_number(nvl, "MAXH2CDATA", DEFAULT_MAXH2CDATA, - 4096, UINT32_MAX); + freebsd::nvlist_up nvl = portal_group()->options(); + p_aparams.tcp.header_digests = parse_bool(nvl.get(), "HDGST", false); + p_aparams.tcp.data_digests = parse_bool(nvl.get(), "DDGST", false); + uint64_t value = parse_number(nvl.get(), "MAXH2CDATA", + DEFAULT_MAXH2CDATA, 4096, UINT32_MAX); if (value % 4 != 0) { log_warnx("Invalid value \"%ju\" for option MAXH2CDATA", (uintmax_t)value); value = DEFAULT_MAXH2CDATA; } - params.tcp.maxh2cdata = value; + p_aparams.tcp.maxh2cdata = value; - switch (protocol) { + switch (protocol()) { case portal_protocol::NVME_TCP: - params.sq_flow_control = parse_bool(nvl, "SQFC", false); - params.dynamic_controller_model = true; - params.max_admin_qsize = parse_number(nvl, "max_admin_qsize", - NVME_MAX_ADMIN_ENTRIES, NVME_MIN_ADMIN_ENTRIES, - NVME_MAX_ADMIN_ENTRIES); - params.max_io_qsize = parse_number(nvl, "max_io_qsize", + p_aparams.sq_flow_control = parse_bool(nvl.get(), "SQFC", + false); + p_aparams.dynamic_controller_model = true; + p_aparams.max_admin_qsize = parse_number(nvl.get(), + "max_admin_qsize", NVME_MAX_ADMIN_ENTRIES, + NVME_MIN_ADMIN_ENTRIES, NVME_MAX_ADMIN_ENTRIES); + p_aparams.max_io_qsize = parse_number(nvl.get(), "max_io_qsize", NVME_MAX_IO_ENTRIES, NVME_MIN_IO_ENTRIES, NVME_MAX_IO_ENTRIES); - params.tcp.pda = 0; + p_aparams.tcp.pda = 0; break; case portal_protocol::NVME_DISCOVERY_TCP: - params.sq_flow_control = false; - params.dynamic_controller_model = true; - params.max_admin_qsize = NVME_MAX_ADMIN_ENTRIES; - params.tcp.pda = 0; + p_aparams.sq_flow_control = false; + p_aparams.dynamic_controller_model = true; + p_aparams.max_admin_qsize = NVME_MAX_ADMIN_ENTRIES; + p_aparams.tcp.pda = 0; break; default: __assert_unreachable(); } - return params; + p_association.reset(nvmf_allocate_association(NVMF_TRTYPE_TCP, true, + &p_aparams)); + if (!p_association) { + log_warn("Failed to create NVMe controller association"); + return false; + } + + return true; } portal_group_up @@ -209,15 +211,12 @@ bool nvmf_transport_group::add_portal(const char *value, portal_protocol protocol) { freebsd::addrinfo_up ai; - enum nvmf_trtype trtype; switch (protocol) { case portal_protocol::NVME_TCP: - trtype = NVMF_TRTYPE_TCP; ai = parse_addr_port(value, "4420"); break; case portal_protocol::NVME_DISCOVERY_TCP: - trtype = NVMF_TRTYPE_TCP; ai = parse_addr_port(value, "8009"); break; default: @@ -230,14 +229,6 @@ nvmf_transport_group::add_portal(const char *value, portal_protocol protocol) return false; } - struct nvmf_association_params aparams = init_aparams(protocol); - nvmf_association_up association(nvmf_allocate_association(trtype, true, - &aparams)); - if (!association) { - log_warn("Failed to create NVMe controller association"); - return false; - } - /* * XXX: getaddrinfo(3) may return multiple addresses; we should turn * those into multiple portals. @@ -246,10 +237,10 @@ nvmf_transport_group::add_portal(const char *value, portal_protocol protocol) portal_up portal; if (protocol == portal_protocol::NVME_DISCOVERY_TCP) { portal = std::make_unique(this, value, - protocol, std::move(ai), aparams, std::move(association)); + protocol, std::move(ai)); } else { portal = std::make_unique(this, value, - protocol, std::move(ai), aparams, std::move(association)); + protocol, std::move(ai)); need_tcp_transport = true; } diff --git a/usr.sbin/ctld/nvmf.hh b/usr.sbin/ctld/nvmf.hh index 0b4f8d45adfd..6f34a2858ef9 100644 --- a/usr.sbin/ctld/nvmf.hh +++ b/usr.sbin/ctld/nvmf.hh @@ -38,13 +38,12 @@ using nvmf_qpair_up = std::unique_ptr; struct nvmf_portal : public portal { nvmf_portal(struct portal_group *pg, const char *listen, - portal_protocol protocol, freebsd::addrinfo_up ai, - const struct nvmf_association_params &aparams, - nvmf_association_up na) : - portal(pg, listen, protocol, std::move(ai)), - p_aparams(aparams), p_association(std::move(na)) {} + portal_protocol protocol, freebsd::addrinfo_up ai) : + portal(pg, listen, protocol, std::move(ai)) {} virtual ~nvmf_portal() override = default; + virtual bool prepare() override; + const struct nvmf_association_params *aparams() const { return &p_aparams; } @@ -58,11 +57,8 @@ private: struct nvmf_discovery_portal final : public nvmf_portal { nvmf_discovery_portal(struct portal_group *pg, const char *listen, - portal_protocol protocol, freebsd::addrinfo_up ai, - const struct nvmf_association_params &aparams, - nvmf_association_up na) : - nvmf_portal(pg, listen, protocol, std::move(ai), aparams, - std::move(na)) {} + portal_protocol protocol, freebsd::addrinfo_up ai) : + nvmf_portal(pg, listen, protocol, std::move(ai)) {} void handle_connection(freebsd::fd_up fd, const char *host, const struct sockaddr *client_sa) override;