From nobody Wed Apr 23 23:12:17 2025 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZjZcx6Jycz5t3y3; Wed, 23 Apr 2025 23:12:17 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZjZcx5MBfz3vkC; Wed, 23 Apr 2025 23:12:17 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1745449937; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=mRxPldrHVwsq87W8chyUOcdt+6WA9RyNFmg/WvGA3Jw=; b=jys7ae0iIwh2hujLtzRx6N5m5ZmZlVirASBtsPotymeBblojFZXXmsPalhSVlbyEoauLo5 yZKcboVgYmR+JhGz5q1V+rjJB3TEdegTfPlNpow4pzenapELbbFJBn5S5sp6+d6PeY1vGh u+pokhs6U2KGYtLLt87O2AMtwLNQh+qc4kYm/EAFo5lmbeRvj27vNS3WITI/cHqc8JPaKd AIZXG0hUzKF83+47gPZdkxdT7ac+XPNOA6S+DHm8lm7Q9cb3lMBb7sF278zHsDL90vfdy1 jH+5WbCkIAshuGpb+NK1E6uPn4REMWceocW2TerADk1hxO/hu7GqpfdG/mhYjw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1745449937; a=rsa-sha256; cv=none; b=DiArKF0HJcGlMY/Qf9VZnTeMlmu8JP9Iy5uH6+ekWBVdiAAwNMKfvpcm8vy0PIdBEbal7r g+mWC4LxYvt56LJDKzvGpbK0xti4WfZodCxpAl2/ZmoFUWPyBWCrtqb8gngkvnvCk4OpPm 6IcKgp2rlrlJkbZS+Zz1PEeW/5IT6n1pcy5mvXo1oTIhi4v1saral+u156UaljmVsdY+PC ZepQL+xGmpJqE0K/qvamakd8IIT20yMcUxoAWjBGbE6ZS18263M4ZCUHUz6teNbX+Jb96S JDOETW9YleEIgBpGQfpTMzDE6vQOukwqmUl3u+Qd6yW3iloxrd8p3M9MqE1PLQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1745449937; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=mRxPldrHVwsq87W8chyUOcdt+6WA9RyNFmg/WvGA3Jw=; b=hUSJypBKKXnrfaYYZ8APWsJjVrQ+gfRyVCF5peo9TcsZwL+CrIQAj6HANWz27fHLP3SBNr T8nvgDDLNNVWTPSKoinBvAxzP9/GBARE0hUgzwcSgvDy/5t6WmEtCW1MItINj+GlZniUUP JAL3evit2rKaDvs59FXEwSBR1ZJNzztFXXNHI47XokyPopKYHepdGp/sU1F/DkGCKD3xnJ rO6MZ8CN1Zpn96HM5/vm3Oog13XNJW+CfkPiw4MWr8Ob9mnPcx05WD4L7H/UjNPRPMTnPU BC0UZaCevVk7pyTwBPuhTQXcrPkf0AXJER58MyWSE8KMoLQV37k18MFawQIYXA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ZjZcx4w8Gz8kx; Wed, 23 Apr 2025 23:12:17 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 53NNCHwe051786; Wed, 23 Apr 2025 23:12:17 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 53NNCHEi051783; Wed, 23 Apr 2025 23:12:17 GMT (envelope-from git) Date: Wed, 23 Apr 2025 23:12:17 GMT Message-Id: <202504232312.53NNCHEi051783@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Chuck Silvers Subject: git: f2605f67a13e - main - mpi3mr: configure larger max I/O size if the HBA firmware supports it List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: chs X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: f2605f67a13eb1f1427ee58a097791ca25359a5e Auto-Submitted: auto-generated The branch main has been updated by chs: URL: https://cgit.FreeBSD.org/src/commit/?id=f2605f67a13eb1f1427ee58a097791ca25359a5e commit f2605f67a13eb1f1427ee58a097791ca25359a5e Author: Chuck Silvers AuthorDate: 2025-04-23 23:11:32 +0000 Commit: Chuck Silvers CommitDate: 2025-04-23 23:11:32 +0000 mpi3mr: configure larger max I/O size if the HBA firmware supports it The max I/O size that an mpi3mr HBA supports is reported in the IOCFacts structure (with 0 representing the legacy max I/O size of 1 MB). By default, set the max I/O size of devices attached to mpi3mr controllers to the smaller of the HBA's max I/O size and the kernel's maxphys. Allow this default to be overriden by a global tunable "hw.mpi3mr.max_sgl_entries" or by a per-controller tunable "dev.mpi3mr.N.max_sgl_entries". Sponsored by: Netflix Reviewed by: imp Differential Revision: https://reviews.freebsd.org/D49090 --- sys/dev/mpi3mr/mpi3mr.c | 11 +++++++++-- sys/dev/mpi3mr/mpi3mr.h | 12 +++++++++--- sys/dev/mpi3mr/mpi3mr_cam.c | 10 ++++++---- sys/dev/mpi3mr/mpi3mr_pci.c | 18 +++++++++++++++++- 4 files changed, 41 insertions(+), 10 deletions(-) diff --git a/sys/dev/mpi3mr/mpi3mr.c b/sys/dev/mpi3mr/mpi3mr.c index 36e540520e42..a83fb2ddbc4b 100644 --- a/sys/dev/mpi3mr/mpi3mr.c +++ b/sys/dev/mpi3mr/mpi3mr.c @@ -1617,6 +1617,7 @@ static int mpi3mr_process_factsdata(struct mpi3mr_softc *sc, (facts_data->MaxPCIeSwitches); sc->facts.max_sasexpanders = (facts_data->MaxSASExpanders); + sc->facts.max_data_length = facts_data->MaxDataLength; sc->facts.max_sasinitiators = (facts_data->MaxSASInitiators); sc->facts.max_enclosures = (facts_data->MaxEnclosures); @@ -1651,6 +1652,10 @@ static int mpi3mr_process_factsdata(struct mpi3mr_softc *sc, sc->facts.io_throttle_low = facts_data->IOThrottleLow; sc->facts.io_throttle_high = facts_data->IOThrottleHigh; + if (sc->facts.max_data_length == MPI3_IOCFACTS_MAX_DATA_LENGTH_NOT_REPORTED) + sc->facts.max_data_length = MPI3MR_DEFAULT_MAX_IO_SIZE; + else + sc->facts.max_data_length *= MPI3MR_PAGE_SIZE_4K; /*Store in 512b block count*/ if (sc->facts.io_throttle_data_length) sc->io_throttle_data_length = @@ -2511,7 +2516,9 @@ static int mpi3mr_alloc_chain_bufs(struct mpi3mr_softc *sc) goto out_failed; } - sz = MPI3MR_CHAINSGE_SIZE; + if (sc->max_sgl_entries > sc->facts.max_data_length / PAGE_SIZE) + sc->max_sgl_entries = sc->facts.max_data_length / PAGE_SIZE; + sz = sc->max_sgl_entries * sizeof(Mpi3SGESimple_t); if (bus_dma_tag_create(sc->mpi3mr_parent_dmat, /* parent */ 4096, 0, /* algnmnt, boundary */ @@ -4961,7 +4968,7 @@ mpi3mr_alloc_requests(struct mpi3mr_softc *sc) struct mpi3mr_cmd *cmd; int i, j, nsegs, ret; - nsegs = MPI3MR_SG_DEPTH; + nsegs = sc->max_sgl_entries; ret = bus_dma_tag_create( sc->mpi3mr_parent_dmat, /* parent */ 1, 0, /* algnmnt, boundary */ sc->dma_loaddr, /* lowaddr */ diff --git a/sys/dev/mpi3mr/mpi3mr.h b/sys/dev/mpi3mr/mpi3mr.h index 31217f823569..1ab6b8815f59 100644 --- a/sys/dev/mpi3mr/mpi3mr.h +++ b/sys/dev/mpi3mr/mpi3mr.h @@ -95,10 +95,11 @@ #define MPI3MR_NAME_LENGTH 32 #define IOCNAME "%s: " +#define MPI3MR_DEFAULT_MAX_IO_SIZE (1 * 1024 * 1024) + #define SAS4116_CHIP_REV_A0 0 #define SAS4116_CHIP_REV_B0 1 -#define MPI3MR_SG_DEPTH (MPI3MR_4K_PGSZ/sizeof(Mpi3SGESimple_t)) #define MPI3MR_MAX_SECTORS 2048 #define MPI3MR_MAX_CMDS_LUN 7 #define MPI3MR_MAX_CDB_LENGTH 16 @@ -109,7 +110,12 @@ #define MPI3MR_RAID_QDEPTH 128 #define MPI3MR_NVME_QDEPTH 128 +/* Definitions for internal SGL and Chain SGL buffers */ #define MPI3MR_4K_PGSZ 4096 +#define MPI3MR_PAGE_SIZE_4K 4096 +#define MPI3MR_DEFAULT_SGL_ENTRIES 256 +#define MPI3MR_MAX_SGL_ENTRIES 2048 + #define MPI3MR_AREQQ_SIZE (2 * MPI3MR_4K_PGSZ) #define MPI3MR_AREPQ_SIZE (4 * MPI3MR_4K_PGSZ) #define MPI3MR_AREQ_FRAME_SZ 128 @@ -125,8 +131,6 @@ #define MPI3MR_THRESHOLD_REPLY_COUNT 100 -#define MPI3MR_CHAINSGE_SIZE MPI3MR_4K_PGSZ - #define MPI3MR_SGEFLAGS_SYSTEM_SIMPLE_END_OF_LIST \ (MPI3_SGE_FLAGS_ELEMENT_TYPE_SIMPLE | MPI3_SGE_FLAGS_DLAS_SYSTEM | \ MPI3_SGE_FLAGS_END_OF_LIST) @@ -335,6 +339,7 @@ struct mpi3mr_ioc_facts U16 max_perids; U16 max_pds; U16 max_sasexpanders; + U32 max_data_length; U16 max_sasinitiators; U16 max_enclosures; U16 max_pcieswitches; @@ -671,6 +676,7 @@ struct mpi3mr_softc { struct mtx target_lock; U16 max_host_ios; + U32 max_sgl_entries; bus_dma_tag_t chain_sgl_list_tag; struct mpi3mr_chain *chain_sgl_list; U16 chain_bitmap_sz; diff --git a/sys/dev/mpi3mr/mpi3mr_cam.c b/sys/dev/mpi3mr/mpi3mr_cam.c index b4999e204ab7..af2bc0f8b55c 100644 --- a/sys/dev/mpi3mr/mpi3mr_cam.c +++ b/sys/dev/mpi3mr/mpi3mr_cam.c @@ -176,7 +176,7 @@ static void mpi3mr_prepare_sgls(void *arg, bus_dmamap_sync(sc->buffer_dmat, cm->dmamap, BUS_DMASYNC_PREWRITE); - KASSERT(nsegs <= MPI3MR_SG_DEPTH && nsegs > 0, + KASSERT(nsegs <= sc->max_sgl_entries && nsegs > 0, ("%s: bad SGE count: %d\n", device_get_nameunit(sc->mpi3mr_dev), nsegs)); KASSERT(scsiio_req->DataLength != 0, ("%s: Data segments (%d), but DataLength == 0\n", @@ -218,7 +218,7 @@ static void mpi3mr_prepare_sgls(void *arg, chain = chain_req->buf; chain_dma = chain_req->buf_phys; - memset(chain_req->buf, 0, PAGE_SIZE); + memset(chain_req->buf, 0, sc->max_sgl_entries * sizeof(Mpi3SGESimple_t)); sges_in_segment = sges_left; chain_length = sges_in_segment * sizeof(Mpi3SGESimple_t); @@ -1154,7 +1154,7 @@ mpi3mr_action_scsiio(struct mpi3mr_cam_softc *cam_sc, union ccb *ccb) return; case CAM_DATA_VADDR: case CAM_DATA_BIO: - if (csio->dxfer_len > (MPI3MR_SG_DEPTH * MPI3MR_4K_PGSZ)) { + if (csio->dxfer_len > (sc->max_sgl_entries * PAGE_SIZE)) { mpi3mr_set_ccbstatus(ccb, CAM_REQ_TOO_BIG); mpi3mr_release_command(cm); xpt_done(ccb); @@ -1305,8 +1305,10 @@ mpi3mr_cam_action(struct cam_sim *sim, union ccb *ccb) { struct mpi3mr_cam_softc *cam_sc; struct mpi3mr_target *targ; + struct mpi3mr_softc *sc; cam_sc = cam_sim_softc(sim); + sc = cam_sc->sc; mpi3mr_dprint(cam_sc->sc, MPI3MR_TRACE, "ccb func_code 0x%x target id: 0x%x\n", ccb->ccb_h.func_code, ccb->ccb_h.target_id); @@ -1357,7 +1359,7 @@ mpi3mr_cam_action(struct cam_sim *sim, union ccb *ccb) "PCI device target_id: %u max io size: %u\n", ccb->ccb_h.target_id, cpi->maxio); } else { - cpi->maxio = PAGE_SIZE * (MPI3MR_SG_DEPTH - 1); + cpi->maxio = PAGE_SIZE * (sc->max_sgl_entries - 1); } mpi3mr_set_ccbstatus(ccb, CAM_REQ_CMP); break; diff --git a/sys/dev/mpi3mr/mpi3mr_pci.c b/sys/dev/mpi3mr/mpi3mr_pci.c index 95bd39c90964..194401c5a847 100644 --- a/sys/dev/mpi3mr/mpi3mr_pci.c +++ b/sys/dev/mpi3mr/mpi3mr_pci.c @@ -178,12 +178,15 @@ mpi3mr_get_tunables(struct mpi3mr_softc *sc) sc->reset_in_progress = 0; sc->reset.type = 0; sc->iot_enable = 1; + sc->max_sgl_entries = maxphys / PAGE_SIZE; + /* * Grab the global variables. */ TUNABLE_INT_FETCH("hw.mpi3mr.debug_level", &sc->mpi3mr_debug); TUNABLE_INT_FETCH("hw.mpi3mr.ctrl_reset", &sc->reset.type); TUNABLE_INT_FETCH("hw.mpi3mr.iot_enable", &sc->iot_enable); + TUNABLE_INT_FETCH("hw.mpi3mr.max_sgl_entries", &sc->max_sgl_entries); /* Grab the unit-instance variables */ snprintf(tmpstr, sizeof(tmpstr), "dev.mpi3mr.%d.debug_level", @@ -197,6 +200,10 @@ mpi3mr_get_tunables(struct mpi3mr_softc *sc) snprintf(tmpstr, sizeof(tmpstr), "dev.mpi3mr.%d.iot_enable", device_get_unit(sc->mpi3mr_dev)); TUNABLE_INT_FETCH(tmpstr, &sc->iot_enable); + + snprintf(tmpstr, sizeof(tmpstr), "dev.mpi3mr.%d.max_sgl_entries", + device_get_unit(sc->mpi3mr_dev)); + TUNABLE_INT_FETCH(tmpstr, &sc->max_sgl_entries); } static struct mpi3mr_ident * @@ -443,7 +450,16 @@ mpi3mr_pci_attach(device_t dev) sc->mpi3mr_dev = dev; mpi3mr_get_tunables(sc); - + + if (sc->max_sgl_entries > MPI3MR_MAX_SGL_ENTRIES) + sc->max_sgl_entries = MPI3MR_MAX_SGL_ENTRIES; + else if (sc->max_sgl_entries < MPI3MR_DEFAULT_SGL_ENTRIES) + sc->max_sgl_entries = MPI3MR_DEFAULT_SGL_ENTRIES; + else { + sc->max_sgl_entries /= MPI3MR_DEFAULT_SGL_ENTRIES; + sc->max_sgl_entries *= MPI3MR_DEFAULT_SGL_ENTRIES; + } + if ((error = mpi3mr_initialize_ioc(sc, MPI3MR_INIT_TYPE_INIT)) != 0) { mpi3mr_dprint(sc, MPI3MR_ERROR, "FW initialization failed\n"); goto load_failed;