From nobody Mon Apr 28 03:25:19 2025 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Zm83452dRz5tknK; Mon, 28 Apr 2025 03:25:20 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Zm83417ybz3cmk; Mon, 28 Apr 2025 03:25:20 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1745810720; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=dILUfG7xQjsOalKA/1y5WrfgGcmEJsl0OQNsgXHtsBk=; b=FebLMhUqcpjXO7jiZ5ycNwRHYk2X2xPqKyJLr218dbuQfWpFZKKnFF4bRad0aVPZ8Mrepe yFmICkNedCNMv44hwJZpT2HAFFWHgLQi8uAvO+rcLZPHPGj/ng/VMUT8IAS8ABZJ5ZjmE2 i4b7062jl/4cQ/T/ueK8rA8VZtWsf68xqRwn8exBs2z86vXfMn6Z/7NjqGK9IJUKBWt8Dk Enh+Aqj44a+nLvB0+Dp5FHRJykGpOvFBTybsuMrlLBt0DmDwyI0239gcNFkT4NL4MMCR0w LVrmX30hsbYb7OM3iZvXzvU71R18uwaA07yww/RAMOoHm9t0SCakkQPXEkiTwg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1745810720; a=rsa-sha256; cv=none; b=cWGB8CU5QlPglCGwLpIv8Jjue2Mzb4ocojK/mg/C6sNVMLjZuEjtVvFaFo/qk/CrXjdybC fcrkjtpK1N5PeB2wQ6Kduq9I+NQRzchGdQk/D9Fbbkq/VydRBmooQerzlFw23sP4UizrZB 7l3rlCF0+8xBKoHSnsEonpKZegslJOaRNniJp/dpbFmexawUXb3tZbyFoz2ix1b/AawCjV JG1fmsXzGA2waltGPnQw5xKQEzR0JPs1uMGwHTi7xDteQokdXMiLs9XNkWGtgsxqE3arom Aot01lNZbBVdDAo5eq9Rhp1rzJ/Rw/NZ1+jK4T1Qd0i9AGuITQE/mOXzYOdF7g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1745810720; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=dILUfG7xQjsOalKA/1y5WrfgGcmEJsl0OQNsgXHtsBk=; b=Qyo08JIl11WESNfNBOgKrPFBVKOqLEFjvnKutVwDmrqbQV3p/V3PHsU46VSp9UlbLx2cv0 MZx15wimFvUcvlLiPZY966LOBz4HFW8x9KWjn1kaYN8+Qu9HExYLv9qyob8BeDoT7dx5Y5 oNzaf0x70DVyEG9wfNv+/bwLt7r3qLo7CT61sh/y4UlP+hMtgmEbUdDN1F5+2aIpj+f2gq xVqoEIzA5v+Odgr0SURTDMbwxaAD5w6/NXi63kVWqAbkvoQ+TLmcPjAOFbJYZ2sm223dar KADGcIAr5KKCcGt9lnDehsrLXCqmkHzO3NMnsLkJRltdpu9UUIpoiyOavghG9Q== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Zm8340WPSz3C4; Mon, 28 Apr 2025 03:25:20 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 53S3PJuq014701; Mon, 28 Apr 2025 03:25:19 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 53S3PJUC014698; Mon, 28 Apr 2025 03:25:19 GMT (envelope-from git) Date: Mon, 28 Apr 2025 03:25:19 GMT Message-Id: <202504280325.53S3PJUC014698@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Warner Losh Subject: git: e977ed50bb2b - main - mpi3mr: Enhance Controller Enable Retry Logic in Host Drivers List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: imp X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: e977ed50bb2b9fce5e285756673286c5fcfb1f31 Auto-Submitted: auto-generated The branch main has been updated by imp: URL: https://cgit.FreeBSD.org/src/commit/?id=e977ed50bb2b9fce5e285756673286c5fcfb1f31 commit e977ed50bb2b9fce5e285756673286c5fcfb1f31 Author: Chandrakanth patil AuthorDate: 2025-04-27 23:34:54 +0000 Commit: Warner Losh CommitDate: 2025-04-28 03:22:55 +0000 mpi3mr: Enhance Controller Enable Retry Logic in Host Drivers This patch improves the retry logic during the IOC enable process. If a controller faults or if the reset history bit is detected during the ready status check, the driver will retry initialization up to three times or until 510 seconds have passed. A soft reset will also be issued if the controller is detected while waiting for the ready status. This enhances reliability during controller initialization. Reviewed by: ssaxena, imp Differential Revision: https://reviews.freebsd.org/D49746 --- sys/dev/mpi3mr/mpi3mr.c | 216 ++++++++++++++++++++++++++---------------------- 1 file changed, 118 insertions(+), 98 deletions(-) diff --git a/sys/dev/mpi3mr/mpi3mr.c b/sys/dev/mpi3mr/mpi3mr.c index 1825244f2082..5688d6bb8d25 100644 --- a/sys/dev/mpi3mr/mpi3mr.c +++ b/sys/dev/mpi3mr/mpi3mr.c @@ -1147,7 +1147,7 @@ enum mpi3mr_iocstate mpi3mr_get_iocstate(struct mpi3mr_softc *sc) return MRIOC_STATE_RESET_REQUESTED; } -static inline void mpi3mr_clear_resethistory(struct mpi3mr_softc *sc) +static inline void mpi3mr_clear_reset_history(struct mpi3mr_softc *sc) { U32 ioc_status; @@ -1177,7 +1177,7 @@ static int mpi3mr_mur_ioc(struct mpi3mr_softc *sc, U16 reset_reason) mpi3mr_dprint(sc, MPI3MR_ERROR, "IOC is unrecoverable MUR not issued\n"); return retval; } - mpi3mr_clear_resethistory(sc); + mpi3mr_clear_reset_history(sc); scratch_pad0 = ((MPI3MR_RESET_REASON_OSTYPE_FREEBSD << MPI3MR_RESET_REASON_OSTYPE_SHIFT) | @@ -1192,7 +1192,7 @@ static int mpi3mr_mur_ioc(struct mpi3mr_softc *sc, U16 reset_reason) do { ioc_status = mpi3mr_regread(sc, MPI3_SYSIF_IOC_STATUS_OFFSET); if ((ioc_status & MPI3_SYSIF_IOC_STATUS_RESET_HISTORY)) { - mpi3mr_clear_resethistory(sc); + mpi3mr_clear_reset_history(sc); ioc_config = mpi3mr_regread(sc, MPI3_SYSIF_IOC_CONFIG_OFFSET); if (!((ioc_status & MPI3_SYSIF_IOC_STATUS_READY) || @@ -1222,24 +1222,44 @@ static int mpi3mr_mur_ioc(struct mpi3mr_softc *sc, U16 reset_reason) * * Return: 0 on success, appropriate error on failure. */ -static int mpi3mr_bring_ioc_ready(struct mpi3mr_softc *sc) +static int mpi3mr_bring_ioc_ready(struct mpi3mr_softc *sc, + U64 *start_time) { - U32 ioc_config, timeout; - enum mpi3mr_iocstate current_state; + enum mpi3mr_iocstate current_state; + U32 ioc_status; + int retval; - ioc_config = mpi3mr_regread(sc, MPI3_SYSIF_IOC_CONFIG_OFFSET); - ioc_config |= MPI3_SYSIF_IOC_CONFIG_ENABLE_IOC; + U32 ioc_config = mpi3mr_regread(sc, MPI3_SYSIF_IOC_CONFIG_OFFSET); + ioc_config |= MPI3_SYSIF_IOC_CONFIG_ENABLE_IOC; mpi3mr_regwrite(sc, MPI3_SYSIF_IOC_CONFIG_OFFSET, ioc_config); - timeout = sc->ready_timeout * 10; - do { - current_state = mpi3mr_get_iocstate(sc); - if (current_state == MRIOC_STATE_READY) - return 0; - DELAY(100 * 1000); - } while (--timeout); + if (*start_time == 0) + *start_time = ticks; + + do { + ioc_status = mpi3mr_regread(sc, MPI3_SYSIF_IOC_STATUS_OFFSET); + if (ioc_status & (MPI3_SYSIF_IOC_STATUS_FAULT | MPI3_SYSIF_IOC_STATUS_RESET_HISTORY)) { + if (ioc_status & MPI3_SYSIF_IOC_STATUS_FAULT) { + mpi3mr_print_fault_info(sc); + retval = mpi3mr_issue_reset(sc, MPI3_SYSIF_HOST_DIAG_RESET_ACTION_SOFT_RESET, MPI3MR_RESET_FROM_BRINGUP); + if (retval) { + mpi3mr_dprint(sc, MPI3MR_ERROR, "%s: Failed to soft reset the IOC, error 0x%d\n", __func__, retval); + return -1; + } + } + mpi3mr_clear_reset_history(sc); + return EAGAIN; + } + + current_state = mpi3mr_get_iocstate(sc); + if (current_state == MRIOC_STATE_READY) + return 0; + + DELAY(100 * 1000); - return -1; + } while (((ticks - *start_time) / hz) < sc->ready_timeout); + + return -1; } static const struct { @@ -2717,14 +2737,16 @@ int mpi3mr_initialize_ioc(struct mpi3mr_softc *sc, U8 init_type) { int retval = 0; enum mpi3mr_iocstate ioc_state; - U64 ioc_info; + U64 ioc_info, start_ticks = 0; U32 ioc_status, ioc_control, i, timeout; Mpi3IOCFactsData_t facts_data; char str[32]; U32 size; + U8 retry = 0; sc->cpu_count = mp_ncpus; +retry_init: ioc_status = mpi3mr_regread(sc, MPI3_SYSIF_IOC_STATUS_OFFSET); ioc_control = mpi3mr_regread(sc, MPI3_SYSIF_IOC_CONFIG_OFFSET); ioc_info = mpi3mr_regread64(sc, MPI3_SYSIF_IOC_INFO_LOW_OFFSET); @@ -2732,28 +2754,25 @@ int mpi3mr_initialize_ioc(struct mpi3mr_softc *sc, U8 init_type) mpi3mr_dprint(sc, MPI3MR_INFO, "SOD ioc_status: 0x%x ioc_control: 0x%x " "ioc_info: 0x%lx\n", ioc_status, ioc_control, ioc_info); - /*The timeout value is in 2sec unit, changing it to seconds*/ + /*The timeout value is in 2sec unit, changing it to seconds*/ sc->ready_timeout = ((ioc_info & MPI3_SYSIF_IOC_INFO_LOW_TIMEOUT_MASK) >> MPI3_SYSIF_IOC_INFO_LOW_TIMEOUT_SHIFT) * 2; ioc_state = mpi3mr_get_iocstate(sc); - mpi3mr_dprint(sc, MPI3MR_INFO, "IOC state: %s IOC ready timeout: %d\n", mpi3mr_iocstate_name(ioc_state), sc->ready_timeout); - if (ioc_state == MRIOC_STATE_BECOMING_READY || - ioc_state == MRIOC_STATE_RESET_REQUESTED) { - timeout = sc->ready_timeout * 10; - do { - DELAY(1000 * 100); - } while (--timeout); - + timeout = sc->ready_timeout * 10; + do { ioc_state = mpi3mr_get_iocstate(sc); - mpi3mr_dprint(sc, MPI3MR_INFO, - "IOC in %s state after waiting for reset time\n", - mpi3mr_iocstate_name(ioc_state)); - } + + if (ioc_state != MRIOC_STATE_BECOMING_READY && + ioc_state != MRIOC_STATE_RESET_REQUESTED) + break; + + DELAY(1000 * 100); + } while (--timeout); if (ioc_state == MRIOC_STATE_READY) { retval = mpi3mr_mur_ioc(sc, MPI3MR_RESET_FROM_BRINGUP); @@ -2774,44 +2793,48 @@ int mpi3mr_initialize_ioc(struct mpi3mr_softc *sc, U8 init_type) mpi3mr_dprint(sc, MPI3MR_ERROR, "%s :Failed to soft reset IOC, error 0x%d\n", __func__, retval); - goto out_failed; + goto err_retry; } } - + ioc_state = mpi3mr_get_iocstate(sc); if (ioc_state != MRIOC_STATE_RESET) { mpi3mr_dprint(sc, MPI3MR_ERROR, "Cannot bring IOC to reset state\n"); - goto out_failed; + goto err_retry; } retval = mpi3mr_setup_admin_qpair(sc); if (retval) { mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to setup Admin queues, error 0x%x\n", retval); - goto out_failed; + if (retval == ENOMEM) + goto err; + goto err_retry; } - - retval = mpi3mr_bring_ioc_ready(sc); + + retval = mpi3mr_bring_ioc_ready(sc, &start_ticks); if (retval) { - mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to bring IOC ready, error 0x%x\n", - retval); - goto out_failed; + mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to bring IOC ready, error 0x%x\n", retval); + if (retval == EAGAIN) + goto err_retry; + goto err; } + if (init_type == MPI3MR_INIT_TYPE_INIT) { retval = mpi3mr_alloc_interrupts(sc, 1); if (retval) { mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to allocate interrupts, error 0x%x\n", retval); - goto out_failed; + goto err; } - + retval = mpi3mr_setup_irqs(sc); if (retval) { mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to setup ISR, error 0x%x\n", retval); - goto out_failed; + goto err; } } @@ -2871,25 +2894,30 @@ int mpi3mr_initialize_ioc(struct mpi3mr_softc *sc, U8 init_type) retval = mpi3mr_issue_iocfacts(sc, &facts_data); if (retval) { - mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to Issue IOC Facts, retval: 0x%x\n", + mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to Issue IOC Facts, error: 0x%x\n", retval); - goto out_failed; + if (retval == ENOMEM) + goto err; + goto err_retry; } retval = mpi3mr_process_factsdata(sc, &facts_data); if (retval) { - mpi3mr_dprint(sc, MPI3MR_ERROR, "IOC Facts data processing failedi, retval: 0x%x\n", + mpi3mr_dprint(sc, MPI3MR_ERROR, "IOC Facts data processing failed, error: 0x%x\n", retval); - goto out_failed; + goto err_retry; } sc->num_io_throttle_group = sc->facts.max_io_throttle_group; mpi3mr_atomic_set(&sc->pend_large_data_sz, 0); - + if (init_type == MPI3MR_INIT_TYPE_RESET) { retval = mpi3mr_validate_fw_update(sc); - if (retval) - goto out_failed; + if (retval) { + if (retval == ENOMEM) + goto err; + goto err_retry; + } } else { sc->reply_sz = sc->facts.reply_sz; } @@ -2898,25 +2926,27 @@ int mpi3mr_initialize_ioc(struct mpi3mr_softc *sc, U8 init_type) retval = mpi3mr_reply_alloc(sc); if (retval) { - mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to allocated reply and sense buffers, retval: 0x%x\n", + mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to allocated reply and sense buffers, error: 0x%x\n", retval); - goto out_failed; + goto err; } - + if (init_type == MPI3MR_INIT_TYPE_INIT) { retval = mpi3mr_alloc_chain_bufs(sc); if (retval) { - mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to allocated chain buffers, retval: 0x%x\n", - retval); - goto out_failed; + mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to allocated chain buffers, error: 0x%x\n", + retval); + goto err; } } - + retval = mpi3mr_issue_iocinit(sc); if (retval) { - mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to Issue IOC Init, retval: 0x%x\n", - retval); - goto out_failed; + mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to Issue IOC Init, error: 0x%x\n", + retval); + if (retval == ENOMEM) + goto err; + goto err_retry; } mpi3mr_print_fw_pkg_ver(sc); @@ -2924,77 +2954,84 @@ int mpi3mr_initialize_ioc(struct mpi3mr_softc *sc, U8 init_type) sc->reply_free_q_host_index = sc->num_reply_bufs; mpi3mr_regwrite(sc, MPI3_SYSIF_REPLY_FREE_HOST_INDEX_OFFSET, sc->reply_free_q_host_index); - + sc->sense_buf_q_host_index = sc->num_sense_bufs; - + mpi3mr_regwrite(sc, MPI3_SYSIF_SENSE_BUF_FREE_HOST_INDEX_OFFSET, sc->sense_buf_q_host_index); if (init_type == MPI3MR_INIT_TYPE_INIT) { retval = mpi3mr_alloc_interrupts(sc, 0); if (retval) { - mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to allocate interrupts, retval: 0x%x\n", + mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to allocate interrupts, error: 0x%x\n", retval); - goto out_failed; + goto err; } retval = mpi3mr_setup_irqs(sc); if (retval) { - printf(IOCNAME "Failed to setup ISR, error: 0x%x\n", - sc->name, retval); - goto out_failed; + mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to setup ISR, error: 0x%x\n", retval); + goto err; } mpi3mr_enable_interrupts(sc); } else mpi3mr_enable_interrupts(sc); - - retval = mpi3mr_create_op_queues(sc); + retval = mpi3mr_create_op_queues(sc); if (retval) { mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to create operational queues, error: %d\n", retval); - goto out_failed; + if (retval == ENOMEM) + goto err; + goto err_retry; } if (!sc->throttle_groups && sc->num_io_throttle_group) { - mpi3mr_dprint(sc, MPI3MR_ERROR, "allocating memory for throttle groups\n"); size = sizeof(struct mpi3mr_throttle_group_info); sc->throttle_groups = (struct mpi3mr_throttle_group_info *) malloc(sc->num_io_throttle_group * size, M_MPI3MR, M_NOWAIT | M_ZERO); - if (!sc->throttle_groups) - goto out_failed; + if (!sc->throttle_groups) { + mpi3mr_dprint(sc, MPI3MR_ERROR, "throttle groups memory allocation failed\n"); + goto err; + } } if (init_type == MPI3MR_INIT_TYPE_RESET) { - mpi3mr_dprint(sc, MPI3MR_INFO, "Re-register events\n"); + mpi3mr_dprint(sc, MPI3MR_XINFO, "Re-register events\n"); retval = mpi3mr_register_events(sc); if (retval) { - mpi3mr_dprint(sc, MPI3MR_INFO, "Failed to re-register events, retval: 0x%x\n", + mpi3mr_dprint(sc, MPI3MR_INFO, "Failed to re-register events, error: 0x%x\n", retval); - goto out_failed; + goto err_retry; } mpi3mr_dprint(sc, MPI3MR_INFO, "Issuing Port Enable\n"); retval = mpi3mr_issue_port_enable(sc, 0); if (retval) { - mpi3mr_dprint(sc, MPI3MR_INFO, "Failed to issue port enable, retval: 0x%x\n", + mpi3mr_dprint(sc, MPI3MR_INFO, "Failed to issue port enable, error: 0x%x\n", retval); - goto out_failed; + goto err_retry; } } retval = mpi3mr_pel_alloc(sc); if (retval) { - mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to allocate memory for PEL, retval: 0x%x\n", + mpi3mr_dprint(sc, MPI3MR_ERROR, "Failed to allocate memory for PEL, error: 0x%x\n", retval); - goto out_failed; + goto err; } - + return retval; -out_failed: +err_retry: + if ((retry++ < 2) && (((ticks - start_ticks) / hz) < (sc->ready_timeout - 60))) { + mpi3mr_dprint(sc, MPI3MR_ERROR, "Retrying controller initialization," + "retry_count: %d\n", retry); + goto retry_init; + } +err: retval = -1; return retval; } @@ -5734,23 +5771,6 @@ static void mpi3mr_flush_io(struct mpi3mr_softc *sc) } } } -/** - * mpi3mr_clear_reset_history - Clear reset history - * @sc: Adapter instance reference - * - * Write the reset history bit in IOC Status to clear the bit, - * if it is already set. - * - * Return: Nothing. - */ -static inline void mpi3mr_clear_reset_history(struct mpi3mr_softc *sc) -{ - U32 ioc_status; - - ioc_status = mpi3mr_regread(sc, MPI3_SYSIF_IOC_STATUS_OFFSET); - if (ioc_status & MPI3_SYSIF_IOC_STATUS_RESET_HISTORY) - mpi3mr_regwrite(sc, MPI3_SYSIF_IOC_STATUS_OFFSET, ioc_status); -} /** * mpi3mr_set_diagsave - Set diag save bit for snapdump