From nobody Thu Oct 20 16:29:12 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MtY2l0mNbz4g0d5; Thu, 20 Oct 2022 16:29:19 +0000 (UTC) (envelope-from ken@freebsd.org) Received: from mithlond.kdm.org (mithlond.kdm.org [96.89.93.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mithlond.kdm.org", Issuer "mithlond.kdm.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MtY2k6nxqz4DKG; Thu, 20 Oct 2022 16:29:18 +0000 (UTC) (envelope-from ken@freebsd.org) Received: from smtpclient.apple (mbp2021.int.kdm.org [10.0.0.25]) (authenticated bits=0) by mithlond.kdm.org (8.15.2/8.14.9) with ESMTPSA id 29KGTCb0016650 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NO); Thu, 20 Oct 2022 12:29:12 -0400 (EDT) (envelope-from ken@freebsd.org) From: Ken Merry Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_2608340F-DAC8-41FE-A3D5-9D8DA3CFCA3C" List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Subject: Re: git: 11778fca4a83 - main - Fix mpr(4) panic during a firmware update. Date: Thu, 20 Oct 2022 12:29:12 -0400 In-Reply-To: Cc: src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org To: Alan Somers References: <202210171652.29HGqtSf007611@gitrepo.freebsd.org> X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (mithlond.kdm.org [96.89.93.250]); Thu, 20 Oct 2022 12:29:12 -0400 (EDT) X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Rspamd-Queue-Id: 4MtY2k6nxqz4DKG X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_2608340F-DAC8-41FE-A3D5-9D8DA3CFCA3C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Oct 20, 2022, at 12:23, Alan Somers wrote: >=20 > On Mon, Oct 17, 2022 at 10:53 AM Kenneth D. Merry > wrote: >>=20 >> The branch main has been updated by ken: >>=20 >> URL: = https://cgit.FreeBSD.org/src/commit/?id=3D11778fca4a83f5e3b597c75785aa5c0e= e0dc518e >>=20 >> commit 11778fca4a83f5e3b597c75785aa5c0ee0dc518e >> Author: Kenneth D. Merry >> AuthorDate: 2022-10-17 16:48:34 +0000 >> Commit: Kenneth D. Merry >> CommitDate: 2022-10-17 16:48:34 +0000 >>=20 >> Fix mpr(4) panic during a firmware update. >>=20 >> Issue Description: >> The RequestCredits field of IOCFacts got changed between the = Phase23 >> firmware to Phase24 firmware. So as part of firmware update = operation, >> driver has to free the resources & pools which are created with = the Phase23 >> Firmware's IOCFacts data (i.e. during driver load time) and has to >> reallocate the resources and pools using Phase24's IOCFacts data. = Here >> driver has freed the interrupts but missed to reallocate the = interrupts and >> hence config page read operation is getting timed out and = controller is >> going for recursive reinit (controller reset) operations and = leading to >> kernel panic. >>=20 >> Fix: >> Reallocate the interrupts if the interrupts are disabled as part = of >> firmware update/downgrade operation. >>=20 >> Submitted by: Sreekanth Ready >> Tested by: ken >> MFC after: 3 days >=20 > Would this commit fix the panic in bug 252575? >=20 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D252575 = Unfortunately, no. This is the panic I ran into on a firmware upgrade using storcli64 on a = Broadcom 9600-16i card going from Phase 23 to Phase 24: mpr0: Reinitializing controller mpr0: Firmware: 24.00.00.00, Driver: 23.00.00.00-fbsd mpr0: IOCCapabilities: = 2fa84c mpr0: Calling Reinit from mpr_wait_command, timeout=3D30, elapsed=3D30 mpr0: Reinitializing controller mpr_config_get_ioc_pg8: request for header completed with error 0 mpr_config_get_ioc_pg8: request for header completed with error 16 mpr_config_get_ioc_pg8: request for header completed with error 16 mpr_config_get_ioc_pg8: request for header completed with error 16 mpr_config_get_ioc_pg8: request for header completed with error 16 mpr_config_get_ioc_pg8: request for header completed with error 16 mpr_config_get_iounit_pg8: request for header completed with error 16 mpr_config_get_iounit_pg8: request for header completed with error 16 mpr_config_get_iounit_pg8: request for header completed with error 16 mpr_config_get_iounit_pg8: request for header completed with error 16 mpr_config_get_iounit_pg8: request for header completed with error 16 mpr_config_get_iounit_pg8: request for header completed with error 16 mpr_config_get_man_pg11: request for header completed with error 16 mpr_config_get_man_pg11: request for header completed with error 16 mpr_config_get_man_pg11: request for header completed with error 16 mpr_config_get_man_pg11: request for header completed with error 16 mpr_config_get_man_pg11: request for header completed with error 16 mpr_config_get_man_pg11: request for header completed with error 16 mpr_config_get_dpm_pg0: request for header completed with error 16 mpr_config_get_dpm_pg0: request for header completed with error 16 mpr_config_get_dpm_pg0: request for header completed with error 16 mpr_config_get_dpm_pg0: request for header completed with error 16 mpr0: Unfreezing SIM queue mpr0: fault_state(0x40001500)! mpr0: Timeout while writing doorbell panic: mpr_iocfacts_allocate failed to get IOC Facts with error 6 cpuid =3D 1 time =3D 1663771142 KDB: stack backtrace: db_trace_self_wrapper() at 0xffffffff8040a9ab =3D = db_trace_self_wrapper+0x2b/frame 0xfffffe02e759d6c0 vpanic() at 0xffffffff805f7ad1 =3D vpanic+0x151/frame 0xfffffe02e759d710 panic() at 0xffffffff805f7973 =3D panic+0x43/frame 0xfffffe02e759d770 mpr_iocfacts_allocate() at 0xffffffff8154bdad =3D = mpr_iocfacts_allocate+0x15dd/frame 0xfffffe02e759d8e0 mpr_reinit() at 0xffffffff81549e0c =3D mpr_reinit+0x14c/frame = 0xfffffe02e759d930 mpr_wait_command() at 0xffffffff8154fb69 =3D = mpr_wait_command+0x1d9/frame 0xfffffe02e759d9b0 mpr_ioctl() at 0xffffffff8155a113 =3D mpr_ioctl+0x1de3/frame = 0xfffffe02e759db40 devfs_ioctl() at 0xffffffff804bab8f =3D devfs_ioctl+0xaf/frame = 0xfffffe02e759db90 vn_ioctl() at 0xffffffff806e9664 =3D vn_ioctl+0x1a4/frame = 0xfffffe02e759dca0 devfs_ioctl_f() at 0xffffffff804bb21e =3D devfs_ioctl_f+0x1e/frame = 0xfffffe02e759dcc0 kern_ioctl() at 0xffffffff80667b2d =3D kern_ioctl+0x26d/frame = 0xfffffe02e759dd30 sys_ioctl() at 0xffffffff80667811 =3D sys_ioctl+0x101/frame = 0xfffffe02e759de00 amd64_syscall() at 0xffffffff8092ae4c =3D amd64_syscall+0x10c/frame = 0xfffffe02e759df30 fast_syscall_common() at 0xffffffff8090369b =3D = fast_syscall_common+0xf8/frame 0xfffffe02e759df30 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x8dfe3a, rsp =3D = 0x821837048, rbp =3D 0x4 --- Uptime: 18h2m58s Dumping 5338 out of 130903 = MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Ken =E2=80=94=20 Ken Merry ken@FreeBSD.ORG= --Apple-Mail=_2608340F-DAC8-41FE-A3D5-9D8DA3CFCA3C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On Oct 20, 2022, at 12:23, Alan Somers <asomers@freebsd.org>= wrote:

On Mon, Oct 17, 2022 at 10:53 AM Kenneth D. Merry = <ken@freebsd.org> = wrote:

The branch main has been updated by ken:

URL: https://cgit.FreeBSD.org/src/commit/?id=3D11778fca4a83f5e3b597c= 75785aa5c0ee0dc518e

commit = 11778fca4a83f5e3b597c75785aa5c0ee0dc518e
Author: =     Kenneth D. Merry <ken@FreeBSD.org>
AuthorDate: 2022-10-17 16:48:34 +0000
Commit: =     Kenneth D. Merry <ken@FreeBSD.org>
CommitDate: 2022-10-17 16:48:34 +0000

   Fix mpr(4) panic during a firmware = update.

   Issue = Description:
   The RequestCredits field of = IOCFacts got changed between the Phase23
   firmware to Phase24 firmware. So as part of = firmware update operation,
   driver has to = free the resources & pools which are created with the Phase23
   Firmware's IOCFacts data (i.e. during = driver load time) and has to
   reallocate = the resources and pools using Phase24's IOCFacts data. Here
   driver has freed the interrupts but missed = to reallocate the interrupts and
   hence = config page read operation is getting timed out and controller is
   going for recursive reinit (controller = reset) operations and leading to
   kernel = panic.

   Fix:
   Reallocate the interrupts if the interrupts = are disabled as part of
   firmware = update/downgrade operation.

   Submitted by:   Sreekanth Ready = <sreekanth.reddy@broadcom.com>
   Tested by: =      ken
   MFC = after:      3 days
Would this = commit fix the panic in bug 252575?

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D252575<= /div>

Unfortunately, = no.

This is = the panic I ran into on a firmware upgrade using storcli64 on a Broadcom = 9600-16i card going from Phase 23 to Phase 24:

mpr0: Reinitializing controller
mpr0: Firmware: 24.00.00.00, Driver: 23.00.00.00-fbsd
mpr0: IOCCapabilities: = 2fa84c<ScsiTaskFull,DiagTrace,EEDP,TransRetry,EventReplay
,MSIXIndex,HostDisc,FastPath,RDPQArray,AtomicReqDesc>
mpr0: Calling Reinit from mpr_wait_command, timeout=3D30, = elapsed=3D30
mpr0: Reinitializing controller
mpr_config_get_ioc_pg8: request for header completed with = error 0
mpr_config_get_ioc_pg8: request for header = completed with error 16
mpr_config_get_ioc_pg8: request = for header completed with error 16
mpr_config_get_ioc_pg8: = request for header completed with error 16
mpr_config_get_ioc_pg8: request for header completed with = error 16
mpr_config_get_ioc_pg8: request for header = completed with error 16
mpr_config_get_iounit_pg8: request = for header completed with error 16
mpr_config_get_iounit_pg8: request for header completed with = error 16
mpr_config_get_iounit_pg8: request for header = completed with error 16
mpr_config_get_iounit_pg8: request = for header completed with error 16
mpr_config_get_iounit_pg8: request for header completed with = error 16
mpr_config_get_iounit_pg8: request for header = completed with error 16
mpr_config_get_man_pg11: request = for header completed with error 16
mpr_config_get_man_pg11: = request for header completed with error 16
mpr_config_get_man_pg11: request for header completed with = error 16
mpr_config_get_man_pg11: request for header = completed with error 16
mpr_config_get_man_pg11: request = for header completed with error 16
mpr_config_get_man_pg11: = request for header completed with error 16
mpr_config_get_dpm_pg0: request for header completed with = error 16
mpr_config_get_dpm_pg0: request for header = completed with error 16
mpr_config_get_dpm_pg0: request = for header completed with error 16
mpr_config_get_dpm_pg0: = request for header completed with error 16
mpr0: = Unfreezing SIM queue
mpr0: fault_state(0x40001500)!
mpr0: Timeout while writing doorbell
panic: = mpr_iocfacts_allocate failed to get IOC Facts with error 6

cpuid =3D 1
time =3D = 1663771142
KDB: stack backtrace:
db_trace_self_wrapper() at 0xffffffff8040a9ab =3D = db_trace_self_wrapper+0x2b/frame 0xfffffe02e759d6c0
vpanic()= at 0xffffffff805f7ad1 =3D vpanic+0x151/frame 0xfffffe02e759d710
panic() at 0xffffffff805f7973 =3D panic+0x43/frame = 0xfffffe02e759d770
mpr_iocfacts_allocate() at = 0xffffffff8154bdad =3D mpr_iocfacts_allocate+0x15dd/frame = 0xfffffe02e759d8e0
mpr_reinit() at 0xffffffff81549e0c =3D = mpr_reinit+0x14c/frame 0xfffffe02e759d930
mpr_wait_command()= at 0xffffffff8154fb69 =3D mpr_wait_command+0x1d9/frame = 0xfffffe02e759d9b0
mpr_ioctl() at 0xffffffff8155a113 =3D = mpr_ioctl+0x1de3/frame 0xfffffe02e759db40
devfs_ioctl() at = 0xffffffff804bab8f =3D devfs_ioctl+0xaf/frame 0xfffffe02e759db90
vn_ioctl() at 0xffffffff806e9664 =3D vn_ioctl+0x1a4/frame = 0xfffffe02e759dca0
devfs_ioctl_f() at 0xffffffff804bb21e =3D= devfs_ioctl_f+0x1e/frame 0xfffffe02e759dcc0
kern_ioctl() = at 0xffffffff80667b2d =3D kern_ioctl+0x26d/frame 0xfffffe02e759dd30
sys_ioctl() at 0xffffffff80667811 =3D sys_ioctl+0x101/frame = 0xfffffe02e759de00
amd64_syscall() at 0xffffffff8092ae4c =3D= amd64_syscall+0x10c/frame 0xfffffe02e759df30
fast_syscall_common() at 0xffffffff8090369b =3D = fast_syscall_common+0xf8/frame 0xfffffe02e759df30
--- = syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x8dfe3a, rsp =3D = 0x821837048, rbp =3D 0x4 ---
Uptime: 18h2m58s
Dumping 5338 out of 130903 = MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Ken
=E2=80=94 
Ken Merry
ken@FreeBSD.ORG
= --Apple-Mail=_2608340F-DAC8-41FE-A3D5-9D8DA3CFCA3C--