Re: cxgbe firmware error on latest RELENG_13
- Reply: mike tancsa : "Re: cxgbe firmware error on latest RELENG_13"
- In reply to: mike tancsa : "cxgbe firmware error on latest RELENG_13"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Mon, 28 Feb 2022 19:43:22 UTC
On 2/28/22 11:36 AM, mike tancsa wrote: > Hi Navdeep, > > I updated a pair of releng13 boxes today as I saw there are new > drivers/firmware. However, the firmware seems to fail on both in > different ways > > t5nex0: <Chelsio T540-CR> mem > 0x91300000-0x9137ffff,0x90000000-0x90ffffff,0x91984000-0x91985fff irq 17 > at device 0.4 on pci2 > t5nex0: failed to load firmware module: 2. cfg 0, fw 0 > t5nex0: firmware on card (1.25.0.40) is older than the version bundled > with this driver, but the driver cannot take corrective action because > it is unable to load the firmware module. It looks like the firmware KLD isn't available even though you have t5fw_cfg_load="yes" in loader.conf. Can you try removing the cxgbe lines from loader.conf temporarily and kldload if_cxgbe manually after boot? It should be able to find the KLD with the firmware and update the card. Then restore the lines in loader.conf and see if it comes up properly. Regards, Navdeep > t5nex0: KLD with default config is not available. > t5nex0: failed to upload config file to card: 2. > t5nex0: failed (2) to configure card with "default" profile, will fall > back to a basic configuration and retry. > cxl0: <port 0> on t5nex0 > cxl0: Ethernet address: 00:07:43:29:98:90 > cxl0: 12 txq, 8 rxq (NIC) > cxl1: <port 1> on t5nex0 > cxl1: Ethernet address: 00:07:43:29:98:98 > cxl1: 12 txq, 8 rxq (NIC) > cxl2: <port 2> on t5nex0 > cxl2: Ethernet address: 00:07:43:29:98:a0 > cxl2: 12 txq, 8 rxq (NIC) > cxl3: <port 3> on t5nex0 > cxl3: Ethernet address: 00:07:43:29:98:a8 > cxl3: 12 txq, 8 rxq (NIC) > t5nex0: PCIe gen3 x8, 4 ports, 34 MSI-X interrupts, 84 eq, 33 iq > > And on another adapter, different box > > t5nex0: <Chelsio T520-LL-CR> mem > 0xb2300000-0xb237ffff,0xb1000000-0xb1ffffff,0xb28a0000-0xb28a1fff irq 16 > at device 0.4 on pci1 > t5nex0: command 0x4 in mbox 4 timed out (0x80000000). > t5nex0: mbox 4 cmdsent 04a0000000000001 0f41000000000000 > 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > 0000000000000000 0000000000000000 > t5nex0: mbox 4 current 04a0000000000001 0f41000000000000 > 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > 0000000000000000 0000000000000000 > t5nex0: firmware reports adapter error: Crash (0x80000000) > t5nex0: device log follows. > Seq# Tstamp Level Facility Message > 0 507376 INFO CORE log initialized @ > 0x200a0000 size 32768 (128 entries) fwrev 0x011a0200 pcie_fw 0x00000000 > 1 510262 INFO CORE bootstrap firmware > took 605 msecs to run > 2 511784 INFO CORE Serial Configuration > version: 0x11101000 VPD version: 0x2 > 3 511785 NOTICE CORE pcie: Serial > Configuration SPARE3 has Cookie Generation enabled. > 4 511786 INFO CORE pcie: npf 7 (pfbitmap > 0x7f) nvf 64 (pf 0..7 0x1010101000000000) vfstride 4 > t5nex0: encountered fatal error, adapter stopped. > t5nex0: firmware reports adapter error: Crash (0x80000000) > t5nex0: device log follows. > Seq# Tstamp Level Facility Message > 0 507376 INFO CORE log initialized @ > 0x200a0000 size 32768 (128 entries) fwrev 0x011a0200 pcie_fw 0x00000000 > 1 510262 INFO CORE bootstrap firmware > took 605 msecs to run > 2 511784 INFO CORE Serial Configuration > version: 0x11101000 VPD version: 0x2 > 3 511785 NOTICE CORE pcie: Serial > Configuration SPARE3 has Cookie Generation enabled. > 4 511786 INFO CORE pcie: npf 7 (pfbitmap > 0x7f) nvf 64 (pf 0..7 0x1010101000000000) vfstride 4 > t5nex0: failed to connect to the firmware: 6, 0. PCIE_FW 0x80000000 > t5nex0: error during attach, adapter is now in recovery mode. > > none0@pci0:1:0:5: class=0x010000 rev=0x00 hdr=0x00 vendor=0x1425 > device=0x5511 subvendor=0x1425 subdevice=0x0000 > vendor = 'Chelsio Communications Inc' > device = 'T520-LL-CR Unified Wire Storage Controller' > class = mass storage > subclass = SCSI > bar [10] = type Memory, range 64, base 0xb2280000, size 524288, > enabled > bar [18] = type Memory, range 64, base 0xb2200000, size 524288, > enabled > bar [20] = type Memory, range 64, base 0xb2890000, size 8192, > enabled > cap 01[40] = powerspec 3 supports D0 D3 current D0 > cap 05[50] = MSI supports 32 messages, 64 bit, vector masks > cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO > max read 512 > link x8(x8) speed 8.0(8.0) ASPM disabled(L0s/L1) > cap 11[b0] = MSI-X supports 40 messages > Table in map 0x20[0x0], PBA in map 0x20[0x1000] > cap 03[d0] = VPD > ecap 0001[100] = AER 2 1 fatal 0 non-fatal 0 corrected > ecap 0003[170] = Serial 1 0000000000000000 > ecap 000e[190] = ARI 1 > ecap 0019[1a0] = PCIe Sec 1 lane errors 0 > ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI > disabled > 0 VFs configured out of 0 supported > First VF RID Offset 0x0008, VF RID Stride 0x0004 > VF Device ID 0x5811 > Page Sizes: 4096 (enabled), 8192, 65536, 262144, > 1048576, 4194304 > ecap 0017[200] = TPH Requester 1 > none1@pci0:1:0:6: class=0x0c0400 rev=0x00 hdr=0x00 vendor=0x1425 > device=0x5611 subvendor=0x1425 subdevice=0x0000 > vendor = 'Chelsio Communications Inc' > device = 'T520-LL-CR Unified Wire Storage Controller' > class = serial bus > subclass = Fibre Channel > bar [10] = type Memory, range 64, base 0xb2100000, size 524288, > enabled > bar [18] = type Memory, range 64, base 0xb2080000, size 524288, > enabled > bar [20] = type Memory, range 64, base 0xb2880000, size 8192, > enabled > cap 01[40] = powerspec 3 supports D0 D3 current D0 > cap 05[50] = MSI supports 32 messages, 64 bit, vector masks > cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO > max read 512 > link x8(x8) speed 8.0(8.0) ASPM disabled(L0s/L1) > cap 11[b0] = MSI-X supports 40 messages > Table in map 0x20[0x0], PBA in map 0x20[0x1000] > cap 03[d0] = VPD > ecap 0001[100] = AER 2 1 fatal 0 non-fatal 0 corrected > ecap 0003[170] = Serial 1 0000000000000000 > ecap 000e[190] = ARI 1 > ecap 0019[1a0] = PCIe Sec 1 lane errors 0 > ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI > disabled > 0 VFs configured out of 0 supported > First VF RID Offset 0x0008, VF RID Stride 0x0004 > VF Device ID 0x5811 > Page Sizes: 4096 (enabled), 8192, 65536, 262144, > 1048576, 4194304 > ecap 0017[200] = TPH Requester 1 > > > t5iov3@pci0:2:0:3: class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 > device=0x5003 subvendor=0x1425 subdevice=0x0000 > vendor = 'Chelsio Communications Inc' > device = 'T540-CR Unified Wire Ethernet Controller' > class = network > subclass = ethernet > bar [10] = type Memory, range 64, base 0x91480000, size 524288, > enabled > bar [18] = type Memory, range 64, base 0x91400000, size 524288, > enabled > bar [20] = type Memory, range 64, base 0x91986000, size 8192, > enabled > cap 01[40] = powerspec 3 supports D0 D3 current D0 > cap 05[50] = MSI supports 8 messages, 64 bit, vector masks > cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO > max read 512 > link x8(x8) speed 8.0(8.0) ASPM L0s/L1(L0s/L1) > cap 11[b0] = MSI-X supports 8 messages > Table in map 0x20[0x0], PBA in map 0x20[0x1000] > cap 03[d0] = VPD > ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected > ecap 0003[170] = Serial 1 0000000000000000 > ecap 000e[190] = ARI 1 > ecap 0019[1a0] = PCIe Sec 1 lane errors 0 > ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI > disabled > 0 VFs configured out of 16 supported > First VF RID Offset 0x0008, VF RID Stride 0x0004 > VF Device ID 0x5803 > Page Sizes: 4096 (enabled), 8192, 65536, 262144, > 1048576, 4194304 > ecap 0017[200] = TPH Requester 1 > t5nex0@pci0:2:0:4: class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 > device=0x5403 subvendor=0x1425 subdevice=0x0000 > vendor = 'Chelsio Communications Inc' > device = 'T540-CR Unified Wire Ethernet Controller' > class = network > subclass = ethernet > bar [10] = type Memory, range 64, base 0x91300000, size 524288, > enabled > bar [18] = type Memory, range 64, base 0x90000000, size 16777216, > enabled > bar [20] = type Memory, range 64, base 0x91984000, size 8192, > enabled > cap 01[40] = powerspec 3 supports D0 D3 current D0 > cap 05[50] = MSI supports 32 messages, 64 bit, vector masks > cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR > max read 4096 > link x8(x8) speed 8.0(8.0) ASPM L0s/L1(L0s/L1) > cap 11[b0] = MSI-X supports 128 messages, enabled > Table in map 0x20[0x0], PBA in map 0x20[0x1000] > cap 03[d0] = VPD > ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected > ecap 0003[170] = Serial 1 0000000000000000 > ecap 000e[190] = ARI 1 > ecap 0019[1a0] = PCIe Sec 1 lane errors 0 > ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI > disabled > 0 VFs configured out of 0 supported > First VF RID Offset 0x0008, VF RID Stride 0x0004 > VF Device ID 0x5803 > Page Sizes: 4096 (enabled), 8192, 65536, 262144, > 1048576, 4194304 > ecap 0017[200] = TPH Requester 1 > none7@pci0:2:0:5: class=0x010000 rev=0x00 hdr=0x00 vendor=0x1425 > device=0x5503 subvendor=0x1425 subdevice=0x0000 > vendor = 'Chelsio Communications Inc' > device = 'T540-CR Unified Wire Storage Controller' > class = mass storage > subclass = SCSI > bar [10] = type Memory, range 64, base 0x91280000, size 524288, > enabled > bar [18] = type Memory, range 64, base 0x91200000, size 524288, > enabled > bar [20] = type Memory, range 64, base 0x91982000, size 8192, > enabled > cap 01[40] = powerspec 3 supports D0 D3 current D0 > cap 05[50] = MSI supports 32 messages, 64 bit, vector masks > cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO > max read 512 > link x8(x8) speed 8.0(8.0) ASPM L0s/L1(L0s/L1) > cap 11[b0] = MSI-X supports 40 messages > Table in map 0x20[0x0], PBA in map 0x20[0x1000] > cap 03[d0] = VPD > ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected > ecap 0003[170] = Serial 1 0000000000000000 > ecap 000e[190] = ARI 1 > ecap 0019[1a0] = PCIe Sec 1 lane errors 0 > ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI > disabled > 0 VFs configured out of 0 supported > First VF RID Offset 0x0008, VF RID Stride 0x0004 > VF Device ID 0x5803 > Page Sizes: 4096 (enabled), 8192, 65536, 262144, > 1048576, 4194304 > ecap 0017[200] = TPH Requester 1 > none8@pci0:2:0:6: class=0x0c0400 rev=0x00 hdr=0x00 vendor=0x1425 > device=0x5603 subvendor=0x1425 subdevice=0x0000 > vendor = 'Chelsio Communications Inc' > device = 'T540-CR Unified Wire Storage Controller' > class = serial bus > subclass = Fibre Channel > bar [10] = type Memory, range 64, base 0x91100000, size 524288, > enabled > bar [18] = type Memory, range 64, base 0x91080000, size 524288, > enabled > bar [20] = type Memory, range 64, base 0x91980000, size 8192, > enabled > cap 01[40] = powerspec 3 supports D0 D3 current D0 > cap 05[50] = MSI supports 32 messages, 64 bit, vector masks > cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO > max read 512 > link x8(x8) speed 8.0(8.0) ASPM L0s/L1(L0s/L1) > cap 11[b0] = MSI-X supports 40 messages > Table in map 0x20[0x0], PBA in map 0x20[0x1000] > cap 03[d0] = VPD > ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected > ecap 0003[170] = Serial 1 0000000000000000 > ecap 000e[190] = ARI 1 > ecap 0019[1a0] = PCIe Sec 1 lane errors 0 > ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI > disabled > 0 VFs configured out of 0 supported > First VF RID Offset 0x0008, VF RID Stride 0x0004 > VF Device ID 0x5803 > Page Sizes: 4096 (enabled), 8192, 65536, 262144, > 1048576, 4194304 > ecap 0017[200] = TPH Requester 1 > > # git status > On branch stable/13 > Your branch is up to date with 'origin/stable/13'. > > nothing to commit, working tree clean > > 13.1-PRERELEASE FreeBSD 13.1-PRERELEASE #1 stable/13-226bb05eb: Mon Feb > 28 15:03:59 EST 2022 > > Loader.conf is > > > t5fw_cfg_load="YES" > if_cxgbe_load="YES" > > hw.cxgbe.pause_settings="0" > hw.cxgbe.attack_filter="1" > hw.cxgbe.drop_pkts_with_l3_errors="1" > hw.cxgbe.fcoecaps_allowed="0" > > ---Mike >