LSI SAS3008 on 13-RC5 and 14-CURRENT
Daniel Morante
daniel at morante.net
Thu Apr 8 06:04:45 UTC 2021
I done extensive testing with many different configurations. I apologize
in advance for the length of this 😉.
I have an arm64 system with an onboard SAS controller (Broadcom®
SAS3008) and expander (Broadcom® SAS35x36R). This system also has an
onboard SATA controller that is currently unusable in any version of
FreeBSD ARM64
(https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254651). As a result
I am limited to using the SAS3008.
I've noticed strange behavior on 13-RC3/4/5 and on the latest 14-CURRENT
snapshot (2021-04-01).In the controller firmware I can create a logical
volume (RAID) or use the individual drives (non-RAID/pass thru). In RAID
mode there is a only logical volume and no individual drives.
I use the DVD (or disc1) ISO image as the installation media. I load
the kernel module (kldload mpr) using a shell and then proceed to
install FreeBSD using either ZFS or UFS on root.
With 13.0 the drives or logical volumes are detected correctly. On 14.0
no logical volumes are detected.
mpr0: <Avago Technologies (LSI) SAS3008> mem
0x42140000-0x4214ffff,0x42100000-0x4213ffff at device 0.0 numa-domain 0
on pci14 mpr0: Firmware: 15.00.00.00, Driver: 23.00.00.00-fbsd mpr0:
IOCCapabilities:
6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray>
mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000a>
enclosureHandle<0x0002> slot 0 mpr0: At enclosure level 0 and connector
name ( ) mpr0: Found device <81<SataDev>,End Device> <6.0Gbps>
handle<0x000b> enclosureHandle<0x0002> slot 1 mpr0: At enclosure level 0
and connector name ( ) mpr0: Found device
<4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x000c>
enclosureHandle<0x0002> slot 16 mpr0: At enclosure level 0 and connector
name ( )
This volume only shows up on 13.0 and not on 14.0:
da1 at mpr0 bus 0 scbus2 target 0 lun 0 da1: <LSI Logical Volume 3000>
Fixed Direct Access SPC-4 SCSI device da1: Serial Number
2692521697153322079 da1: 150.000MB/s transfers da1: Command Queueing
enabled da1: 226974MB (464842752 512 byte sectors)
In Non-RAID mode individual drives will be shown in both 13.0 and 14.0.
I can proceed through the OS installation, but it's painfully slow.
Towards the end (at which point I assume it's writing bootcode... before
asking to set the root password), the following messages begin to appear
in the console:
mpr0: IOC Fault 0x40002622, Resetting mpr0: Reinitializing controller
mpr0: Firmware: 15.00.00.00, Driver: 23.00.00.00-fbsd mpr0:
IOCCapabilities:
6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray>
mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000a>
enclosureHandle<0x0002> slot 0 mpr0: At enclosure level 0 and connector
name ( ) mpr0: Found device <81<SataDev>,End Device> <6.0Gbps>
handle<0x000b> enclosureHandle<0x0002> slot 1 mpr0: At enclosure level 0
and connector name ( ) mpr0: Found device
<4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x000c>
enclosureHandle<0x0002> slot 16 mpr0: At enclosure level 0 and connector
name ( ) mpr0: IOC Fault 0x40002622, Resetting mpr0: Reinitializing
controller mpr0: Firmware: 15.00.00.00, Driver: 23.00.00.00-fbsd mpr0:
IOCCapabilities:
6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray>
mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000a>
enclosureHandle<0x0002> slot 0 mpr0: At enclosure level 0 and connector
name ( ) mpr0: Found device <81<SataDev>,End Device> <6.0Gbps>
handle<0x000b> enclosureHandle<0x0002> slot 1 mpr0: At enclosure level 0
and connector name ( ) mpr0: Found device
<4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x000c>
enclosureHandle<0x0002> slot 16 mpr0: At enclosure level 0 and connector
name ( ) mpr0: IOC Fault 0x40002622, Resetting mpr0: Reinitializing
controller mpr0: Firmware: 15.00.00.00, Driver: 23.00.00.00-fbsd mpr0:
IOCCapabilities:
6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray>
mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000a>
enclosureHandle<0x0002> slot 0 mpr0: At enclosure level 0 and connector
name ( ) mpr0: Found device <81<SataDev>,End Device> <6.0Gbps>
handle<0x000b> enclosureHandle<0x0002> slot 1 mpr0: At enclosure level 0
and connector name ( ) mpr0: Found device
<4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x000c>
enclosureHandle<0x0002> slot 16 mpr0: At enclosure level 0 and connector
name ( ) mpr0: IOC Fault 0x40002622, Resetting mpr0: Reinitializing
controller mpr0: Firmware: 15.00.00.00, Driver: 23.00.00.00-fbsd mpr0:
IOCCapabilities:
6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray>
(da1:mpr0:0:0:0): Invalidating pack g_vfs_done():da1p1[WRITE(offset=512,
length=4096)]error = 6 g_vfs_done():da1p1[WRITE(offset=33792,
length=4096)]error = 6 da1 at mpr0 bus 0 scbus2 target 0 lun 0 da1: <LSI
Logical Volume 3000> s/n 2692521697153322079 detached
g_vfs_done():da1p1[WRITE(offset=99840, length=16384)]error = 6 mpr0:
Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000a>
enclosureHandle<0x0002> slot 0 mpr0: At enclosure level 0 and connector
name ( ) mpr0: Found device <81<SataDev>,End Device> <6.0Gbps>
handle<0x000b> enclosureHandle<0x0002> slot 1 mpr0: At enclosure level 0
and connector name ( ) mpr0: Found device
<4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x000c>
enclosureHandle<0x0002> slot 16 mpr0: At enclosure level 0 and connector
name ( ) mpr0: IOC Fault 0x40002622, Resetting mpr0: Reinitializing
controller (da1:mpr0:0:0:0): WRITE(10). CDB: 2a 00 00 00 0a cb 00 08 00
00 (da1:mpr0:0:0:0): CAM status: SCSI Bus Reset Sent/Received
(da1:mpr0:0:0:0): Error 5, Periph was invalidated
g_vfs_done():da1p1[WRITE(offset=1394176, length=1048576)]error = 5
(da1:mpr0:0:0:0): WRITE(6). CDB: 0a 00 01 0b 20 00 (da1:mpr0:0:0:0): CAM
status: SCSI Bus Reset Sent/Received (da1:mpr0:0:0:0): Error 5, Periph
was invalidated g_vfs_done():da1p1[WRITE(offset=116224,
length=16384)]error = 5 fsync: giving up on dirty (error = 5)
(da1:mpr0:0:0:0): WRITE(10). CDB: 2a 00 01 d7 0d 08 00 00 08 00
0xffffa0002797b5b8: (da1:mpr0:0:0:0): CAM status: SCSI Bus Reset
Sent/Received type VCHR (da1:mpr0:0:0:0): Error 5, Periph was
invalidated usecount 1, writecount 0, refcount 6 seqc users
0(da1:mpr0:0:0:0): WRITE(10). CDB: 2a 00 01 52 b0 a8 00 00 08 00 rdev
0xffffa0000af73400 (da1:mpr0:0:0:0): CAM status: SCSI Bus Reset
Sent/Received hold count flags () mpr0: Firmware: 15.00.00.00,
Driver: 23.00.00.00-fbsd flags ()mpr0: IOCCapabilities:
6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray>
(da1:mpr0:0:0:0): Error 5, Periph was invalidated v_object
0xffffa00027a6f528 ref 0 pages 27 cleanbuf 2 dirtybuf 1
lock type
devfs: EXCL by thread 0xffffa0000aac3000 (pid 271, syncer, tid 101634)
with exclusive waiters pending dev da1p1 mpr0: Found device
<81<SataDev>,End Device> <6.0Gbps> handle<0x000a>
enclosureHandle<0x0002> slot 0 mpr0: At enclosure level 0 and connector
name ( ) mpr0: Found device <81<SataDev>,End Device> <6.0Gbps>
handle<0x000b> enclosureHandle<0x0002> slot 1 mpr0: At enclosure level 0
and connector name ( ) mpr0: Found device
<4411<SmpInit,SspTarg,SepDev>,End Device> <12.0Gbps> handle<0x000c>
enclosureHandle<0x0002> slot 16 mpr0: At enclosure level 0 and connector
name ( )
The above continues indefinitely and the system eventually locks up (no
kernel panic). I've tried it countless times and it always ends the
same. Regardless of how I configure the controller (RAID 1, 0, or pass
thru).
To rule out a bad controller I test with Ubuntu 20.10 for arm64. No
issues, everything is detected, SAS, onboard SATA, and Ethernet. Not
surprising since Gigabyte has Ubuntu on the compatibility list for this
system. All hardware is confirmed to be functional.
This is were things get strange...
I go back to test FreeBSD 13.0 and I no longer have issues with the SAS
controller (onboard SATA still not working). It seems that
installing/booting Ubuntu did something to 'fix' the problem? Maybe
it
loaded some type of firmware or places some persistent data in UEFI?
(I'm not sure how that works). Regardless I was able to install FreeBSD
on to the SAS controller, and I can boot it.
Has anyone seen this happen before? Should I file another bug report?
14.0 btw, still can't see logical volumes. Should that be a separate
bug report?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4571 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.freebsd.org/pipermail/freebsd-arm/attachments/20210408/b976ab19/attachment.bin>
More information about the freebsd-arm
mailing list