[Bug 295491] sesutil(8) does not list any drive information

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=295491

--- Comment #1 from Michael Osipov <michaelo@FreeBSD.org> ---
  Root cause found, with workaround

  I dug into the ses(4) and mrsas(4) source to understand why the Array Device
Slot elements show Status: OK but no Device Names/Model/Ident/Size.

  Diagnostics performed:

  Dumped the raw SES Additional Element Status diagnostic page (page 0x0A)
directly from /dev/ses0:

  camcontrol cmd /dev/ses0 -c "1C 01 0A 10 00 00" -i 4096 - > /tmp/aes_page.bin
  hexdump -C /tmp/aes_page.bin

  This showed that every one of the 8 Array Device Slot elements has a valid,
non-zero, unique SAS phy_addr (WWN) in its Additional Element Status
descriptor, e.g.:

  Element 0: phy_addr = 50 00 03 9e 68 01 32 f2
  Element 1: phy_addr = 50 00 03 9e 68 01 1a 9a
  Element 2: phy_addr = 50 00 c5 00 fc 0f 7b 69
  ...

  These correctly distinguish the two 300G boot disks (OUI 5000 39e6...) from
the six 2T data disks (OUI 5000 c500 fc0f...), matching mrsasutil show drives.
So the firmware is supplying correct per-slot SAS
  address correlation data — this rules out a firmware/controller limitation.

  Looking at dmesg, I noticed the enclosure and the disks are enumerated on
different CAM buses:

  ses0 at mrsas0 bus 1 scbus1 target 252 lun 0
  da0 at mrsas0 bus 0 scbus0 target 1 lun 0
  da1 at mrsas0 bus 0 scbus0 target 2 lun 0
  ... (da0-da7 all on bus 0 / scbus0)

  ses(4)'s device-correlation code (ses_path_iter_devid_callback() in
sys/cam/scsi/scsi_enc_ses.c) restricts its XPT_DEV_MATCH lookup to the
enclosure's own CAM bus by default, controlled by the sysctl
  kern.cam.enc.search_globally (default 0). Since ses0 lives on scbus1 and all
the da* devices live on scbus0, the default same-bus search finds nothing to
match against, even though the phy/SAS-address data
  needed for correlation is perfectly valid.

  Workaround confirmed working:

  root@denbgvo010:~
  # sysctl kern.cam.enc.search_globally=1
  kern.cam.enc.search_globally: 0 -> 1
  root@denbgvo010:~
  # camcontrol rescan ses0
  Re-scan of 1:252:0 was successful
  root@denbgvo010:~
  # sesutil show
  ses0: <BROADCOM VirtualSES 03>; ID: 300162b2222a6500
  Desc            Dev     Model                     Ident               
Size/Status
  -               da0     HPE EG000300JWSJP         Y532A0E9FJSL         300G
  -               da1     HPE EG000300JWSJP         Y532A01XFJSL         300G
  -               da2     HPE EG002400JXLWC         WBMAHAQ0             2T
  -               da3     HPE EG002400JXLWC         WBMAHBQY             2T
  -               da4     HPE EG002400JXLWC         WBMAHD4W             2T
  -               da5     HPE EG002400JXLWC         WBMAH4R1             2T
  -               da6     HPE EG002400JXLWC         WBMAHAH4             2T
  -               da7     HPE EG002400JXLWC         WBMAHA3D             2T

  sesutil show now correctly reports Model, Ident, Size, and Device Names for
all 8 slots. To persist this across reboots (the sysctl is CTLFLAG_RWTUN, so it
also applies as a boot-time tunable):

  echo "kern.cam.enc.search_globally=1" >> /etc/sysctl.conf

  Suggested follow-up for FreeBSD:

  - At minimum, this sysctl and its relevance to controllers like
mrsas(4)/MegaRAID that expose their SES target on a separate CAM bus from the
drives they manage should be documented (e.g. in sesutil(8),
  ses(4), or mrsasutil(8) man pages), since this is a legitimate, apparently
common topology for this HBA family.
  - It may also be worth reconsidering whether search_globally should default
to 1, or whether mrsas(4) could/should enumerate its virtual SES target on the
same bus as the PDs it manages, to avoid requiring
  this workaround at all.

  I suspect this is also the explanation for the older Bug 248630 I referenced
initially, though I haven't independently verified that report's
hardware/topology.

-- 
You are receiving this mail because:
You are the assignee for the bug.