RE: [EXTERNAL] Re: What is scbus and how it is numbered on FreeBSD?

Many thanks, Warner!

Agree. Wish these could be better documented.

Wei


From: Warner Losh <imp@bsdimp.com>
Sent: Wednesday, March 5, 2025 3:54 AM
To: Wei Hu <weh@microsoft.com>
Cc: Li-Wen Hsu <lwhsu@freebsd.org>; FreeBSD Hackers <freebsd-hackers@freebsd.org>
Subject: [EXTERNAL] Re: What is scbus and how it is numbered on FreeBSD?



On Wed, Feb 26, 2025 at 9:01 PM Wei Hu <weh@microsoft.com<mailto:weh@microsoft.com>> wrote:

Hi Warner,



I have some questions regarding the FreeBSD scbus and its relationship to the SCSI concept of bus (of bus:target:lun). Multiple people inside Microsoft raised same questions recently and I am not exactly sure. For example, my FreeBSD VM has 3 SCSI disks and 2 Nvme disks. Dmesg shows:



# dmesg | grep "da[0-9] at"

da0 at blkvsc0 bus 0 scbus2 target 0 lun 0

da1 at blkvsc1 bus 0 scbus3 target 1 lun 0

da2 at storvsc3 bus 0 scbus5 target 0 lun 0

nda0 at nvme0 bus 0 scbus6 target 0 lun 1

nda1 at nvme1 bus 0 scbus7 target 0 lun 1



# camcontral devlist

<Msft Virtual Disk 1.0>            at scbus2 target 0 lun 0 (da0,pass0)

<Msft Virtual Disk 1.0>            at scbus3 target 1 lun 0 (da1,pass1)

<Msft Virtual Disk 1.0>            at scbus5 target 0 lun 0 (da2,pass2)

<Microsoft NVMe Direct Disk NVMDV001>  at scbus6 target 0 lun 1 (nda0,pass3)

<Microsoft NVMe Direct Disk NVMDV001>  at scbus7 target 0 lun 1 (nda1,pass4)



- Are the scbus numbers showing above referring to the host adapters?

The scbus number is the bus of bus:target:lun. They are assigned sequentially as the different SIMs register with CAM. We never try to reuse numbers if a device departs and a new device arrives. These are allocated in xpt_bus_register() in the sim. The sim has no control over these numbers. Internally to CAM, these are called 'path_id' or similar. These are always unique.


- Are scbus umbered after the discovering order at boot time or based on any rules? Why are scbus0, 1 and 4 missing from dmesg?

One: if a SIM registers and unregisters. This would cause gaps like this. Registering multiple buses can cause this (we only report the periphs present, camcontrol devlist -v will report all the buses). Wired entries can also cause this. I think we'll need to see camcontrol devlist -v to know which of these.


- Is "bus 0" showed in the dmesg output the SCSI concept of bus? Does it have any relationship to the scbus numbers?

'bus 0' here is an extra layer. Years ago, the ahc and ahd devices could support multiple parallel scsi buses on on card as one sim. So you'd see 'bus 0' and 'bus 1' on those systems. Each of those buses would also have a scbus as well. All the traffic for both of these buses would go through the same sim/sim_action() instance.  So there's a weak relationship to scbus numbers since usually bus 0 as scbusN will have a bus 1 as scbusN+1 because we serialize registration and these drivers register 0 and 1. There's a surprising number of SIMs that do this (most often they call this parameter a 'channel' which is a better term imho).

These could all be better documented.

Warner


Thanks so much,

Wei