From nobody Tue Mar 04 19:53:50 2025 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Z6mbL1HPmz5qc80 for ; Tue, 04 Mar 2025 19:54:06 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Z6mbK2HLKz3rhd for ; Tue, 04 Mar 2025 19:54:05 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20230601.gappssmtp.com header.s=20230601 header.b=mNl3TSlZ; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::102c) smtp.mailfrom=wlosh@bsdimp.com Received: by mail-pj1-x102c.google.com with SMTP id 98e67ed59e1d1-2fa8ada6662so12450897a91.1 for ; Tue, 04 Mar 2025 11:54:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1741118044; x=1741722844; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Cev6icgv7tylqn5n2yWPxP4AXHgwBNPCVIKASqym2g8=; b=mNl3TSlZOT3jZ3f/Il5izUmRsPlRxL5HaCSTVTrRTwTYi7vmbF1uUID4aenU/40/4/ ks5ur5xrXB0P41Nb4xlX/BXTD59zxS25ICNYP+JjomK0D6DWsTqSdZ28pLV2Idl0Prj4 z27JPAcYtRy56zxP8yc9kDafHoCC6sEPb9CVrsay/K44gDTyQ28ZgZv8rJLEKWr37J4M nJqSLI0zu89ouJByDCR+Dlxa3bvJFIMGkjlgYpK1XiIYTyrELzuk4t45gDc0yoSEpEuz 7bNkqHvu2vB3S2V4HSiq7Z74LUP8HQcs7NldHVgUtEmIObB0HQ0iZ/U4dN0oi6m98m4g CIUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741118044; x=1741722844; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Cev6icgv7tylqn5n2yWPxP4AXHgwBNPCVIKASqym2g8=; b=ugqXTwsjMYTx0GPKrjcbt5kWUa8WXEi7jiVF+QPQ8/V+8DMm9AXscJRnPQezuVWA0G PvylIpOLRoqzyOZDDGHraJlEpLB1H/GwD3lygHDYoosbGfTLKb1lHzdQrSBHTLgRqxAv L1/2O+0Kbwe8nLcK0QaKWfZwMbFqaYeI6ogO32AOuFkh5Sky95fqF/aNTgFP2AAzXA1T jlioqkINrAyopBuqfLCEUQ6rJ/3hshOfMemkS1CIukblNSexGD+dTR0HWNyPJPKivYjj dlxPOrSynqBUBXuczzrCBJqGMTRfBWOnAglMvr0te/Avh0RehJQZ8YWgLGthukEofqrW LAdg== X-Forwarded-Encrypted: i=1; AJvYcCUz81d0g8+SsZtD/MFYIHYZUzRrdwkiuC97ejxcYSmUnMYNbm4gI46pTNzWp5JnZ8g7/o49BqXqSutxR2XVg9c=@freebsd.org X-Gm-Message-State: AOJu0Yyj2gigfb93ozZeyLqXGpkkRWgPtoVlSsUnuavheuy+YPvOqfO2 kyy+IRXEUHGCFRAdGxv/G48bmfeHTUFqCFGibaLZvZaIZxTx4cnhE7aKwLJY89zaiYMgJ5IiCfo gd5pdSVNS0CEOFesOg3bMR6Z5YNFpWVGIiFe0f9ro/XjbjP47YGE= X-Gm-Gg: ASbGncvGa4awTy5NjSbrvy0BzjxJBLEUO9lsGLUvHGGlOOjssTixXEiMpKig1dCHn7P /iLwaDad0Nq6BOZre1xLWGtflN+EEjbhzfFLPZAYp0RLDTnU9wyX0wBJRx8OUo3Vr+LRqIOIy5Z rNrxXPxT58cxjH7uhQYzHjnKAq3620h8kA1RbZ X-Google-Smtp-Source: AGHT+IGppkV5z8Gh6gJpXN+vVeeJKz0A6zv72yI2IJkQfZyfUkvnHs0dqJ3p4P6krwhdzXxRgWnahJcIsUBYfOk9P1c= X-Received: by 2002:a17:90b:274b:b0:2fe:b174:31fe with SMTP id 98e67ed59e1d1-2ff497174f8mr938033a91.2.1741118043892; Tue, 04 Mar 2025 11:54:03 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Tue, 4 Mar 2025 11:53:50 -0800 X-Gm-Features: AQ5f1JoTKZfp1vqoqY7olcEhW1Je2lSMdmfynTPBtFMaxCulwcOsEmPJPoKunrk Message-ID: Subject: Re: What is scbus and how it is numbered on FreeBSD? To: Wei Hu Cc: Li-Wen Hsu , FreeBSD Hackers Content-Type: multipart/alternative; boundary="00000000000084cdad062f89a250" X-Spamd-Result: default: False [-1.73 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.73)[-0.730]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20230601.gappssmtp.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102c:from]; R_SPF_NA(0.00)[no SPF record]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[bsdimp-com.20230601.gappssmtp.com:+] X-Rspamd-Queue-Id: 4Z6mbK2HLKz3rhd X-Spamd-Bar: - --00000000000084cdad062f89a250 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Feb 26, 2025 at 9:01=E2=80=AFPM Wei Hu wrote: > Hi Warner, > > > > I have some questions regarding the FreeBSD scbus and its relationship to > the SCSI concept of bus (of bus:target:lun). Multiple people inside > Microsoft raised same questions recently and I am not exactly sure. For > example, my FreeBSD VM has 3 SCSI disks and 2 Nvme disks. Dmesg shows: > > > > # dmesg | grep "da[0-9] at" > > da0 at blkvsc0 bus 0 scbus2 target 0 lun 0 > > da1 at blkvsc1 bus 0 scbus3 target 1 lun 0 > > da2 at storvsc3 bus 0 scbus5 target 0 lun 0 > > nda0 at nvme0 bus 0 scbus6 target 0 lun 1 > > nda1 at nvme1 bus 0 scbus7 target 0 lun 1 > > > > # camcontral devlist > > at scbus2 target 0 lun 0 (da0,pass0) > > at scbus3 target 1 lun 0 (da1,pass1) > > at scbus5 target 0 lun 0 (da2,pass2) > > at scbus6 target 0 lun 1 > (nda0,pass3) > > at scbus7 target 0 lun 1 > (nda1,pass4) > > > > - Are the scbus numbers showing above referring to the host adapters? > The scbus number is the bus of bus:target:lun. They are assigned sequentially as the different SIMs register with CAM. We never try to reuse numbers if a device departs and a new device arrives. These are allocated in xpt_bus_register() in the sim. The sim has no control over these numbers. Internally to CAM, these are called 'path_id' or similar. These are always unique. > - Are scbus umbered after the discovering order at boot time or based on > any rules? Why are scbus0, 1 and 4 missing from dmesg? > One: if a SIM registers and unregisters. This would cause gaps like this. Registering multiple buses can cause this (we only report the periphs present, camcontrol devlist -v will report all the buses). Wired entries can also cause this. I think we'll need to see camcontrol devlist -v to know which of these. > - Is "bus 0" showed in the dmesg output the SCSI concept of bus? Does it > have any relationship to the scbus numbers? > 'bus 0' here is an extra layer. Years ago, the ahc and ahd devices could support multiple parallel scsi buses on on card as one sim. So you'd see 'bus 0' and 'bus 1' on those systems. Each of those buses would also have a scbus as well. All the traffic for both of these buses would go through the same sim/sim_action() instance. So there's a weak relationship to scbus numbers since usually bus 0 as scbusN will have a bus 1 as scbusN+1 because we serialize registration and these drivers register 0 and 1. There's a surprising number of SIMs that do this (most often they call this parameter a 'channel' which is a better term imho). These could all be better documented. Warner > Thanks so much, > > Wei > --00000000000084cdad062f89a250 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Wed, Feb 26,= 2025 at 9:01=E2=80=AFPM Wei Hu <we= h@microsoft.com> wrote:

Hi Warner,

=C2=A0

I have some questions regard= ing the FreeBSD scbus and its relationship to the SCSI concept of bus (of b= us:target:lun). Multiple people inside Microsoft raised same questions rece= ntly and I am not exactly sure. For example, my FreeBSD VM has 3 SCSI disks and 2 Nvme disks. Dmesg shows:

=C2=A0

# dmesg | grep "da[0-9]= at"

da0 at blkvsc0 bus 0 scbus2 target 0 lun 0

da1 at blkvsc1 bus 0 scbus3 = target 1 lun 0

da2 at storvsc3 bus 0 scbus5= target 0 lun 0

nda0 at nvme0 bus 0 scbus6 t= arget 0 lun 1

nda1 at nvme1 bus 0 scbus7 t= arget 0 lun 1

=C2=A0

# camcontral devlist<= u>

<Msft Virtual Disk 1.0>= ;=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at scbu= s2 target 0 lun 0 (da0,pass0)

<Msft Virtual Disk 1.0>= ;=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at scbu= s3 target 1 lun 0 (da1,pass1)

<Msft Virtual Disk 1.0>= ;=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at scbu= s5 target 0 lun 0 (da2,pass2)

<Microsoft NVMe Direct Di= sk NVMDV001>=C2=A0 at scbus6 target 0 lun 1 (nda0,pass3)

<Microsoft NVMe Direct Di= sk NVMDV001>=C2=A0 at scbus7 target 0 lun 1 (nda1,pass4)

=C2=A0

- Are the scbus numbers show= ing above referring to the host adapters?


The scbus number is the bus of bus:target:lun. They ar= e assigned sequentially as the different SIMs register with CAM. We never t= ry to reuse numbers if a device departs and a new device arrives. These are= allocated in xpt_bus_register() in the sim. The sim has no control over th= ese numbers. Internally to CAM, these are called 'path_id' or simil= ar. These are always unique.
=C2=A0

- Are scbus umbered after th= e discovering order at boot time or based on any rules? Why are scbus0, 1 a= nd 4 missing from dmesg?


<= div>One: if a SIM registers and unregisters. This would cause gaps like thi= s. Registering multiple buses can cause this (we only report the periphs pr= esent, camcontrol devlist -v will report all the buses). Wired entries can = also cause this. I think we'll need to see camcontrol devlist -v to kno= w which of these.
=C2=A0

- Is "bus 0" showed in the dmesg output the SCSI con= cept of bus? Does it have any relationship to the scbus numbers?

<= /div>

'bus 0' here is an extr= a layer. Years ago, the ahc and ahd devices could support multiple parallel= scsi buses on on card as one sim. So you'd see 'bus 0' and = 9;bus 1' on those systems. Each of those buses would also have a scbus = as well. All the traffic for both of these buses would go through the same = sim/sim_action() instance.=C2=A0 So there's a weak relationship to scbu= s numbers since usually bus 0 as scbusN will have a bus 1 as scbusN+1 becau= se we serialize registration and these drivers register 0 and 1. There'= s a surprising number of SIMs that do this (most often they call this param= eter a 'channel' which is a better term imho).

=
These could all be better documented.

Warner<= /div>
=C2=A0

Thanks so much,

Wei =C2=A0

--00000000000084cdad062f89a250--