From nobody Thu Nov 10 06:18:02 2022 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4N7BTv4jsdz4f6dR for ; Thu, 10 Nov 2022 06:18:11 +0000 (UTC) (envelope-from crb@chrisbowman.com) Received: from mail-pf1-x430.google.com (mail-pf1-x430.google.com [IPv6:2607:f8b0:4864:20::430]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4N7BTq2nRhz4LNW for ; Thu, 10 Nov 2022 06:18:07 +0000 (UTC) (envelope-from crb@chrisbowman.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=chrisbowman-com.20210112.gappssmtp.com header.s=20210112 header.b=TPtWNgZI; spf=none (mx1.freebsd.org: domain of crb@chrisbowman.com has no SPF policy when checking 2607:f8b0:4864:20::430) smtp.mailfrom=crb@chrisbowman.com; dmarc=none Received: by mail-pf1-x430.google.com with SMTP id b29so975588pfp.13 for ; Wed, 09 Nov 2022 22:18:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chrisbowman-com.20210112.gappssmtp.com; s=20210112; h=to:date:message-id:subject:mime-version:from:from:to:cc:subject :date:message-id:reply-to; bh=1u5FFgpFHIanLOnqo/0YLkw7LmUspUZpj3eTML268i4=; b=TPtWNgZIDh/3AhhIWU5a6gM+B3aPtZ9WJ/hf90gwXqZlUsSiQPuqzmk7F28fnCfmev YFq1zJyc1DP4CqyxjPcfW1eZQ/PeENPNAaFEPAepLBWzZK/k/Df2pokjfI5ryH7DGLSj +fH7+LzBXyv08qKDKPLKRkgiWNJexyPj4OOyBMOw0GuINZ/+iqoERVsuqwRtf4TXmPx6 ujkj46kxFug/rQJToBfhkC/G53H1ppKYQdNneI4rz6VvbsXiBaFpGg2AHiQzOKJLKi/X AZt2iyeYXPquAq/q15C4Q79VRrIAcTYnryHeU/JIajmKkHrZ9HmGKR7FH5YYeduEpKtU GXvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:date:message-id:subject:mime-version:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=1u5FFgpFHIanLOnqo/0YLkw7LmUspUZpj3eTML268i4=; b=JygACxsiBLBK35e+Zvbp3eGoTU8m/UJ/w0GqhZxKq11z8id+Th56Ph5CVud/Eu6iOO LAprpr4hWhqWLnAW7rEjdIWjOlsViIrwmED4Oi/gZ4uYxElI7QvgjEAtZ9oeoY/B1Joi KDU6Qhbkl97MwH4C0PrKyr11DfNNrT88+LSC849nfv3KhxOey2QBLs0JR1WkjE2srbp7 rjfgD1yiI4wyHU4cNeCDqWYPUPJF8nXRaUFoog4IYgmW97fvebjwToN+Twd1ZJgIehFt 9liJoQkY1NEB02xAUKa0DqcDVqXLVOPZQAUDIBckXRxgMY2ut1gK++n/s0QwRtdSpEMA olaA== X-Gm-Message-State: ACrzQf1VFxwVNAl+Mf41izUerE84pZtlLJxa84L0mcYUJr8Xhls3Za17 2TX7uL0zEFQPG4Nw0T0rtXi4W90u+M4dZigz X-Google-Smtp-Source: AMsMyM5Z6WpLnCsG/FBBdTvsAsjt43MXCWTSJ90b4PReQ7w62EQcTB+qhSLkB1IwCfk3J/Qp3A84ow== X-Received: by 2002:a65:5206:0:b0:462:644c:87ff with SMTP id o6-20020a655206000000b00462644c87ffmr1889856pgp.552.1668061085100; Wed, 09 Nov 2022 22:18:05 -0800 (PST) Received: from smtpclient.apple ([2600:1700:5430:10b1:f53e:9382:8311:a0f0]) by smtp.gmail.com with ESMTPSA id iw3-20020a170903044300b00186c3727294sm10160163plb.270.2022.11.09.22.18.04 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 09 Nov 2022 22:18:04 -0800 (PST) From: Christopher Bowman Content-Type: multipart/alternative; boundary="Apple-Mail=_2483E1C2-56D4-4603-A76A-F869C386CB20" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Subject: I could use some help Message-Id: Date: Wed, 9 Nov 2022 22:18:02 -0800 To: hackers@freebsd.org X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.80 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[chrisbowman-com.20210112.gappssmtp.com:s=20210112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::430:from]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DKIM_TRACE(0.00)[chrisbowman-com.20210112.gappssmtp.com:+]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[chrisbowman.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4N7BTq2nRhz4LNW X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_2483E1C2-56D4-4603-A76A-F869C386CB20 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 OK I=E2=80=99m really confused and I could use some help: 13.0 runs fine on my Xilinx Zynq based board (DIgilent Arty Z20). = However when I compile 13.1 release it doesn=E2=80=99t boot. The kernel = stops during boot as follows: Using DTB from loaded file '/boot/dtb/zynq-artyz7.dtb'. Loading DTB overlays: 'artyz7_ssd_overlay.dtb' /boot/dtb/overlays/artyz7_ssd_overlay.dtb size=3D0x1a1 Kernel entry at 0x17a00200... Kernel args: (null) applying DTB overlay '/boot/dtb/overlays/artyz7_ssd_overlay.dtb' WARNING: Cannot find freebsd,dts-version property, cannot check DTB = compliance ---<>--- Copyright (c) 1992-2021 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 .The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-STABLE #22 n248064-ce9c3848ff3: Wed Nov 9 22:04:45 PST = 2022 crb@eclipse.ChrisBowman.com:/usr/obj/usr/src/arm.armv7/sys/ARTYZ7 = arm FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git = llvmorg-11.0.1-0-g43ff75f2c3fe) CPU: ARM Cortex-A9 r3p0 (ECO: 0x00000000) CPU Features:=20 Multiprocessing, Thumb2, Security, VMSAv7, Coherent Walk Optional instructions:=20 UMULL, SMULL, SIMD(ext) LoUU:2 LoC:2 LoUIS:2=20 Cache level 1: 32KB/32B 4-way data cache WB Read-Alloc Write-Alloc 32KB/32B 4-way instruction cache Read-Alloc real memory =3D 536866816 (511 MB) avail memory =3D 515162112 (491 MB) FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs random: unblocking device. random: entropy device external interface ofwbus0: simplebus0: on ofwbus0 simplebus1: on ofwbus0 l2cache0: mem 0xf8f02000-0xf8f02fff irq 8 on = simplebus1 l2cache0: cannot allocate IRQ, not using interrupt l2cache0: Part number: 0x3, release: 0x8 l2cache0: L2 Cache enabled: 512KB/32B 8 ways gic0: mem = 0xf8f01000-0xf8f01fff,0xf8f00100-0xf8f001ff on simplebus1 gic0: pn 0x39, arch 0x1, rev 0x2, implementer 0x43b irqs 96 mp_tmr0: mem 0xf8f00200-0xf8f0021f irq 29 on = simplebus1 Timecounter "MPCore" frequency 50000000 Hz quality 800 mp_tmr1: mem 0xf8f00600-0xf8f0061f irq 36 on = simplebus1 Event timer "MPCore" frequency 50000000 Hz quality 1000 cpulist0: on ofwbus0 cpu0: on cpulist0 cpu1: on cpulist0 uart0: mem 0xe0000000-0xe0000fff irq 9 on simplebus1 uart0: console (-1,n,8,1) zy7_qspi0: mem 0xe000d000-0xe000dfff = irq 13 on simplebus1 zy7_qspi0: must have ref-clock property device_attach: zy7_qspi0 attach returned 6 cgem0: mem = 0xe000b000-0xe000bfff irq 15 on simplebus1 miibus0: on cgem0 rgephy0: PHY 0 on = miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, = 1000baseT-FDX, 1000baseT-FDX-master, auto rgephy1: PHY 1 on = miibus0 rgephy1: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, = 1000baseT-FDX, 1000baseT-FDX-master, auto cgem0: Ethernet address: 56:99:3e:50:9a:8e sdhci_fdt0: mem = 0xe0100000-0xe0100fff irq 17 on simplebus1 sdhci_fdt0: 1 slot(s) allocated mmc0: on sdhci_fdt0 zy7_devcfg0: mem 0xf8007000-0xf80070ff irq 28 on = simplebus1 Timecounters tick every 1.000 msec sdhci_fdt0-slot0: Controller timeout sdhci_fdt0-slot0: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D REGISTER = DUMP =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D sdhci_fdt0-slot0: Sys addr: 0x00060000 | Version: 0x00008901 sdhci_fdt0-slot0: Blk size: 0x00005008 | Blk cnt: 0x00000001 sdhci_fdt0-slot0: Argument: 0x00000000 | Trn mode: 0x00000013 sdhci_fdt0-slot0: Present: 0x01ff0202 | Host ctl: 0x00000001 sdhci_fdt0-slot0: Power: 0x0000000f | Blk gap: 0x00000000 sdhci_fdt0-slot0: Wake-up: 0x00000000 | Clock: 0x00004007 sdhci_fdt0-slot0: Timeout: 0x00000006 | Int stat: 0x00000001 sdhci_fdt0-slot0: Int enab: 0x01ff00fb | Sig enab: 0x01ff00fa sdhci_fdt0-slot0: AC12 err: 0x00000000 | Host ctl2:0x00000000 sdhci_fdt0-slot0: Caps: 0x69ec0080 | Caps2: 0x00000000 sdhci_fdt0-slot0: Max curr: 0x00000001 | ADMA err: 0x00000000 sdhci_fdt0-slot0: ADMA addr:0x00000000 | Slot int: 0x00000000 sdhci_fdt0-slot0: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D sdhci_fdt0-slot0: Controller timeout sdhci_fdt0-slot0: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D REGISTER = DUMP =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D sdhci_fdt0-slot0: Sys addr: 0x00000000 | Version: 0x00008901 sdhci_fdt0-slot0: Blk size: 0x00005008 | Blk cnt: 0x00000001 sdhci_fdt0-slot0: Argument: 0xaaaa0000 | Trn mode: 0x00000013 sdhci_fdt0-slot0: Present: 0x01ff0000 | Host ctl: 0x00000001 sdhci_fdt0-slot0: Power: 0x0000000f | Blk gap: 0x00000000 sdhci_fdt0-slot0: Wake-up: 0x00000000 | Clock: 0x00004007 sdhci_fdt0-slot0: Timeout: 0x00000006 | Int stat: 0x00000001 sdhci_fdt0-slot0: Int enab: 0x01ff00fb | Sig enab: 0x01ff00fa sdhci_fdt0-slot0: AC12 err: 0x00000000 | Host ctl2:0x00000000 sdhci_fdt0-slot0: Caps: 0x69ec0080 | Caps2: 0x00000000 sdhci_fdt0-slot0: Max curr: 0x00000001 | ADMA err: 0x00000000 sdhci_fdt0-slot0: ADMA addr:0x00000000 | Slot int: 0x00000000 sdhci_fdt0-slot0: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D The controller register dumps continue for a while and then I get: mmc0: CMD7 failed, RESULT: 1 Release APs Trying to mount root from ufs:/dev/mmcsd0s2a [rw]... mountroot: waiting for device /dev/mmcsd0s2a... Mounting from ufs:/dev/mmcsd0s2a failed with error 19. Trying to mount root from ufs:mmcsd0s2a []... mountroot: waiting for device mmcsd0s2a... Mounting from ufs:mmcsd0s2a failed with error 19. Loader variables: vfs.root.mountfrom=3Dufs:/dev/mmcsd0s2a vfs.root.mountfrom.options=3Drw Manual root filesystem specification: : [options] Mount using filesystem and with the specified (optional) option list. eg. ufs:/dev/da0s1a zfs:zroot/ROOT/default cd9660:/dev/cd0 ro (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /) ? List valid disk boot devices . Yield 1 second (for background tasks) Abort manual input mountroot>=20 If I replace the kernel with a 13.0 kernel, it boots just fine. So I go and do a git bisect between release/13.0.0 and release/13.1.0 and I get the following: git bisect start '--first-parent' # status: waiting for both good and bad commits # good: [ea31abc261ffc01b6ff5671bffb15cf910a07f4b] 13.0: update to = RELEASE git bisect good ea31abc261ffc01b6ff5671bffb15cf910a07f4b # status: waiting for bad commit, 1 good commit known # bad: [fc952ac2212b121aa6eefc273f5960ec3e0a466d] Update in preparation = of 13.1-RELEASE git bisect bad fc952ac2212b121aa6eefc273f5960ec3e0a466d # skip: [4c44dbde5491516eba8725dc51d39c1dcc817472] MFC jail: Handle a = parent jail when a child is added to it git bisect skip 4c44dbde5491516eba8725dc51d39c1dcc817472 # good: [476f87219f408343846254743c7189076be80c04] wpi: Fix a lock leak = in an error path in wpi_run() git bisect good 476f87219f408343846254743c7189076be80c04 # bad: [05bf7d68c56830e52dee14dc87c07d6716e8195e] aesni: Fix an = out-of-bounds read in AES_GCM_decrypt() git bisect bad 05bf7d68c56830e52dee14dc87c07d6716e8195e # good: [014ae00ef6edca2687d618e0bda138086a1e1230] date: Capitalize = seconds string in synopses git bisect good 014ae00ef6edca2687d618e0bda138086a1e1230 # bad: [08d995ca8f6f1008a10e4bf4d924824c040f842a] swapoff_one(): only = check free pages count manually turning swap off git bisect bad 08d995ca8f6f1008a10e4bf4d924824c040f842a # bad: [81b6dba1a08b031bdf7463c1704d27ae1e0daa0f] ktls: Fix assertion = for TLS 1.0 CBC when using non-zero starting seqno. git bisect bad 81b6dba1a08b031bdf7463c1704d27ae1e0daa0f # bad: [67efa8b29930f12dae2bf237fa7c2ce1dafbd6b1] net80211: add a = driver-private pointer to struct ieee80211_node git bisect bad 67efa8b29930f12dae2bf237fa7c2ce1dafbd6b1 # good: [109330155000bfec215ee39148254d2a0b628798] module(9): Document = that evhand can be NULL git bisect good 109330155000bfec215ee39148254d2a0b628798 # bad: [4c8e29637456bbbe709425f691f637914658009f] LinuxKPI: add = module_pci_driver() and pci_alloc_irq_vectors() git bisect bad 4c8e29637456bbbe709425f691f637914658009f # bad: [4a03ae8d17ddf3d3b57ca281000fd98e200b92cc] nfscl: Fix use after = free for forced dismount git bisect bad 4a03ae8d17ddf3d3b57ca281000fd98e200b92cc # bad: [de957de097857fabb69a59a9ba36276c5e735de5] bhyve: Fix the = WITH_BHYVE_SNAPSHOT build git bisect bad de957de097857fabb69a59a9ba36276c5e735de5 # bad: [5c2e6d9610f1b3f1d7c5d69b925212a7b1fd9391] hwpmc: initialize = arm64 counter/interrupt state git bisect bad 5c2e6d9610f1b3f1d7c5d69b925212a7b1fd9391 # bad: [ce9c3848ff369467749f59fd24f8b9f1241e725c] uma: Fix handling of = reserves in zone_import() git bisect bad ce9c3848ff369467749f59fd24f8b9f1241e725c # good: [d5ebaa6f8f850bb6f6273f01386832efcb295827] uma: Improve = M_USE_RESERVE handling in keg_fetch_slab() git bisect good d5ebaa6f8f850bb6f6273f01386832efcb295827 # first bad commit: [ce9c3848ff369467749f59fd24f8b9f1241e725c] uma: Fix = handling of reserves in zone_import() If I do git log ce9c3848ff369467749f59fd24f8b9f1241e725c it does seem = that d5ebaa6f8f850bb6f6273f01386832efcb295827 is the previous commit and = that it works just fine but ce9c3848ff369467749f59fd24f8b9f1241e725c = doesn=E2=80=99t boot. It=E2=80=99s literally the same file system and = DTB, the only difference is the kernel installed. What=E2=80=99s confusing to me is that looking at = ce9c3848ff369467749f59fd24f8b9f1241e725c I can=E2=80=99t see how that = commit would result in my kernel hanging while booting. I=E2=80=99m clearly not an expert and I=E2=80=99m not sure I used git = bisect right but I think I did. Maybe I need to be on stable/13 when I = bisect? I don=E2=80=99t know and if anyone could give me some pointers I=E2=80=99m= happy to do some leg work but I=E2=80=99ve about come to the end of my = road trying to figure out what=E2=80=99s blocking me. Thanks in advance, Christopher= --Apple-Mail=_2483E1C2-56D4-4603-A76A-F869C386CB20 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 OK = I=E2=80=99m really confused and I could use some help:

13.0 runs fine on my Xilinx Zynq based = board (DIgilent Arty Z20).  However when I compile 13.1 release it = doesn=E2=80=99t boot.  The kernel stops during boot as = follows:

Using DTB from loaded file = '/boot/dtb/zynq-artyz7.dtb'.


Loading DTB overlays: 'artyz7_ssd_overlay.dtb'


/boot/dtb/overlays/artyz7_ssd_overlay.dtb = size=3D0x1a1

Kernel entry at = 0x17a00200...


Kernel args: = (null)


applying DTB overlay = '/boot/dtb/overlays/artyz7_ssd_overlay.dtb'


WARNING: Cannot find freebsd,dts-version property, cannot = check DTB compliance
---<<BOOT>>---
Copyright (c) = 1992-2021 The FreeBSD Project.
Copyright (c) 1979, = 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
.The Regents of the University of California. All rights = reserved.
FreeBSD is a registered trademark of The = FreeBSD Foundation.
FreeBSD 13.0-STABLE #22 = n248064-ce9c3848ff3: Wed Nov  9 22:04:45 PST 2022
    crb@eclipse.ChrisBowman.com:/usr/obj/usr/src/arm.armv7/sys/= ARTYZ7 arm
FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git = llvmorg-11.0.1-0-g43ff75f2c3fe)
CPU: ARM Cortex-A9 = r3p0 (ECO: 0x00000000)
CPU = Features: 
  Multiprocessing, Thumb2, = Security, VMSAv7, Coherent Walk
Optional = instructions: 
  UMULL, SMULL, = SIMD(ext)
LoUU:2 LoC:2 LoUIS:2 
Cache level 1:
 32KB/32B 4-way data = cache WB Read-Alloc Write-Alloc
 32KB/32B = 4-way instruction cache Read-Alloc
real memory =  =3D 536866816 (511 MB)
avail memory =3D = 515162112 (491 MB)
FreeBSD/SMP: Multiprocessor = System Detected: 2 CPUs
random: unblocking = device.
random: entropy device external = interface
ofwbus0: <Open Firmware Device = Tree>
simplebus0: <Flattened device tree = simple bus> on ofwbus0
simplebus1: <Flattened = device tree simple bus> on ofwbus0
l2cache0: = <PL310 L2 cache controller> mem 0xf8f02000-0xf8f02fff irq 8 on = simplebus1
l2cache0: cannot allocate IRQ, not using = interrupt
l2cache0: Part number: 0x3, release: = 0x8
l2cache0: L2 Cache enabled: 512KB/32B 8 = ways
gic0: <ARM Generic Interrupt Controller> = mem 0xf8f01000-0xf8f01fff,0xf8f00100-0xf8f001ff on simplebus1
gic0: pn 0x39, arch 0x1, rev 0x2, implementer 0x43b irqs = 96
mp_tmr0: <ARM MPCore Timers> mem = 0xf8f00200-0xf8f0021f irq 29 on simplebus1
Timecounter "MPCore" frequency 50000000 Hz quality = 800
mp_tmr1: <ARM MPCore Timers> mem = 0xf8f00600-0xf8f0061f irq 36 on simplebus1
Event = timer "MPCore" frequency 50000000 Hz quality 1000
cpulist0: <Open Firmware CPU Group> on = ofwbus0
cpu0: <Open Firmware CPU> on = cpulist0
cpu1: <Open Firmware CPU> on = cpulist0
uart0: <Cadence UART> mem = 0xe0000000-0xe0000fff irq 9 on simplebus1
uart0: = console (-1,n,8,1)
zy7_qspi0: <Zynq Quad-SPI = Flash Controller> mem 0xe000d000-0xe000dfff irq 13 on = simplebus1
zy7_qspi0: must have ref-clock = property
device_attach: zy7_qspi0 attach returned = 6
cgem0: <Cadence CGEM Gigabit Ethernet = Interface> mem 0xe000b000-0xe000bfff irq 15 on simplebus1
miibus0: <MII bus> on cgem0
rgephy0:= <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 0 on = miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, = 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, = auto
rgephy1: <RTL8169S/8110S/8211 1000BASE-T = media interface> PHY 1 on miibus0
rgephy1: =  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, = 1000baseT-FDX, 1000baseT-FDX-master, auto
cgem0: = Ethernet address: 56:99:3e:50:9a:8e
sdhci_fdt0: = <Zynq-7000 generic fdt SDHCI controller> mem 0xe0100000-0xe0100fff = irq 17 on simplebus1
sdhci_fdt0: 1 slot(s) = allocated
mmc0: <MMC/SD bus> on = sdhci_fdt0
zy7_devcfg0: <Zynq devcfg block> = mem 0xf8007000-0xf80070ff irq 28 on simplebus1
Timecounters tick every 1.000 msec
sdhci_fdt0-slot0: Controller timeout
sdhci_fdt0-slot0: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D = REGISTER DUMP =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
sdhci_fdt0-slot0: Sys addr: 0x00060000 | Version: =  0x00008901
sdhci_fdt0-slot0: Blk size: = 0x00005008 | Blk cnt:  0x00000001
sdhci_fdt0-slot0: Argument: 0x00000000 | Trn mode: = 0x00000013
sdhci_fdt0-slot0: Present: =  0x01ff0202 | Host ctl: 0x00000001
sdhci_fdt0-slot0: Power:    0x0000000f | Blk gap: =  0x00000000
sdhci_fdt0-slot0: Wake-up: =  0x00000000 | Clock:    0x00004007
sdhci_fdt0-slot0: Timeout:  0x00000006 | Int stat: = 0x00000001
sdhci_fdt0-slot0: Int enab: 0x01ff00fb | = Sig enab: 0x01ff00fa
sdhci_fdt0-slot0: AC12 err: = 0x00000000 | Host ctl2:0x00000000
sdhci_fdt0-slot0: = Caps:     0x69ec0080 | Caps2:   =  0x00000000
sdhci_fdt0-slot0: Max curr: = 0x00000001 | ADMA err: 0x00000000
sdhci_fdt0-slot0: = ADMA addr:0x00000000 | Slot int: 0x00000000
sdhci_fdt0-slot0: = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
sdhci_fdt0-slot0: Controller timeout
sdhci_fdt0-slot0: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D = REGISTER DUMP =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
sdhci_fdt0-slot0: Sys addr: 0x00000000 | Version: =  0x00008901
sdhci_fdt0-slot0: Blk size: = 0x00005008 | Blk cnt:  0x00000001
sdhci_fdt0-slot0: Argument: 0xaaaa0000 | Trn mode: = 0x00000013
sdhci_fdt0-slot0: Present: =  0x01ff0000 | Host ctl: 0x00000001
sdhci_fdt0-slot0: Power:    0x0000000f | Blk gap: =  0x00000000
sdhci_fdt0-slot0: Wake-up: =  0x00000000 | Clock:    0x00004007
sdhci_fdt0-slot0: Timeout:  0x00000006 | Int stat: = 0x00000001
sdhci_fdt0-slot0: Int enab: 0x01ff00fb | = Sig enab: 0x01ff00fa
sdhci_fdt0-slot0: AC12 err: = 0x00000000 | Host ctl2:0x00000000
sdhci_fdt0-slot0: = Caps:     0x69ec0080 | Caps2:   =  0x00000000
sdhci_fdt0-slot0: Max curr: = 0x00000001 | ADMA err: 0x00000000
sdhci_fdt0-slot0: = ADMA addr:0x00000000 | Slot int: 0x00000000
sdhci_fdt0-slot0: = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

The controller register = dumps continue for a while and then I get:

mmc0: CMD7 failed, = RESULT: 1
Release APs
Trying = to mount root from ufs:/dev/mmcsd0s2a [rw]...
mountroot: waiting for device /dev/mmcsd0s2a...
Mounting from ufs:/dev/mmcsd0s2a failed with error = 19.
Trying to mount root from ufs:mmcsd0s2a = []...
mountroot: waiting for device = mmcsd0s2a...
Mounting from ufs:mmcsd0s2a failed = with error 19.

Loader variables:
  = vfs.root.mountfrom=3Dufs:/dev/mmcsd0s2a
  = vfs.root.mountfrom.options=3Drw

Manual root filesystem = specification:
  <fstype>:<device> = [options]
      Mount <device> = using filesystem <fstype>
    =   and with the specified (optional) option list.

    eg. = ufs:/dev/da0s1a
        = zfs:zroot/ROOT/default
        = cd9660:/dev/cd0 ro
        =   (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 = /)

  ? =               List valid disk boot = devices
  .           =     Yield 1 second (for background tasks)
  <empty line>    Abort manual = input

mountroot> 

If I replace the kernel with a 13.0 = kernel, it boots just fine.

So I go and do a git bisect between release/13.0.0 and = release/13.1.0

and I get the following:

git bisect start = '--first-parent'
# status: waiting for both good and bad = commits
# good: = [ea31abc261ffc01b6ff5671bffb15cf910a07f4b] 13.0: update to = RELEASE
git = bisect good ea31abc261ffc01b6ff5671bffb15cf910a07f4b
# = status: waiting for bad commit, 1 good commit known
# bad: = [fc952ac2212b121aa6eefc273f5960ec3e0a466d] Update in preparation of = 13.1-RELEASE
git bisect bad = fc952ac2212b121aa6eefc273f5960ec3e0a466d
# skip: = [4c44dbde5491516eba8725dc51d39c1dcc817472] MFC jail: Handle a parent = jail when a child is added to it
git bisect skip = 4c44dbde5491516eba8725dc51d39c1dcc817472
# good: = [476f87219f408343846254743c7189076be80c04] wpi: Fix a lock leak in an = error path in wpi_run()
git bisect good = 476f87219f408343846254743c7189076be80c04
# bad: = [05bf7d68c56830e52dee14dc87c07d6716e8195e] aesni: Fix an out-of-bounds = read in AES_GCM_decrypt()
git bisect bad = 05bf7d68c56830e52dee14dc87c07d6716e8195e
# good: = [014ae00ef6edca2687d618e0bda138086a1e1230] date: Capitalize seconds = string in synopses
git bisect good = 014ae00ef6edca2687d618e0bda138086a1e1230
# bad: = [08d995ca8f6f1008a10e4bf4d924824c040f842a] swapoff_one(): only check = free pages count manually turning swap off
git = bisect bad 08d995ca8f6f1008a10e4bf4d924824c040f842a
# bad: = [81b6dba1a08b031bdf7463c1704d27ae1e0daa0f] ktls: Fix assertion for TLS = 1.0 CBC when using non-zero starting seqno.
git = bisect bad 81b6dba1a08b031bdf7463c1704d27ae1e0daa0f
# bad: = [67efa8b29930f12dae2bf237fa7c2ce1dafbd6b1] net80211: add a = driver-private pointer to struct ieee80211_node
git = bisect bad 67efa8b29930f12dae2bf237fa7c2ce1dafbd6b1
# good: = [109330155000bfec215ee39148254d2a0b628798] module(9): Document that = evhand can be NULL
git bisect good = 109330155000bfec215ee39148254d2a0b628798
# bad: = [4c8e29637456bbbe709425f691f637914658009f] LinuxKPI: add = module_pci_driver() and pci_alloc_irq_vectors()
git = bisect bad 4c8e29637456bbbe709425f691f637914658009f
# bad: = [4a03ae8d17ddf3d3b57ca281000fd98e200b92cc] nfscl: Fix use after free for = forced dismount
git bisect bad = 4a03ae8d17ddf3d3b57ca281000fd98e200b92cc
# bad: = [de957de097857fabb69a59a9ba36276c5e735de5] bhyve: Fix the = WITH_BHYVE_SNAPSHOT build
git bisect bad = de957de097857fabb69a59a9ba36276c5e735de5
# bad: = [5c2e6d9610f1b3f1d7c5d69b925212a7b1fd9391] hwpmc: initialize arm64 = counter/interrupt state
git bisect bad = 5c2e6d9610f1b3f1d7c5d69b925212a7b1fd9391
# bad: = [ce9c3848ff369467749f59fd24f8b9f1241e725c] uma: Fix handling of reserves = in zone_import()
git bisect bad = ce9c3848ff369467749f59fd24f8b9f1241e725c
# good: = [d5ebaa6f8f850bb6f6273f01386832efcb295827] uma: Improve M_USE_RESERVE = handling in keg_fetch_slab()
git bisect good = d5ebaa6f8f850bb6f6273f01386832efcb295827
# first bad commit: = [ce9c3848ff369467749f59fd24f8b9f1241e725c] uma: Fix handling of reserves = in zone_import()

If I do git log = ce9c3848ff369467749f59fd24f8b9f1241e725c it does seem = that d5ebaa6f8f850bb6f6273f01386832efcb295827 is the previous = commit and that it works just fine but ce9c3848ff369467749f59fd24f8b9f1241e725c doesn=E2=80=99t = boot.  It=E2=80=99s literally the same file system and DTB, the = only difference is the kernel installed.

What=E2=80=99s confusing to me is that = looking at ce9c3848ff369467749f59fd24f8b9f1241e725c I can=E2=80=99t see = how that commit would result in my kernel hanging while = booting.

I=E2=80=99m clearly not an expert and I=E2=80=99m not = sure I used git bisect right but I think I did.  Maybe I need to be = on stable/13 = when I bisect?

I don=E2=80=99t know and if anyone could give me some = pointers I=E2=80=99m happy to do some leg work but I=E2=80=99ve about = come to the end of my road trying to figure out what=E2=80=99s blocking = me.

Thanks in advance,
Christopher
= --Apple-Mail=_2483E1C2-56D4-4603-A76A-F869C386CB20--