From nobody Mon Mar 07 13:46:09 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 8B93519FCB88; Mon, 7 Mar 2022 13:46:13 +0000 (UTC) (envelope-from SRS0=34E+=TS=klop.ws=ronald-lists@realworks.nl) Received: from smtp-relay-int.realworks.nl (smtp-relay-int.realworks.nl [194.109.157.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4KC09J2yY3z4vRp; Mon, 7 Mar 2022 13:46:12 +0000 (UTC) (envelope-from SRS0=34E+=TS=klop.ws=ronald-lists@realworks.nl) Date: Mon, 7 Mar 2022 14:46:09 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=klop.ws; s=rw2; t=1646660769; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=loPRI9HeODZSw7f9ycpQAjFrmToqcGoe6XqdEdwAgNg=; b=br12TzQSUvGpgQR9Z56F/OEV88yVvYb0RON3JDglBicovXSf9SBAA7RCVbWEY/GeCAUKRC MBpIMHtDqYhU6Jvb5GZWlqm6cM6U+vC+rTv5sgf1gWWHyxKTJ+jMskmAe6vwzlpM7NZiNH I4tEFDLeDOsxvi5O2sAaNH3ELc4QW2oRjbVTmrYN7vr+XJxrATmCWFTDCyTMJgHFmRCTa7 aTP6mP36eiydAFa297epEo9GHPNTK2xKBubYuY9uav3JexxW8yAu5LDS17zlfl9Iz5hMXw 9lMspIzK3ZmdeQRLwfmJNCxmjadskrQnOYJO259Hu2PP9ArhhPFFxgi9qa/JBQ== From: Ronald Klop To: Mark Johnston Cc: bob prohaska , Mark Millard , freebsd-arm@freebsd.org, freebsd-current Message-ID: <132978150.92.1646660769467@mailrelay> In-Reply-To: <1800459695.1.1646649539521@mailrelay> References: <1800459695.1.1646649539521@mailrelay> Subject: Re: panic: data abort in critical section or under mutex (was: Re: panic: Unknown kernel exception 0 esr_el1 2000000 (on 14-CURRENT/aarch64 Feb 28)) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_91_1813909039.1646660769454" X-Mailer: Realworks (599.211.aa1e011) Importance: Normal X-Priority: 3 (Normal) X-Rspamd-Queue-Id: 4KC09J2yY3z4vRp X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=klop.ws header.s=rw2 header.b=br12TzQS; dmarc=pass (policy=quarantine) header.from=klop.ws; spf=pass (mx1.freebsd.org: domain of "SRS0=34E@realworks.nl" designates 194.109.157.24 as permitted sender) smtp.mailfrom="SRS0=34E@realworks.nl" X-Spamd-Result: default: False [-3.18 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[klop.ws:s=rw2]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:194.109.157.0/24]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[klop.ws:+]; DMARC_POLICY_ALLOW(-0.50)[klop.ws,quarantine]; HAS_X_PRIO_THREE(0.00)[3]; NEURAL_HAM_SHORT(-0.98)[-0.978]; MLMMJ_DEST(0.00)[freebsd-arm,freebsd-current]; FORGED_SENDER(0.30)[ronald-lists@klop.ws,SRS0=34E@realworks.nl]; RCVD_COUNT_ZERO(0.00)[0]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:3265, ipnet:194.109.0.0/16, country:NL]; TAGGED_FROM(0.00)[=TS=klop.ws=ronald-lists]; FROM_NEQ_ENVFROM(0.00)[ronald-lists@klop.ws,SRS0=34E@realworks.nl]; FREEMAIL_CC(0.00)[www.zefox.net,yahoo.com,freebsd.org] X-ThisMailContainsUnwantedMimeParts: N ------=_Part_91_1813909039.1646660769454 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Dear Mark Johnston, I did some binary search in the kernels and came to the conclusion that https://cgit.freebsd.org/src/commit/?id=1517b8d5a7f58897200497811de1b18809c07d3e still works and https://cgit.freebsd.org/src/commit/?id=407c34e735b5d17e2be574808a09e6d729b0a45a panics. I suspect your commit in https://cgit.freebsd.org/src/commit/?id=c84bb8cd771ce4bed58152e47a32dda470bef23a. Last panic: panic: vm_fault failed: ffff00000046e708 error 1 cpuid = 1 time = 1646660058 KDB: stack backtrace: db_trace_self() at db_trace_self db_trace_self_wrapper() at db_trace_self_wrapper+0x30 vpanic() at vpanic+0x174 panic() at panic+0x44 data_abort() at data_abort+0x2e8 handle_el1h_sync() at handle_el1h_sync+0x10 --- exception, esr 0x96000004 _rm_rlock_debug() at _rm_rlock_debug+0x8c osd_get() at osd_get+0x5c zio_execute() at zio_execute+0xf8 taskqueue_run_locked() at taskqueue_run_locked+0x178 taskqueue_thread_loop() at taskqueue_thread_loop+0xc8 fork_exit() at fork_exit+0x74 fork_trampoline() at fork_trampoline+0x14 KDB: enter: panic [ thread pid 0 tid 100129 ] Stopped at kdb_enter+0x44: undefined f902011f db> A more recent kernel (912df91) still panics. See below. Do you have time to look into this? What can I provide in information to help? Regards, Ronald. Van: Ronald Klop Datum: maandag, 7 maart 2022 11:38 Aan: Mark Millard CC: bob prohaska , freebsd-current , freebsd-arm@freebsd.org Onderwerp: Re: panic: data abort in critical section or under mutex (was: Re: panic: Unknown kernel exception 0 esr_el1 2000000 (on 14-CURRENT/aarch64 Feb 28)) > > Yes, I spoke to soon too. Often it crashes as soon as I start a parallel poudriere build. But this time it went very far. As soon as nightly backups kicked in it was game over again. > I had read the mail of Bob on the arm@ ML. But I wanted to let the conclusion that it is about the same problem to the developers. (Have seen enough of wrong guessing of causes in my work. ) > > I will need to go further into the binary search of working kernels. > > This was: FreeBSD 14.0-CURRENT #0 912df91: Wed Mar 2 00:36:35 UTC 2022 > Fatal data abort: > x0: ffff000000f1efd8 x0: ffff000000f1efd8 (mac_policy_rm + 0) (mac_policy_rm + 0) > > x1: 2 x1: 2 > > x2: ffff00000087dcf2 x2: ffff00000087dcf2 (cam_status_table + 2f28a) > (cam_status_table + 2f28a) x3: ffff00000087dcf2 > x3: ffff00000087dcf2 (cam_status_table + 2f28a) (cam_status_table + 2f28a) > > x4: 102 x4: 102 > > x5: 7 x5: 1 > > x6: 0 x6: ff > > x7: 0 x7: ffffa00011fc2800 > x8: 1 > > x8: 1 x9: ffff000000f37c10 > x9: ffff0000419d9090 (pcpu0 + 90) (g_ctx + 40278fe4) > > x10: ffffa0017be2a600 x10: ffffa000010fa600 > x11: 394aed08d0003a48 > > x12: 350001a8b946a108 x11: 0 > > x12: ffff000000f37c10 x13: badecce4 (pcpu0 + 90) > > x13: ffffa0001fbde6b0 x14: 0 > > x14: 4965ae49 x15: 1 > > x15: 1000193 x16: ffff0000016a4238 > x16: ffff000100482d38 (__stop_set_modmetadata_set + d00) (__stop_set_modmetadata_set + 448) > > x17: ffff00000044a998 x17: ffff00000058ff30 (free + 0) (if_inc_counter + 0) > > x18: ffff0000b49a23c0 x18: ffff000103f11b80 (g_ctx + b3242314) > (next_index + 3a228c0) x19: 102 > > > x19: 102 x20: ffff0000b49a2428 > x20: ffff000103f11be8 (g_ctx + b324237c) (next_index + 3a22928) > > x21: ffff00000087dcf2 x21: ffff00000087dcf2 (cam_status_table + 2f28a) (cam_status_table + 2f28a) > > x22: ffff000000f1efd8 x22: ffff000000f1efd8 (mac_policy_rm + 0) (mac_policy_rm + 0) > > x23: ffff00000086f107 x23: 0 (cam_status_table + 2069f) > > x24: ffffa0001fbde6c8 x24: ffffa0008cba0d00 > x25: 0 > > x25: ffff00000088aa11 x26: 4 (do_execve.fexecv_proc_title + 76b7) > > x27: 0 x26: ffffa0017be2a600 > x28: ffff00010209fcf0 > x27: ffffa00025626a80 (next_index + 1bb0a30) > > x28: ffff000103f11ce0 x29: ffff0000b49a23e0 (next_index + 3a22a20) (g_ctx + b3242334) > > x29: ffff000103f11ba0 sp: ffff0000b49a23c0 > (next_index + 3a228e0) lr: ffff00000046ef98 > sp: ffff000103f11b80 > (_rm_runlock_debug + 60) lr: ffff00000046ef98 > elr: ffff00000046dc0c (_rm_runlock_debug + 60) (_rm_assert + a4) > > elr: ffff00000046dc0cspsr: 45 > (_rm_assert + a4) far: 10 > > esr: 96000004 > spsr: 45 > > panic: data abort in critical section or under mutex > cpuid = 1 > time = 1646609483 > KDB: stack backtrace: > db_trace_self() at db_trace_self > db_trace_self_wrapper() at db_trace_self_wrapper+0x30 > vpanic() at vpanic+0x174 > panic() at panic+0x44 > data_abort() at data_abort+0x2d4 > handle_el1h_sync() at handle_el1h_sync+0x10 > --- exception, esr 0x96000004 > _rm_assert() at _rm_assert+0xa4 > _rm_runlock_debug() at _rm_runlock_debug+0x5c > mac_inpcb_check_deliver() at mac_inpcb_check_deliver+0x74 > tcp_input_with_port() at tcp_input_with_port+0xab4 > tcp_input() at tcp_input+0xc > ip_input() at ip_input+0x2e8 > netisr_dispatch_src() at netisr_dispatch_src+0xe4 > ether_demux() at ether_demux+0x178 > ether_nh_input() at ether_nh_input+0x3e8 > netisr_dispatch_src() at netisr_dispatch_src+0xe4 > ether_input() at ether_input+0x80 > if_input() at if_input+0xc > gen_intr() at gen_intr+0x444 > ithread_loop() at ithread_loop+0x2a0 > fork_exit() at fork_exit+0x74 > fork_trampoline() at fork_trampoline+0x14 > KDB: enter: panic > [ thread pid 12 tid 100063 ] > Stopped at kdb_enter+0x44: undefined f902011f > db> > > NB: db> reboot/reset/halt does not work on my RPI4. Luckily I have a wifi connected power switch on it. > > Regards, > Ronald. > > > Van: Mark Millard > Datum: maandag, 7 maart 2022 02:01 > Aan: Ronald Klop > CC: freebsd-current , bob prohaska > Onderwerp: Re: panic: data abort in critical section or under mutex (was: Re: panic: Unknown kernel exception 0 esr_el1 2000000 (on 14-CURRENT/aarch64 Feb 28)) >> >> From: Ronald Klop wrote on >> Date: Sun, 6 Mar 2022 23:22:42 +0100 (CET) : >> >> > Did some binary search with kernels from artifact.ci.freebsd.org. >> > >> > I suspect "rmlock: Micro-optimize read locking" as cause. >> > >> > https://cgit.freebsd.org/src/commit/?id=c84bb8cd771ce4bed58152e47a32dda470bef23a >> > >> > >> > And "rmlock: Add required compiler barriers to _rm_runlock()" as solution. >> > >> > https://cgit.freebsd.org/src/commit/?id=89ae8eb74e87ac19aa2d7abe4ba16bcccd32bb9f >> > >> > >> > So I probably just had a bad day. >> >> Well, there is a report of a buildkernel crash after that pair: >> >> https://lists.freebsd.org/archives/freebsd-arm/2022-March/001078.html >> >> that references additional information at: >> >> http://www.zefox.net/~fbsd/rpi3/crashes/20220304/readme >> >> and reported: >> >> QUOTE >> The console connection dropped before the crash (unrelated) I didn't >> get the preamble, all I have is the backtrace and buildkernel log. >> Here's the backtrace: >> db> bt >> Tracing pid 14795 tid 100098 td 0xffffa00017815600 >> db_trace_self() at db_trace_self >> db_stack_trace() at db_stack_trace+0x11c >> db_command() at db_command+0x368 >> db_command_loop() at db_command_loop+0x54 >> db_trap() at db_trap+0xf8 >> kdb_trap() at kdb_trap+0x1cc >> handle_el1h_sync() at handle_el1h_sync+0x10 >> --- exception, esr 0xf2000000 >> kdb_enter() at kdb_enter+0x44 >> vpanic() at vpanic+0x1b0 >> panic() at panic+0x44 >> data_abort() at data_abort+0x2e8 >> handle_el1h_sync() at handle_el1h_sync+0x10 >> --- exception, esr 0x96000004 >> _rm_rlock_debug() at _rm_rlock_debug+0x8c >> sysctl_root_handler_locked() at sysctl_root_handler_locked+0x140 >> sysctl_root() at sysctl_root+0x1ac >> userland_sysctl() at userland_sysctl+0x140 >> sys___sysctl() at sys___sysctl+0x68 >> do_el0_sync() at do_el0_sync+0x520 >> handle_el0_sync() at handle_el0_sync+0x40 >> --- exception, esr 0x56000000 >> END QUOTE >> >> The above material does reference _rm_rlock_debug . Might be >> related? >> >> The readme reports: >> >> main-n253603-0b25cbc79d3: Thu Mar 3 22:48:31 PST 2022 >> >> for the system doing the buildkernel. This is after >> 89ae8eb74e8 . >> >> (It also mentions another panic earlier in the week, >> apparently not reported to the lists at the time.) >> >> === >> Mark Millard >> marklmi at yahoo.com >> >> >> >> > ------=_Part_91_1813909039.1646660769454 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit Dear Mark Johnston,

I did some binary search in the kernels and came to the conclusion that https://cgit.freebsd.org/src/commit/?id=1517b8d5a7f58897200497811de1b18809c07d3e still works and https://cgit.freebsd.org/src/commit/?id=407c34e735b5d17e2be574808a09e6d729b0a45a panics.

I suspect your commit in https://cgit.freebsd.org/src/commit/?id=c84bb8cd771ce4bed58152e47a32dda470bef23a.

Last panic:

panic: vm_fault failed: ffff00000046e708 error 1
cpuid = 1
time = 1646660058
KDB: stack backtrace:
db_trace_self() at db_trace_self
db_trace_self_wrapper() at db_trace_self_wrapper+0x30
vpanic() at vpanic+0x174
panic() at panic+0x44
data_abort() at data_abort+0x2e8
handle_el1h_sync() at handle_el1h_sync+0x10
--- exception, esr 0x96000004
_rm_rlock_debug() at _rm_rlock_debug+0x8c
osd_get() at osd_get+0x5c
zio_execute() at zio_execute+0xf8
taskqueue_run_locked() at taskqueue_run_locked+0x178
taskqueue_thread_loop() at taskqueue_thread_loop+0xc8
fork_exit() at fork_exit+0x74
fork_trampoline() at fork_trampoline+0x14
KDB: enter: panic
[ thread pid 0 tid 100129 ]
Stopped at      kdb_enter+0x44: undefined       f902011f
db>

A more recent kernel (912df91) still panics. See below.

Do you have time to look into this? What can I provide in information to help?

Regards,
Ronald.

 

Van: Ronald Klop <ronald-lists@klop.ws>
Datum: maandag, 7 maart 2022 11:38
Aan: Mark Millard <marklmi@yahoo.com>
CC: bob prohaska <fbsd@www.zefox.net>, freebsd-current <freebsd-current@freebsd.org>, freebsd-arm@freebsd.org
Onderwerp: Re: panic: data abort in critical section or under mutex (was: Re: panic: Unknown kernel exception 0 esr_el1 2000000 (on 14-CURRENT/aarch64 Feb 28))

Yes, I spoke to soon too. Often it crashes as soon as I start a parallel poudriere build. But this time it went very far. As soon as nightly backups kicked in it was game over again.
I had read the mail of Bob on the arm@ ML. But I wanted to let the conclusion that it is about the same problem to the developers. (Have seen enough of wrong guessing of causes in my work. )

I will need to go further into the binary search of working kernels.

This was: FreeBSD 14.0-CURRENT #0 912df91: Wed Mar  2 00:36:35 UTC 2022 
Fatal data abort:                                                                                                                   
  x0: ffff000000f1efd8  x0: ffff000000f1efd8 (mac_policy_rm + 0) (mac_policy_rm + 0)                                                
                                                                                                                                    
  x1:                2  x1:                2                                                                                        
                                                                                                                                    
  x2: ffff00000087dcf2  x2: ffff00000087dcf2 (cam_status_table + 2f28a)                                                             
 (cam_status_table + 2f28a)  x3: ffff00000087dcf2                                                                                   
  x3: ffff00000087dcf2 (cam_status_table + 2f28a) (cam_status_table + 2f28a)                                                        
                                                                                                                                    
  x4:              102  x4:              102                                                                                        
                                                                                                                                    
  x5:                7  x5:                1                                                                                        
                                                                                                                                    
  x6:                0  x6:               ff                                                                                        
                                                                                                                                    
  x7:                0  x7: ffffa00011fc2800                                                                                        
  x8:                1                                                                                                              
                                                                                                                                    
  x8:                1  x9: ffff000000f37c10                                                                                        
  x9: ffff0000419d9090 (pcpu0 + 90) (g_ctx + 40278fe4)                                                                              
                                                                                                                                    
 x10: ffffa0017be2a600 x10: ffffa000010fa600                                                                                        
 x11: 394aed08d0003a48                                                                                                              
                                                                                                                                    
 x12: 350001a8b946a108 x11:                0                                                                                        
                                                                                                                                    
 x12: ffff000000f37c10 x13:         badecce4 (pcpu0 + 90)                                                                           
                                                                                                                                    
 x13: ffffa0001fbde6b0 x14:                0                                                                                        
                                                                                                                                    
 x14:         4965ae49 x15:                1                                                                                        
                                                                                                                                    
 x15:          1000193 x16: ffff0000016a4238                                                                                        
 x16: ffff000100482d38 (__stop_set_modmetadata_set + d00) (__stop_set_modmetadata_set + 448)                                        
                                                                                                                                    
 x17: ffff00000044a998 x17: ffff00000058ff30 (free + 0) (if_inc_counter + 0)                                                        
                                                                                                                                    
 x18: ffff0000b49a23c0 x18: ffff000103f11b80 (g_ctx + b3242314)                                                                     
 (next_index + 3a228c0) x19:              102                                                                                       

                                                                                                                                   
 x19:              102 x20: ffff0000b49a2428                                                                                        
 x20: ffff000103f11be8 (g_ctx + b324237c) (next_index + 3a22928)                                                                    

 x21: ffff00000087dcf2 x21: ffff00000087dcf2 (cam_status_table + 2f28a) (cam_status_table + 2f28a)

 x22: ffff000000f1efd8 x22: ffff000000f1efd8 (mac_policy_rm + 0) (mac_policy_rm + 0)

 x23: ffff00000086f107 x23:                0 (cam_status_table + 2069f)

 x24: ffffa0001fbde6c8 x24: ffffa0008cba0d00
 x25:                0

 x25: ffff00000088aa11 x26:                4 (do_execve.fexecv_proc_title + 76b7)

 x27:                0 x26: ffffa0017be2a600
 x28: ffff00010209fcf0
 x27: ffffa00025626a80 (next_index + 1bb0a30)

 x28: ffff000103f11ce0 x29: ffff0000b49a23e0 (next_index + 3a22a20) (g_ctx + b3242334)

 x29: ffff000103f11ba0  sp: ffff0000b49a23c0
 (next_index + 3a228e0)  lr: ffff00000046ef98
  sp: ffff000103f11b80
 (_rm_runlock_debug + 60)  lr: ffff00000046ef98
 elr: ffff00000046dc0c (_rm_runlock_debug + 60) (_rm_assert + a4)

 elr: ffff00000046dc0cspsr:               45
 (_rm_assert + a4) far:               10

 esr:         96000004
spsr:               45

panic: data abort in critical section or under mutex
cpuid = 1
time = 1646609483
KDB: stack backtrace:
db_trace_self() at db_trace_self
db_trace_self_wrapper() at db_trace_self_wrapper+0x30
vpanic() at vpanic+0x174
panic() at panic+0x44
data_abort() at data_abort+0x2d4
handle_el1h_sync() at handle_el1h_sync+0x10
--- exception, esr 0x96000004
_rm_assert() at _rm_assert+0xa4
_rm_runlock_debug() at _rm_runlock_debug+0x5c
mac_inpcb_check_deliver() at mac_inpcb_check_deliver+0x74
tcp_input_with_port() at tcp_input_with_port+0xab4
tcp_input() at tcp_input+0xc
ip_input() at ip_input+0x2e8
netisr_dispatch_src() at netisr_dispatch_src+0xe4
ether_demux() at ether_demux+0x178
ether_nh_input() at ether_nh_input+0x3e8
netisr_dispatch_src() at netisr_dispatch_src+0xe4
ether_input() at ether_input+0x80
if_input() at if_input+0xc
gen_intr() at gen_intr+0x444
ithread_loop() at ithread_loop+0x2a0
fork_exit() at fork_exit+0x74
fork_trampoline() at fork_trampoline+0x14
KDB: enter: panic
[ thread pid 12 tid 100063 ]
Stopped at      kdb_enter+0x44: undefined       f902011f
db>

NB: db> reboot/reset/halt does not work on my RPI4. Luckily I have a wifi connected power switch on it.

Regards,
Ronald.

 

Van: Mark Millard <marklmi@yahoo.com>
Datum: maandag, 7 maart 2022 02:01
Aan: Ronald Klop <ronald-lists@klop.ws>
CC: freebsd-current <freebsd-current@freebsd.org>, bob prohaska <fbsd@www.zefox.net>
Onderwerp: Re: panic: data abort in critical section or under mutex (was: Re: panic: Unknown kernel exception 0 esr_el1 2000000 (on 14-CURRENT/aarch64 Feb 28))

From: Ronald Klop <ronald-lists_at_klop.ws> wrote on
Date: Sun, 6 Mar 2022 23:22:42 +0100 (CET) :

> Did some binary search with kernels from artifact.ci.freebsd.org.
>
> I suspect "rmlock: Micro-optimize read locking" as cause.
>
> https://cgit.freebsd.org/src/commit/?id=c84bb8cd771ce4bed58152e47a32dda470bef23a
>
>
> And "rmlock: Add required compiler barriers to _rm_runlock()" as solution.
>
> https://cgit.freebsd.org/src/commit/?id=89ae8eb74e87ac19aa2d7abe4ba16bcccd32bb9f
>
>
> So I probably just had a bad day.

Well, there is a report of a buildkernel crash after that pair:

https://lists.freebsd.org/archives/freebsd-arm/2022-March/001078.html

that references additional information at:

http://www.zefox.net/~fbsd/rpi3/crashes/20220304/readme

and reported:

QUOTE
The console connection dropped before the crash (unrelated) I didn't
get the preamble, all  I have is the backtrace and buildkernel log.
Here's the backtrace:
db> bt
Tracing pid 14795 tid 100098 td 0xffffa00017815600
db_trace_self() at db_trace_self
db_stack_trace() at db_stack_trace+0x11c
db_command() at db_command+0x368
db_command_loop() at db_command_loop+0x54
db_trap() at db_trap+0xf8
kdb_trap() at kdb_trap+0x1cc
handle_el1h_sync() at handle_el1h_sync+0x10
--- exception, esr 0xf2000000
kdb_enter() at kdb_enter+0x44
vpanic() at vpanic+0x1b0
panic() at panic+0x44
data_abort() at data_abort+0x2e8
handle_el1h_sync() at handle_el1h_sync+0x10
--- exception, esr 0x96000004
_rm_rlock_debug() at _rm_rlock_debug+0x8c
sysctl_root_handler_locked() at sysctl_root_handler_locked+0x140
sysctl_root() at sysctl_root+0x1ac
userland_sysctl() at userland_sysctl+0x140
sys___sysctl() at sys___sysctl+0x68
do_el0_sync() at do_el0_sync+0x520
handle_el0_sync() at handle_el0_sync+0x40
--- exception, esr 0x56000000
END QUOTE

The above material does reference _rm_rlock_debug . Might be
related?

The readme reports:

main-n253603-0b25cbc79d3: Thu Mar  3 22:48:31 PST 2022

for the system doing the buildkernel. This is after
89ae8eb74e8 .

(It also mentions another panic earlier in the week,
apparently not reported to the lists at the time.)

===
Mark Millard
marklmi at yahoo.com
 
------=_Part_91_1813909039.1646660769454--