Regarding Dtrace on Arm

[Robert Watson]

On Wed, 11 May 2016, Mark Johnston wrote:

>> Hello All,
>>
>> I was using Dtrace using the FBT provider on an arm platform with witness 
>> enabled. When FBT is used with the kernel module, it generates a kernel 
>> panic or the system becomes unresponsive. Is this problem know or seen 
>> before. I am copying the kernel backtrace below for reference. The issue 
>> seems to be with a blockable sleep lock(kld_sx) acquired which is 
>> conflicting with the td->td_critnest positive value.
>
> Unfortunately, I think this is expected. The ARM port of DTrace will call 
> into the kernel linker from probe context to perform stack unwinding. This 
> is a bug since the kernel linker cannot be entered in arbitrary contexts - 
> the thread might already hold the linker lock, or might hold a critical 
> section as in your example.
>
> This is a known issue, but I'm not sure if anyone is working on fixing it.

Andrew Turner and I have discussed this on multiple occasions -- the issue is 
that to unwind stacks on ARMv7/ARMv8, you need access to unwind tables that 
are accessed via the linker.  In our Cambridge L41 course, our workaround was 
to compile everything into the kernel rather than use modules -- and, 
unfortunately, we've not had a chance to get back to sorting this out.  The 
solution is not that difficult, it just requires a bit of work: we need a 
reference-count/lockless synchronisation model that allows safe lookup of 
unwind state from preemptive contexts (not just DTrace, but also HWPMC), to 
ensure that unwind tables are kept stable while they could be in use from 
those contexts.

Robert