code in GEOM thread could not use vnode API (Was: alq_open_flags() panics in _mtx_lock_flags())

[Lawrence Stewart]

On 10/04/11 03:01, Kostik Belousov wrote:
> On Mon, Oct 03, 2011 at 04:18:59PM +0400, Lev Serebryakov wrote:
>> Hello, Lawrence.
>> You wrote 3 октября 2011 г., 13:57:04:
>>
>>> I know nothing about VFS, but wonder if it's something to do with the
>>> credentials you pass in?
>>    They looks Ok. Not NULL, at least :) I'm passing
>>   "curtrhead->td_ucred".
>>
>>> Lev, are you able to share your code with us?
>>    Yes, of course. Minimal code, which trigger this error, attached.
>>   Build, load module and PANIC!
>>
>>    This is geom_zero module, which try to create ALQ with name
>>    "/var/log/zero.alq.log" on it load (not creation! So, you don't need
>>    even create such GEOM!). Please note, that "init" callback of GEOM
>>    class is called in g_event GEOM thread.
>>
>>     Code is for 9.0/10.0
>>
>>    My code is rewritten with usage of working thread already, and it
>> works without any panic. But worker thread looks like overkill for
>> simple case "write 24 bytes record to ALQ for each BIO" to me.
>>
>>    So I could prove, that this code PANIC because alq_open_flags() is
>>   called from one of three GEOM threads.
> Look at the body of e.g. g_io_schedule_down(), the
> 		THREAD_NO_SLEEPING();
> call right before the call to geom start method.

Thanks Kostik, that's the context I was missing.

> Basically, you cannot use any kernel subsystem in the context of
> the up/down threads that could sleep. This includes VFS, VM and
> everything that is layered on top of it.
>
> The decision is deliberate. Up and down shall only schedule the processing,
> or perform the fast processing.
>
> BTW, it must be obvious that trying to perform any VFS operation from geom
> up or down causes deadlock.

So Lev, I believe what you can do is alq_open_flags() in a separate 
context (module init routine or a sysctl handler are good places) and 
then you should be able to use it in the sleep sensitive context, as ALQ 
is designed to work in such places if you pass the ALQ_NOWAIT flag to 
the other API calls.

If you want an example, have a look at sys/netinet/siftr.c. The packet 
fast path is non-sleepable, so I create the ALQ when the enable sysctl 
is called (siftr_sysctl_enabled_handler -> siftr_manage_ops -> 
alq_open), although I defer the pkt processing work to a separate thread 
which does the alq_writes. You should be able to do the writes in the 
geom fast path without a separate thread if you prefer.

Cheers,
Lawrence