Questions about locking; turnstiles and sleeping threads

[Adrian Chadd]

Hm, the more I dig into this, the more I realise it's not a 1:45am
question to ask.

Specifically, callout_stop_safe() takes 'safe', which says "are we
waiting around for this callout to finish if it started". Ie,
callout_drain() is callout_stop_safe(c, 1) ; callout_stop() is
callout_stop_safe(c, 0).

If safe is 1, then it'll potentially put the current thread to sleep
in order to wait for it to synchronise with the callout that's
running. It's sleeping with cc_lock which is the per-callwheel lock
and it's doing that with whatever other locks are held. That's the
situation which is tripping things up.

The manpage says that no locks should be held that the callout may
block on, which isn't the case here at all - I'm trying to grab a lock
in another thread that the caller _into_ the callout subsystem holds.
The manpage doesn't mention anything about this. Sniffle.



-adrian

On 13 November 2014 01:42, Alfred Perlstein <alfred at freebsd.org> wrote:
> OK that makes more sense.
>
> I've cc'd Hans for the usb issue.
>
>
> On 11/13/14, 1:38 AM, Adrian Chadd wrote:
>>
>> It looks like the initial firings are because the check I put in
>> didn't check to see if it's MPSAFE.
>>
>> eg:
>>
>> ip6_input -> tcp6_input -> tcp_input -> tcp_do_segment ->
>> tcp_timer_active -> callout_stop_safe; called with tcpinp held.
>> closefp() -> closef() -> fdrop -> soclose() -> sofree() ->
>> tcp_usr_detach() -> tcp_discardcb() -> callout_stop_safe() with the
>> tcpinp and tcp locks held.
>> ioctl -> sys_ioctl-> devfs_ioctl_f -> acpi_ackSleepState ->
>> callout_stop_safe; with ACPI global lock held;
>> suspend path -> hdaa_suspend -> callout_stop_safe() with HDA driver mutex
>> held
>>
>> So we can't just put the simple witness check from _sleep() in
>> _callout_stop_safe(), it looks like it's mis-firing on MPSAFE callouts
>> (which the tcp timers all are) and that won't go via the sleepq.
>> It looks like the acpi callout is also mpsafe, as well as the HDA jack
>> callout.
>>
>> However, I did pick up this:
>>
>> detach path -> usbd_transfer_drain() -> usbd_transfer_stop() ->
>> ehci_device_intr_close() -> usbd_transfer_done() ->
>> callout_stop_safe() with USB HUB mutex held
>>
>> The usbd_transfer_done() callout is initialised with a mutex, but the
>> witness code should've detected it wasn't callout->c_lock and thus
>> warned.
>>
>>
>>
>> -adrian
>>
>> On 13 November 2014 01:13, Alfred Perlstein <alfred at freebsd.org> wrote:
>>>
>>> On 11/13/14, 1:09 AM, Adrian Chadd wrote:
>>>>
>>>> On 13 November 2014 00:59, Alfred Perlstein <alfred at freebsd.org> wrote:
>>>>>
>>>>> On 11/12/14, 11:25 PM, Adrian Chadd wrote:
>>>>>>
>>>>>> On 12 November 2014 19:48, Adrian Chadd <adrian at freebsd.org> wrote:
>>>>>>>
>>>>>>> kan pointed out that we should likely do a WITNESS_WARN() in the
>>>>>>> relevant spots in the callout code so we catch these before it
>>>>>>> happens.
>>>>>>>
>>>>>>> I'll see what we can add to -HEAD to be pro-active about it.
>>>>>>
>>>>>> Amusingly, I tried adding it and it made my laptop turn to soup very
>>>>>> quickly - among other things, the TCP timer callouts are all setup
>>>>>> with non sleep locks held.
>>>>>>
>>>>> Howso?  You only have to worry about callout_drain(), most other
>>>>> callout
>>>>> functions should be safe-ish....
>>>>
>>>> yeah, except for all the places where they drain the timer once
>>>> something happens so it doesn't fire.
>>>>
>>>> :)
>>>
>>>
>>> What is the backtrace...?
>>>
>>>
>> _______________________________________________
>> freebsd-arch at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-arch
>> To unsubscribe, send any mail to "freebsd-arch-unsubscribe at freebsd.org"
>>
>