Kernel panics in tcp_twclose

[Julien Charbon]

 Hi Palle,

On 25/09/15 16:14, Palle Girgensohn wrote:
>> 24 sep 2015 kl. 11:39 skrev Palle Girgensohn <girgen at FreeBSD.org>:
>>> 24 sep 2015 kl. 09:57 skrev Julien Charbon <jch at FreeBSD.org>: On
>>> 24/09/15 09:03, Julien Charbon wrote:
>>>> On 24/09/15 08:55, Palle Girgensohn wrote:
>>>>>> 24 sep 2015 kl. 07:51 skrev Palle Girgensohn 
>>>>>> <girgen at pingpong.net>:
>>>>>>> 24 sep 2015 kl. 00:05 skrev Palle Girgensohn 
>>>>>>> <girgen at pingpong.net>:
>>>>>>>> 23 sep 2015 kl. 20:32 skrev Julien Charbon
>>>>>>>> <jch at freebsd.org>: On 23/09/15 20:26, Palle Girgensohn
>>>>>>>> wrote:
>>>>>>> Kernels and userland are updated to 10.2-p3 with the
>>>>>>> patch removing the suspicous KASSERT. dtrace running
>>>>>>> continously redirecting to a log file.
>>>>> Just had a crash. Unfortunately, the kernel was stuck at the
>>>>> db> prompt, and the remote keyboard was unresponsive (HP ILO,
>>>>> not impressed). So I had to reset the power and never got a
>>>>> core dump...
>>>>> 
>>>>> panic: tcp_tw_2msl_stop: inp should not be released here 
>>>>> cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at
>>>>> db_trace_self_wrapper+0x2b/frame 0xfffffe175acd16a0
>>>>> kdb_backtrace() at kdb_backtrace+0x39/frame 
>>>>> 0xfffffe175acd1750 vpanic() at vpanic+0x126/frame
>>>>> 0xfffffe175acd1790 kassert_panic() at
>>>>> kassert_panic+0x139/frame 0xfffffe175acd1800 tcp_twclose() at
>>>>> tcp_twclose+0x2cb/frame 0xfffffe175acd1850 tcp_tw_2msl_scan()
>>>>> at tcp_tw_2msl_scan+0x13b/frame 0xfffffe175acd1890
>>>>> tcp_slowtimo() at tcp_slowtimo+0x68/frame 0xfffffe175acd18c0
>>>>> pfslowtimo() at pfslowtimo+0x54/frame 0xfffffe175acd18f0
>>>>> softclock_call_cc() at softclock_call_cc+0x193/frame
>>>>> 0xfffffe175acd19d0 softclock() at softclock+0x47/frame
>>>>> 0xfffffe175acd19f0 intr_event_execute_handlers() at
>>>>> intr_event_execute_handlers+0x93/frame 0xfffffe 175acd1a30 
>>>>> ithread_loop() at ithread_loop+0xa6/frame 0xfffffe175acd1a70 
>>>>> fork_exit() at fork_exit+0x84/frame 0xfffffe175acd1ab0 
>>>>> fork_trampoline() at fork_trampoline+0xe/frame
>>>>> 0xfffffe175acd1ab0 --- trap 0, rip = 0, rsp =
>>>>> 0xfffffe175acd1b70, rbp = 0 --- KDB: enter: panic [ thread
>>>>> pid 12 tid 100043 ] Stopped at      kdb_enter+0x3e: movq
>>>>> $0,kdb_why db>
>>>> 
>>>> Thanks a log for this backstrace.  This is what at expected,
>>>> when tcp_close() in call in INP_TIMEWAIT case, in_pcbfree() can
>>>> be called one extra time that leads to:
>>>> 
>>>> tcp_tw_2msl_stop: inp should not be released here
>>>> 
>>>> Let me try to come with a tentative fix for this case.
>>> 
>>> See joined my tentative patch for these case.  It is only a
>>> first tentative patch as I am still waiting on -net feedbacks on
>>> what should be the rule here.
>>> 
>>> By the way:
>>> 
>>> - I see nothing specific to VIMAGE here
>>> 
>>> - Anyone aware of tcp_close() (or tcp_drop()) calls
>>> modified/introduced recently in 10.2 that could explained why
>>> this issue only appears only now?
>> 
>> Running a machine with the patch now (it just crashed and rebooted
>> with the new kernel).
>> 
>> Hoping it will have a "soothing" effect... ;-)
>> 
>> dtrace running as previously. No output yet, though.
> 
> First of, loud cheers and a big *thank you* to Julien for helping us
> get our systems to stop crashing. This really means a lot to us!
> Thank you!

 Glab to see your system more stable now.  You are welcome, thanks to
you for reporting this issue with accuracy.  We got lucky than it took
/only/ three different kernel panics to get a good overview.  This part
of the code being quite tricky as you have three entangled layers that
tries to clean up theirs things the right way: socket, inp and tcptw.

> Dtrace still shows nothing.

 I will try to provide you more generic Dtrace script, it seem the
current one is too specific.

--
Julien

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20150928/896c43a1/attachment.bin>