Kernel panics in tcp_twclose using jails + VIMAGE

[Palle Girgensohn]

> 18 sep 2015 kl. 15:56 skrev Julien Charbon <jch at freebsd.org>:
> 
> Hi Palle,
> 
> On 18/09/15 11:12, Palle Girgensohn wrote:
>> We see daily panics on our production systems (web server, apache
>> running MPM event, openjdk8. Kernel with VIMAGE. Jails using netgraph
>> interfaces [not epair]).
>> 
>> The problem started after the summer. Normal port upgrades seems to
>> be the only difference. The problem occurs with 10.2-p2 kernel as
>> well as 10.1-p4 and 10.1-p15.
>> 
>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203175
>> 
>> Any ideas?
> 
> Thanks for you detailed report.  I am not aware of any tcp_twclose()
> related issues (without VIMAGE) since FreeBSD 10.0 (does not mean there
> are none).  Few interesting facts (at least for me):
> 
> - Your crash happens when unlocking a inp exclusive lock with INP_WUNLOCK()
> 
> - Something is already wrong before calling turnstile_broadcast() as it
> is called with ts = NULL:
> 
> turnstile_broadcast (ts=0x0, queue=1) at
> /usr/src/sys/kern/subr_turnstile.c:838
> __rw_wunlock_hard () at /usr/src/sys/kern/kern_rwlock.c:988
> tcp_twclose () at /usr/src/sys/netinet/tcp_timewait.c:540
> tcp_tw_2msl_scan () at /usr/src/sys/netinet/tcp_timewait.c:748
> tcp_slowtimo () at /usr/src/sys/netinet/tcp_timer.c:198
> 
> I won't go to far here as I am not expert enough in VIMAGE, but one
> question anyway:
> 
> - Can you correlate this kernel panic to a particular event?  Like for
> example a VIMAGE/VNET jail destruction.
> 
> I will test that on my side on a 10.2 machine.
> 
> --
> Julien
> 

Hi, thank for your reply. It is not related to jail destruction. It *might* be related to apache httpd (MPM event) forking during normal operation, but we have not found any specific event that triggers the panic. The system crash during normal operation, no excessive load (but load is at least partly responsible, a more loaded server is more likely to crash).

Note that we use netgraph, not epair, although I don't believe it makes a difference.

Palle

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20150918/164b71c2/attachment.bin>