Kernel panics in tcp_twclose

[Julien Charbon]

 Hi Konstantin, Hi Palle,

On 18/09/15 18:06, Konstantin Belousov wrote:
> On Fri, Sep 18, 2015 at 03:56:25PM +0200, Julien Charbon wrote:
>>  Hi Palle,
>>
>> On 18/09/15 11:12, Palle Girgensohn wrote:
>>> We see daily panics on our production systems (web server, apache
>>> running MPM event, openjdk8. Kernel with VIMAGE. Jails using netgraph
>>> interfaces [not epair]).
>>>
>>> The problem started after the summer. Normal port upgrades seems to
>>> be the only difference. The problem occurs with 10.2-p2 kernel as
>>> well as 10.1-p4 and 10.1-p15.
>>>
>>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203175
>>>
>>> Any ideas?
>>
>>  Thanks for you detailed report.  I am not aware of any tcp_twclose()
>> related issues (without VIMAGE) since FreeBSD 10.0 (does not mean there
>> are none).  Few interesting facts (at least for me):
>>
>>  - Your crash happens when unlocking a inp exclusive lock with INP_WUNLOCK()
>>
>>  - Something is already wrong before calling turnstile_broadcast() as it
>> is called with ts = NULL:
> In the kernel without witness this is a 99%-sure indication of attempt to
> unlock not owned lock.

 Thanks, this is useful.  So far I did not find any path where
tcp_twclose() can call INP_WUNLOCK without having the exclusive lock
held, that makes this issue interesting.

>>  I won't go to far here as I am not expert enough in VIMAGE, but one
>> question anyway:
>>
>>  - Can you correlate this kernel panic to a particular event?  Like for
>> example a VIMAGE/VNET jail destruction.
>>
>>  I will test that on my side on a 10.2 machine.

 I did not find any issues while testing 10.2 + VIMAGE on my side. Thus
Palle what I would suggest:

 - First, test with stable/10 to see if by chance this issue has already
been fixed in stable branch.

 - Second, if issue is still in stable/10, compile 10.2 kernel with
these options:

options        DDB
options        DEADLKRES
options        INVARIANTS
options        INVARIANT_SUPPORT
options        WITNESS
options        WITNESS_SKIPSPIN

 To see where the original fault is coming from.

 Thanks.

--
Julien

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20150921/cb5bc0cf/attachment.bin>