threads/136345: Recursive read rwlocks in thread A cause deadlock with write lock in thread B

[Nick Esborn]

On Jul 16, 2009, at 5:48 AM, Attilio Rao wrote:

> 2009/7/16 Attilio Rao <attilio at freebsd.org>:
>> 2009/7/15 Nick Esborn <nick at desert.net>:
>>> The following reply was made to PR threads/136345; it has been  
>>> noted by GNATS.
>>>
>>> From: Nick Esborn <nick at desert.net>
>>> To: bug-followup at FreeBSD.org,
>>> rink at FreeBSD.org
>>> Cc:
>>> Subject: Re: threads/136345: Recursive read rwlocks in thread A  
>>> cause deadlock with write lock in thread B
>>> Date: Wed, 15 Jul 2009 14:32:38 -0700
>>>
>>> This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
>>> --Apple-Mail-19-950902279
>>> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
>>> Content-Transfer-Encoding: 7bit
>>>
>>> Even after the above patch, I still run into occasional MySQL thread
>>> deadlocks, which I originally described in what is now threads/ 
>>> 135673.
>>>
>>> I also posted on freebsd-current a few days ago:
>>>
>>>   http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009328.html
>>>
>>> I'd be happy to collect whatever data would be helpful in tracking
>>> down this deadlock.  This only seems to happen under our production
>>> workload, so that might make it harder to capture meaningful debug
>>> data, but I'm certainly willing to try.  I can also arrange for
>>> developer access to the system in question, if that would help
>>> significantly.
>>
>> So did you backport this to 7 and still experience deadlocks?
>> I just committed the fix to HEAD not to STABLE branch.
>
> Ok, I got, you just upgraded.
> Can you try the following things?:
> - Upgrade to the -CURRENT of today
> - Recompile the kernel with the following options:
> KDB, DDB, SCHED_ULE, PREEMPTION, FULL_PREEMPTION, INVARIANT_SUPPORT,
> INVARIANTS, WITNESS
> - When the deadlock takes place break into DDB and please retrieve the
> following info:
> db> show allpcpu
> db> ps
> db> alltrace
> db> show alllock
>
> - Save them with a serial console output or using the textdump(4)  
> format.
>
> (if necessary read the ddb(4) and textdump(4) before to set up the
> whole system).
> This would shade a light if the problem lives within the kernel or  
> not.
>
> Thanks,
> Attilio
>
>
> -- 
> Peace can only be achieved by understanding - A. Einstein

I can definitely do the upgrade.

KDB, DDB, SCHED_ULE, and PREEMPTION are already turned on.  I will try  
FULL_PREEMPTION, INVARIANT_SUPPORT, INVARIANTS, and WITNESS, but when  
I first upgraded to 8.0, this server was unable to handle its workload  
with the INVARIANTS and WITNESS options turned on.

Also, it can take a while for it to become clear that the deadlock has  
occurred -- usually our monitoring picks it up when replication falls  
behind.  So it may be 15-20 minutes after the deadlock that I am able  
to run the above db commands.  Of course the thread will still be  
deadlocked.  Hopefully that doesn't reduce the value of the data  
obtained.

Thanks,

-nick

--
nick at desert.net - all messages cryptographically signed

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 194 bytes
Desc: This is a digitally signed message part
Url : http://lists.freebsd.org/pipermail/freebsd-threads/attachments/20090716/512bfdc6/PGP.pgp