Perl application hang in umtxn state

Fri Oct 31 00:54:59 PDT 2008

Hi

On Thu, Oct 30, 2008 at 10:32 AM, Václav Haisman <v.haisman at sh.cvut.cz> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Dominique Goncalves wrote:
>> Hi,
>>
>> I run FreeBSD 7.1-PRERELEASE and with a Perl GTK2 application, Perl
>> hang in umtxn state:
>>
>>   PID USERNAME    THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
>> 76288 dom           1  53    0 67860K 46080K umtxn    0:02  0.00% perl5.8.8
>>
>> And the only way to stop the application is kill -9
>>
>> I tried to use ktrace/kdump and ktrace.out show "_umtx_op"
>>
>> $ kdump -f ktrace.out
>> [...]
>>  13199 perl5.8.8 RET   lstat 0
>>  13199 perl5.8.8 CALL  gettimeofday(0xbfbfb144,0)
>>  13199 perl5.8.8 RET   gettimeofday 0
>>  13199 perl5.8.8 CALL  _umtx_op(0xbfbfd18c,0x3,0x1,0,0)
>>  13199 perl5.8.8 RET   _umtx_op 0
>>  13199 perl5.8.8 CALL  sigprocmask(SIG_BLOCK,0xbfbfd120,0x8121490)
>>  13199 perl5.8.8 RET   sigprocmask 0
>>  13199 perl5.8.8 CALL  _umtx_op(0x287fb8a0,0x5,0,0,0)
>>  13199 perl5.8.8 RET   _umtx_op RESTART
>>  13199 perl5.8.8 PSIG  SIGKILL SIG_DFL
>>
>>
>> How to debug this hang? (btw I'm not a developer)
> You can try rebuilding perl with debugging information. I am not sure
> this is the best way but I have used this in my /etc/make.conf:
>
> # Disable binaries stripping.
> STRIP=
> # Add debugging info to binaries.
> CFLAGS?=
> CFLAGS+= -g3 -ggdb
> CXXFLAGS?=
> CXXFLAGS+= -g3 -ggdb

Ok, I've recompiled Perl and others applications with these options.

> When you have that, try running your test case again. When it blocks
> again, attach GDB to the hung process: gdb -p PID `which perl`. And then
> obtain the call stack: (gdb) bt.

# gdb -p 2528 `which perl`
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...(no debugging
symbols found)...
Attaching to program: /usr/bin/perl, process 2528
Reading symbols from /usr/local/lib/perl5/5.8.8/mach/CORE/libperl.so...done.
Loaded symbols for /usr/local/lib/perl5/5.8.8/mach/CORE/libperl.so
Reading symbols from /lib/libm.so.5...done.
Loaded symbols for /lib/libm.so.5
Reading symbols from /lib/libcrypt.so.4...done.
Loaded symbols for /lib/libcrypt.so.4
Reading symbols from /lib/libutil.so.7...done.
Loaded symbols for /lib/libutil.so.7
Reading symbols from /lib/libc.so.7...done.
Loaded symbols for /lib/libc.so.7
Reading symbols from
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/Cwd/Cwd.so...done.
Loaded symbols for /usr/local/lib/perl5/site_perl/5.8.8/mach/auto/Cwd/Cwd.so
Reading symbols from /usr/local/lib/perl5/5.8.8/mach/auto/POSIX/POSIX.so...done.
Loaded symbols for /usr/local/lib/perl5/5.8.8/mach/auto/POSIX/POSIX.so
Reading symbols from
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/Glib/Glib.so...done.
Loaded symbols for /usr/local/lib/perl5/site_perl/5.8.8/mach/auto/Glib/Glib.so
Reading symbols from /usr/local/lib/libgobject-2.0.so.0...done.
Loaded symbols for /usr/local/lib/libgobject-2.0.so.0
Reading symbols from /usr/local/lib/libglib-2.0.so.0...done.
Loaded symbols for /usr/local/lib/libglib-2.0.so.0
Reading symbols from /usr/local/lib/libintl.so.8...done.
Loaded symbols for /usr/local/lib/libintl.so.8
Reading symbols from /usr/local/lib/libiconv.so.3...done.
Loaded symbols for /usr/local/lib/libiconv.so.3
Reading symbols from /usr/local/lib/libpcre.so.0...done.
Loaded symbols for /usr/local/lib/libpcre.so.0
Reading symbols from
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/Cairo/Cairo.so...done.
Loaded symbols for /usr/local/lib/perl5/site_perl/5.8.8/mach/auto/Cairo/Cairo.so
Reading symbols from /usr/local/lib/libcairo.so.2...done.
Loaded symbols for /usr/local/lib/libcairo.so.2
Reading symbols from /usr/local/lib/libfreetype.so.9...done.
Loaded symbols for /usr/local/lib/libfreetype.so.9
Reading symbols from /lib/libz.so.4...done.
Loaded symbols for /lib/libz.so.4
Reading symbols from /usr/local/lib/libfontconfig.so.1...done.
Loaded symbols for /usr/local/lib/libfontconfig.so.1
Reading symbols from /usr/local/lib/libpixman-1.so.9...done.
Loaded symbols for /usr/local/lib/libpixman-1.so.9
Reading symbols from /usr/local/lib/libpng.so.5...done.
Loaded symbols for /usr/local/lib/libpng.so.5
Reading symbols from /usr/local/lib/libxcb-render-util.so.0...done.
Loaded symbols for /usr/local/lib/libxcb-render-util.so.0
Reading symbols from /usr/local/lib/libxcb-render.so.0...done.
Loaded symbols for /usr/local/lib/libxcb-render.so.0
Reading symbols from /usr/local/lib/libxcb.so.1...done.
Loaded symbols for /usr/local/lib/libxcb.so.1
Reading symbols from /usr/local/lib/libXrender.so.1...done.
Loaded symbols for /usr/local/lib/libXrender.so.1
Reading symbols from /usr/local/lib/libX11.so.6...done.
Loaded symbols for /usr/local/lib/libX11.so.6
Reading symbols from /lib/libthr.so.3...done.
[New Thread 0x8125400 (LWP 100093)]
Loaded symbols for /lib/libthr.so.3
Reading symbols from /usr/local/lib/libexpat.so.6...done.
Loaded symbols for /usr/local/lib/libexpat.so.6
Reading symbols from /usr/local/lib/libXau.so.6...done.
Loaded symbols for /usr/local/lib/libXau.so.6
Reading symbols from /usr/local/lib/libXdmcp.so.6...done.
Loaded symbols for /usr/local/lib/libXdmcp.so.6
Reading symbols from /usr/lib/librpcsvc.so.4...done.
Loaded symbols for /usr/lib/librpcsvc.so.4
Reading symbols from
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/Gtk2/Gtk2.so...done.
Loaded symbols for /usr/local/lib/perl5/site_perl/5.8.8/mach/auto/Gtk2/Gtk2.so
Reading symbols from /usr/local/lib/libgtk-x11-2.0.so.0...done.
Loaded symbols for /usr/local/lib/libgtk-x11-2.0.so.0
Reading symbols from /usr/local/lib/libgdk-x11-2.0.so.0...done.
Loaded symbols for /usr/local/lib/libgdk-x11-2.0.so.0
Reading symbols from /usr/local/lib/libatk-1.0.so.0...done.
Loaded symbols for /usr/local/lib/libatk-1.0.so.0
Reading symbols from /usr/local/lib/libgdk_pixbuf-2.0.so.0...done.
Loaded symbols for /usr/local/lib/libgdk_pixbuf-2.0.so.0
Reading symbols from /usr/local/lib/libpangocairo-1.0.so.0...done.
Loaded symbols for /usr/local/lib/libpangocairo-1.0.so.0
Reading symbols from /usr/local/lib/libgio-2.0.so.0...done.
Loaded symbols for /usr/local/lib/libgio-2.0.so.0
Reading symbols from /usr/local/lib/libXext.so.6...done.
Loaded symbols for /usr/local/lib/libXext.so.6
Reading symbols from /usr/local/lib/libXinerama.so.1...done.
Loaded symbols for /usr/local/lib/libXinerama.so.1
Reading symbols from /usr/local/lib/libXi.so.6...done.
Loaded symbols for /usr/local/lib/libXi.so.6
Reading symbols from /usr/local/lib/libXrandr.so.2...done.
Loaded symbols for /usr/local/lib/libXrandr.so.2
Reading symbols from /usr/local/lib/libXcursor.so.1...done.
Loaded symbols for /usr/local/lib/libXcursor.so.1
Reading symbols from /usr/local/lib/libXcomposite.so.1...done.
Loaded symbols for /usr/local/lib/libXcomposite.so.1
Reading symbols from /usr/local/lib/libXdamage.so.1...done.
Loaded symbols for /usr/local/lib/libXdamage.so.1
Reading symbols from /usr/local/lib/libpangoft2-1.0.so.0...done.
Loaded symbols for /usr/local/lib/libpangoft2-1.0.so.0
Reading symbols from /usr/local/lib/libXfixes.so.3...done.
Loaded symbols for /usr/local/lib/libXfixes.so.3
Reading symbols from /usr/local/lib/libpango-1.0.so.0...done.
Loaded symbols for /usr/local/lib/libpango-1.0.so.0
Reading symbols from /usr/local/lib/libgmodule-2.0.so.0...done.
Loaded symbols for /usr/local/lib/libgmodule-2.0.so.0
Reading symbols from /usr/local/lib/perl5/5.8.8/mach/auto/IO/IO.so...done.
Loaded symbols for /usr/local/lib/perl5/5.8.8/mach/auto/IO/IO.so
Reading symbols from /usr/local/lib/perl5/5.8.8/mach/auto/Fcntl/Fcntl.so...done.
Loaded symbols for /usr/local/lib/perl5/5.8.8/mach/auto/Fcntl/Fcntl.so
Reading symbols from
/usr/local/lib/perl5/5.8.8/mach/auto/File/Glob/Glob.so...done.
Loaded symbols for /usr/local/lib/perl5/5.8.8/mach/auto/File/Glob/Glob.so
Reading symbols from
/usr/local/lib/perl5/5.8.8/mach/auto/Storable/Storable.so...done.
Loaded symbols for /usr/local/lib/perl5/5.8.8/mach/auto/Storable/Storable.so
Reading symbols from
/usr/local/lib/perl5/5.8.8/mach/auto/List/Util/Util.so...done.
Loaded symbols for /usr/local/lib/perl5/5.8.8/mach/auto/List/Util/Util.so
Reading symbols from
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/DateTime/DateTime.so...done.
Loaded symbols for
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/DateTime/DateTime.so
Reading symbols from
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/Params/Validate/Validate.so...done.
Loaded symbols for
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/Params/Validate/Validate.so
Reading symbols from
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/List/MoreUtils/MoreUtils.so...done.
Loaded symbols for
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/List/MoreUtils/MoreUtils.so
Reading symbols from
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/Gtk2/Spell/Spell.so...done.
Loaded symbols for
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/Gtk2/Spell/Spell.so
Reading symbols from /usr/local/lib/libgtkspell.so.0...done.
Loaded symbols for /usr/local/lib/libgtkspell.so.0
Reading symbols from /usr/local/lib/libenchant.so.1...done.
Loaded symbols for /usr/local/lib/libenchant.so.1
Reading symbols from
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/Gnome2/VFS/VFS.so...done.
Loaded symbols for
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/Gnome2/VFS/VFS.so
Reading symbols from /usr/local/lib/libgnomevfs-2.so.0...done.
Loaded symbols for /usr/local/lib/libgnomevfs-2.so.0
Reading symbols from /usr/local/lib/libgconf-2.so.4...done.
Loaded symbols for /usr/local/lib/libgconf-2.so.4
Reading symbols from /usr/local/lib/libgthread-2.0.so.0...done.
Loaded symbols for /usr/local/lib/libgthread-2.0.so.0
Reading symbols from /usr/local/lib/libxml2.so.5...done.
Loaded symbols for /usr/local/lib/libxml2.so.5
Reading symbols from /usr/local/lib/libdbus-glib-1.so.2...done.
Loaded symbols for /usr/local/lib/libdbus-glib-1.so.2
Reading symbols from /usr/local/lib/libdbus-1.so.3...done.
Loaded symbols for /usr/local/lib/libdbus-1.so.3
Reading symbols from /usr/lib/libssl.so.5...done.
Loaded symbols for /usr/lib/libssl.so.5
Reading symbols from /lib/libcrypto.so.5...done.
Loaded symbols for /lib/libcrypto.so.5
Reading symbols from /usr/local/lib/libavahi-glib.so.1...done.
Loaded symbols for /usr/local/lib/libavahi-glib.so.1
Reading symbols from /usr/local/lib/libavahi-common.so.3...done.
Loaded symbols for /usr/local/lib/libavahi-common.so.3
Reading symbols from /usr/local/lib/libavahi-client.so.3...done.
Loaded symbols for /usr/local/lib/libavahi-client.so.3
Reading symbols from /usr/local/lib/libORBit-2.so.0...done.
Loaded symbols for /usr/local/lib/libORBit-2.so.0
Reading symbols from /lib/libssp.so.0...done.
Loaded symbols for /lib/libssp.so.0
Reading symbols from
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/HTML/Parser/Parser.so...done.
Loaded symbols for
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/HTML/Parser/Parser.so
Reading symbols from
/usr/local/lib/perl5/5.8.8/mach/auto/Encode/Encode.so...done.
Loaded symbols for /usr/local/lib/perl5/5.8.8/mach/auto/Encode/Encode.so
Reading symbols from
/usr/local/lib/perl5/5.8.8/mach/auto/Socket/Socket.so...done.
Loaded symbols for /usr/local/lib/perl5/5.8.8/mach/auto/Socket/Socket.so
Reading symbols from
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/XML/Parser/Expat/Expat.so...done.
Loaded symbols for
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/XML/Parser/Expat/Expat.so
Reading symbols from
/usr/local/lib/perl5/5.8.8/mach/auto/Unicode/Normalize/Normalize.so...done.
Loaded symbols for
/usr/local/lib/perl5/5.8.8/mach/auto/Unicode/Normalize/Normalize.so
Reading symbols from
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/Compress/Raw/Zlib/Zlib.so...done.
Loaded symbols for
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/Compress/Raw/Zlib/Zlib.so
Reading symbols from
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/Compress/Raw/Bzip2/Bzip2.so...done.
Loaded symbols for
/usr/local/lib/perl5/site_perl/5.8.8/mach/auto/Compress/Raw/Bzip2/Bzip2.so
Reading symbols from
/usr/local/lib/perl5/5.8.8/mach/auto/MIME/Base64/Base64.so...done.
Loaded symbols for /usr/local/lib/perl5/5.8.8/mach/auto/MIME/Base64/Base64.so
Reading symbols from
/usr/local/lib/perl5/5.8.8/mach/auto/Digest/MD5/MD5.so...done.
Loaded symbols for /usr/local/lib/perl5/5.8.8/mach/auto/Digest/MD5/MD5.so
Reading symbols from
/usr/local/lib/perl5/5.8.8/mach/auto/Filter/Util/Call/Call.so...done.
Loaded symbols for /usr/local/lib/perl5/5.8.8/mach/auto/Filter/Util/Call/Call.so
Reading symbols from
/usr/local/lib/gtk-2.0/2.10.0/loaders/libpixbufloader-png.so...done.
Loaded symbols for /usr/local/lib/gtk-2.0/2.10.0/loaders/libpixbufloader-png.so
Reading symbols from
/usr/local/lib/pango/1.6.0/modules/pango-basic-fc.so...done.
Loaded symbols for /usr/local/lib/pango/1.6.0/modules/pango-basic-fc.so
Reading symbols from /usr/local/lib/enchant/libenchant_aspell.so...done.
Loaded symbols for /usr/local/lib/enchant/libenchant_aspell.so
Reading symbols from /usr/local/lib/libaspell.so.16...done.
Loaded symbols for /usr/local/lib/libaspell.so.16
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /usr/local/lib/enchant/libenchant_ispell.so...done.
Loaded symbols for /usr/local/lib/enchant/libenchant_ispell.so
Reading symbols from /usr/local/lib/enchant/libenchant_myspell.so...done.
Loaded symbols for /usr/local/lib/enchant/libenchant_myspell.so
Reading symbols from /usr/local/lib/enchant/libenchant_zemberek.so...done.
Loaded symbols for /usr/local/lib/enchant/libenchant_zemberek.so
Reading symbols from /usr/local/lib/gio/modules/libgiofam.so...done.
Loaded symbols for /usr/local/lib/gio/modules/libgiofam.so
Reading symbols from /usr/local/lib/libfam.so.0...done.
Loaded symbols for /usr/local/lib/libfam.so.0
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
[Switching to Thread 0x8125400 (LWP 100093)]
0x282269a9 in _umtx_op () from /lib/libc.so.7
(gdb) bt
#0  0x282269a9 in _umtx_op () from /lib/libc.so.7
#1  0x28818e5d in __error () from /lib/libthr.so.3
#2  0x2881b8a0 in _thread_state_running () from /lib/libthr.so.3
#3  0x00000005 in ?? ()
#4  0x00000000 in ?? ()
#5  0x00000000 in ?? ()
#6  0x00000000 in ?? ()
#7  0x2881a96c in ?? () from /lib/libthr.so.3
#8  0xbfbfcde8 in ?? ()
#9  0x288150cb in pthread_mutex_destroy () from /lib/libthr.so.3
Previous frame identical to this frame (corrupt stack?)

> I suspect there will be some pthreads call on top of the stack. I am not
> FreeBSD developer but this seems really similar to situations reported
> in thread "umtxn and Apache 2.2" and mine "Process in "uwait" state".
>
>>
>> $ uname -a
>> FreeBSD freebsd7 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #15 r184123:
>> Tue Oct 21 16:24:59 CEST 2008
>> root at freebsd7:/usr/obj/usr/src/sys/GENERIC  i386
>>
>> Thanks for your help,
>> Regards.
>>
>
> - --
> VH
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iFYEAREIAAYFAkkJfx8ACgkQhQBMvHf/WHnzuwDffY0A82o1mpzdVBixQ6bjeqon
> pLikFIEvlXke3ADeIGmhTjnC5TPnJE01XiiZyPuOcrrgYpXhFzTdPw==
> =KOCn
> -----END PGP SIGNATURE-----
>

Regards.

-- 
There's this old saying: "Give a man a fish, feed him for a day. Teach
a man to fish, feed him for life."