How do I debug this?

[Matthew Seaman]

We've got an intermittent crash bug with postgresql92-server-9.2.17_2
which seems to be occurring in process startup:

# gdb -c /usr/local/pgsql/data/postgres.core /usr/local/bin/postgres
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Core was generated by `postgres'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libthr.so.3...done.
Loaded symbols for /lib/libthr.so.3
Reading symbols from /usr/local/lib/libxml2.so.2...done.
Loaded symbols for /usr/local/lib/libxml2.so.2
Reading symbols from /usr/local/lib/libicuuc.so.55...done.
Loaded symbols for /usr/local/lib/libicuuc.so.55
Reading symbols from /usr/local/lib/libicui18n.so.55...done.
Loaded symbols for /usr/local/lib/libicui18n.so.55
Reading symbols from /usr/local/lib/libssl.so.8...done.
Loaded symbols for /usr/local/lib/libssl.so.8
Reading symbols from /usr/local/lib/libcrypto.so.8...done.
Loaded symbols for /usr/local/lib/libcrypto.so.8
Reading symbols from /lib/libm.so.5...done.
Loaded symbols for /lib/libm.so.5
Reading symbols from /lib/libc.so.7...done.
Loaded symbols for /lib/libc.so.7
Reading symbols from /lib/libz.so.6...done.
Loaded symbols for /lib/libz.so.6
Reading symbols from /usr/lib/liblzma.so.5...done.
Loaded symbols for /usr/lib/liblzma.so.5
Reading symbols from /usr/local/lib/libicudata.so.55...done.
Loaded symbols for /usr/local/lib/libicudata.so.55
Reading symbols from /usr/lib/libc++.so.1...done.
Loaded symbols for /usr/lib/libc++.so.1
Reading symbols from /lib/libcxxrt.so.1...done.
Loaded symbols for /lib/libcxxrt.so.1
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /usr/local/lib/postgresql/auto_explain.so...done.
Loaded symbols for /usr/local/lib/postgresql/auto_explain.so
Reading symbols from /usr/local/lib/postgresql/plpgsql.so...done.
Loaded symbols for /usr/local/lib/postgresql/plpgsql.so
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x0000000800d1cd0a in _r_debug_postinit () from /libexec/ld-elf.so.1
[New Thread 805806400 (LWP 100362/)]
(gdb) bt full
#0  0x0000000800d1cd0a in _r_debug_postinit () from /libexec/ld-elf.so.1
No symbol table info available.
#1  0x0000000800d1a46d in .text () from /libexec/ld-elf.so.1
No symbol table info available.
#2  0x0000000804ffe174 in ?? () from /lib/libgcc_s.so.1
No symbol table info available.
#3  0x00007fffffffe410 in ?? ()
No symbol table info available.
#4  0x00000008050078f1 in ?? () from /lib/libgcc_s.so.1
No symbol table info available.
#5  0x00007fffffffe410 in ?? ()
No symbol table info available.
#6  0x0000000800d1d9a8 in dlclose () from /libexec/ld-elf.so.1
No symbol table info available.
Previous frame inner to this frame (corrupt stack?)

Application is:

postgresql92-server-9.2.17_2

On:

% freebsd-version -ku
10.3-RELEASE-p4
10.3-RELEASE-p5

This is apparently a postgres child process crashing on startup
somewhere before where main() gets called.  Doesn't happen often -- only
when the server is reasonably busy and then it will take somewhere
between a day and a week between incidents.  We can't correlate it with
any particular SQL being run.  This was a production server so we've had
to move the customers assigned to it back to some slightly older but
very hardware but running the same version of postgresql (same package
compiled on our own pkgrepo) and same 10.3-RELEASE-p4/p5 -- where they
are running quite happily.

	Matthew






-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20160822/0838adff/attachment.sig>