[Pgpool-general] pgpool segfaulting and dumping core

[Lonni J Friedman]

On Thu, Sep 22, 2011 at 8:58 AM, Tatsuo Ishii <ishii at sraoss.co.jp> wrote:
>> Greetings,
>> I'm running pgpool-II-3.0.4 on a Linux-x86_64 server, which is load
>> balancing for a cluster of 3 PostgreSQL-9.0.4 servers (1 master, 2
>> standby).  I'm using pgpool for load balancing only (not managing
>> streaming replication or failover).  Last night, pgpool started
>> segfaulting repeatedly, and dumping core each time.  What triggered
>> this was a DNS outage, which resulted in pgpool being unable to
>> connect to some of the database servers, and write queries failing, as
>> it couldn't send write queries to a 'master'.  While its obviously not
>> pgpool's fault that the DNS stopped working, I would expect it to not
>> segfault simply because it can't send write queries to a master.
>>
>> Is this a known bug in 3.0.4 that is fixed in 3.1?  I've been holding
>> off on upgrading to 3.1 until there was a compelling reason to
>> upgrade.    Is someone interested in obtaining the core dumps for
>> analysis?
>
> Yes, I'm interested. Please show me a backtrace.

OK, here you go:

############
$ gdb /usr/sbin/pgpool coredump
GNU gdb (GDB) Red Hat Enterprise Linux (7.1-29.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/pgpool...Reading symbols from
/usr/lib/debug/usr/sbin/pgpool.debug...done.
done.
[New Thread 18977]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install
/usr/lib/debug/.build-id/53/01d144075c3c79c94529528e5c39bc1a3c188e
Reading symbols from /usr/lib64/libpq.so.5...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libpq.so.5
Reading symbols from /usr/lib64/libpcp.so.0.0.0...Reading symbols from
/usr/lib/debug/usr/lib64/libpcp.so.0.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libpcp.so.0.0.0
Reading symbols from /lib64/libpam.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpam.so.0
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libldap_r-2.4.so.2...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libldap_r-2.4.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libaudit.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libaudit.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /usr/lib64/libfreebl3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libfreebl3.so
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging
symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libkrb5support.so.0...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /usr/lib64/liblber-2.4.so.2...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/liblber-2.4.so.2
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_dns.so.2
Core was generated by `pgpool: lfriedman cbs 10.31.97.107(52153) idle   '.
Program terminated with signal 11, Segmentation fault.
#0  0x00000000004427cf in ReadyForQuery (frontend=0x918560,
backend=0x918150, send_ready=1)
    at pool_proto_modules.c:1250
1250			state = TSTATE(backend,
Missing separate debuginfos, use: debuginfo-install
audit-libs-2.0.4-1.el6.x86_64 cyrus-sasl-lib-2.1.23-8.el6.x86_64
glibc-2.12-1.7.el6.x86_64 keyutils-libs-1.4-1.el6.x86_64
krb5-libs-1.8.2-3.el6.x86_64 libcom_err-1.41.12-3.el6.x86_64
libselinux-2.0.94-2.el6.x86_64
nss-softokn-freebl-3.12.7-1.1.el6.x86_64 openldap-2.4.19-15.el6.x86_64
openssl-1.0.0-4.el6.x86_64 pam-1.1.1-4.el6.x86_64
postgresql-libs-8.4.4-2.el6.x86_64 zlib-1.2.3-25.el6.x86_64
(gdb) bt
#0  0x00000000004427cf in ReadyForQuery (frontend=0x918560,
backend=0x918150, send_ready=1)
    at pool_proto_modules.c:1250
#1  0x00000000004441aa in ProcessBackendResponse (frontend=0x918560,
backend=0x918150,
    state=0x7fff3ab808d8, num_fields=0x7fff3ab808de) at
pool_proto_modules.c:2231
#2  0x000000000041769a in pool_process_query (frontend=0x918560,
backend=0x918150,
    reset_request=<value optimized out>) at pool_process_query.c:378
#3  0x000000000040a4cd in do_child (unix_fd=<value optimized out>,
    inet_fd=<value optimized out>) at child.c:328
#4  0x0000000000405345 in fork_a_child (unix_fd=3, inet_fd=4, id=98)
at main.c:1033
#5  0x0000000000405a50 in failover () at main.c:1603
#6  0x00000000004076bf in main (argc=<value optimized out>,
argv=<value optimized out>)
    at main.c:563
############

I checked a second of the coredumps, and it has the same backtrace.
Let me know if you need anything else.  thanks!