View Issue Details

IDProjectCategoryView StatusLast Update
0000598Pgpool-IIBugpublic2020-05-21 11:30
ReporterXavokAssigned ToMuhammad Usama 
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionfixed 
Product Version4.1.0 
Target Version4.1.2Fixed in Version4.1.2 
Summary0000598: segmentation fault
DescriptionHi
I constantly get an error in the log:
WARNING: PCP process with pid: 23498 was terminated by segmentation fault
Is this normal?
In the attachment, my configuration file and extended log
I use 2xPostgreSQL 11.6 + 1 PgPool-II
Tagspcp, pgpool

Activities

Xavok

2020-03-20 13:06

reporter  

pgpool.log (1,152,940 bytes)
pgpool.conf (40,916 bytes)

t-ishii

2020-03-20 17:47

developer   ~0003273

Last edited: 2020-03-20 18:04

View 2 revisions

Of course it's not normal. Can you provide stack trace of the segfault process?

Xavok

2020-03-21 19:08

reporter   ~0003274

Added GBD, did you need this?

gdb.txt (11,863 bytes)
[root@atlas-nsk-pgpool-02 sysadmin]# gdb -p 22956
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 22956
Reading symbols from /usr/bin/pgpool...Reading symbols from /usr/lib/debug/usr/bin/pgpool.debug...done.
done.
Reading symbols from /usr/pgsql-11/lib/libpq.so.5...Reading symbols from /usr/pgsql-11/lib/libpq.so.5...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /usr/pgsql-11/lib/libpq.so.5
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libmemcached.so.11...Reading symbols from /lib64/libmemcached.so.11...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libmemcached.so.11
Reading symbols from /lib64/libpam.so.0...Reading symbols from /lib64/libpam.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libpam.so.0
Reading symbols from /lib64/libssl.so.10...Reading symbols from /lib64/libssl.so.10...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libssl.so.10
Reading symbols from /lib64/libcrypto.so.10...Reading symbols from /lib64/libcrypto.so.10...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypto.so.10
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libgssapi_krb5.so.2...Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libldap_r-2.4.so.2...Reading symbols from /lib64/libldap_r-2.4.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libldap_r-2.4.so.2
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libsasl2.so.3...Reading symbols from /lib64/libsasl2.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libsasl2.so.3
Reading symbols from /lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libstdc++.so.6
Reading symbols from /lib64/libaudit.so.1...Reading symbols from /lib64/libaudit.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libaudit.so.1
Reading symbols from /lib64/libkrb5.so.3...Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libfreebl3.so...Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /lib64/libkrb5support.so.0...Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/liblber-2.4.so.2...Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/liblber-2.4.so.2
Reading symbols from /lib64/libssl3.so...Reading symbols from /lib64/libssl3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libssl3.so
Reading symbols from /lib64/libsmime3.so...Reading symbols from /lib64/libsmime3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libsmime3.so
Reading symbols from /lib64/libnss3.so...Reading symbols from /lib64/libnss3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss3.so
Reading symbols from /lib64/libnssutil3.so...Reading symbols from /lib64/libnssutil3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnssutil3.so
Reading symbols from /lib64/libplds4.so...Reading symbols from /lib64/libplds4.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libplds4.so
Reading symbols from /lib64/libplc4.so...Reading symbols from /lib64/libplc4.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libplc4.so
Reading symbols from /lib64/libnspr4.so...Reading symbols from /lib64/libnspr4.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnspr4.so
Reading symbols from /lib64/libcap-ng.so.0...Reading symbols from /lib64/libcap-ng.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcap-ng.so.0
Reading symbols from /lib64/libselinux.so.1...Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libpcre.so.1...Reading symbols from /lib64/libpcre.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
0x00007f578554f933 in __select_nocancel () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install audit-libs-2.8.5-4.el7.x86_64 cyrus-sasl-lib-2.1.26-23.el7.x86_64 glibc-2.17-292.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-37.el7_7.2.x86_64 libcap-ng-0.7.5-4.el7.x86_64 libcom_err-1.42.9-16.el7.x86_64 libgcc-4.8.5-39.el7.x86_64 libmemcached-1.0.16-5.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 libstdc++-4.8.5-39.el7.x86_64 nspr-4.21.0-1.el7.x86_64 nss-3.44.0-7.el7_7.x86_64 nss-softokn-freebl-3.44.0-8.el7_7.x86_64 nss-util-3.44.0-4.el7_7.x86_64 openldap-2.4.44-21.el7_6.x86_64 openssl-libs-1.0.2k-19.el7.x86_64 pam-1.1.8-22.el7.x86_64 pcre-8.32-17.el7.x86_64 postgresql11-libs-11.6-2PGDG.rhel7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) bt full
#0  0x00007f578554f933 in __select_nocancel () from /lib64/libc.so.6
No symbol table info available.
#1  0x000000000040b271 in pool_pause (timeout=0x7ffd509e5180) at main/pgpool_main.c:2919
        rfds = {fds_bits = {32, 0 <repeats 15 times>}}
        n = <optimized out>
#2  0x00000000004120b6 in PgpoolMain (discard_status=discard_status@entry=0 '\000', clear_memcache_oidmaps=clear_memcache_oidmaps@entry=0 '\000') at main/pgpool_main.c:506
        r = <optimized out>
        t = {tv_sec = 1, tv_usec = 681768}
        i = 2
        local_sigjmp_buf = {{__jmpbuf = {1, -6324522164246095322, 9869, 140725956006560, 0, 1, 6323174008762587686, -6324517339762730458}, __mask_was_saved = 1, __saved_mask = {
              __val = {18446744066192964103, 140013813826812, 140013813829392, 140725956006720, 140725956006704, 140011638882339, 140725956006744, 140013813829392, 0, 41185760, 
                140725956007312, 140013813829608, 0, 8589934593, 0, 140725956007312}}}}
        first = 0 '\000'
#3  0x0000000000408e29 in main (argc=<optimized out>, argv=<optimized out>) at main/main.c:356
        opt = <optimized out>
        debug_level = <optimized out>
        optindex = 0
        discard_status = 0 '\000'
        clear_memcache_oidmaps = 0 '\000'
        pcp_conf_file_path = "/etc/pgpool-II/pcp.conf", '\000' <repeats 8169 times>
        conf_file_path = "/etc/pgpool-II/pgcluster35.conf", '\000' <repeats 8161 times>
        hba_file_path = "/etc/pgpool-II/pool_hba.conf", '\000' <repeats 8164 times>
        pool_passwd_key_file_path = "/root/.pgpoolkey\000\000\000\000\000\000\000\000\004\000\000\000\024\000\000\000\003\000\000\000GNU\000\234\243\321\037\001\213\356\267\031ͳK\350\000\277\026A5\r\n\000\000\000\000\021\000\000\000\034\000\000\000\004\000\000\000\b\000\000\000@\204\t\000\202!\020\240\250@\000\023\304Ё\030\002\"\030@\030\"D\v\200\030\b\001\">\226\002\034\000\000\000\035\000\000\000\037\000\000\000\"\000\000\000$\000\000\000'\000\000\000+\000\000\000\060\000\000\000\061\000\000\000\062\000\000\000\065\000\000\000\000\000\000\000\066\000\000\000\000\000\000\000\071\000\000\000<\000\000\000=\000\000\000\271\201\272\305\352\323\357\016\021\177\027\257\354\a\262\236\370\262E2\271\215"...
        long_options = {{name = 0x4cb7c7 "hba-file", has_arg = 1, flag = 0x0, val = 97}, {name = 0x4cb7d0 "debug", has_arg = 0, flag = 0x0, val = 100}, {name = 0x4cb7d6 "config-file", 
            has_arg = 1, flag = 0x0, val = 102}, {name = 0x4cb7e2 "key-file", has_arg = 1, flag = 0x0, val = 107}, {name = 0x4cb7eb "pcp-file", has_arg = 1, flag = 0x0, val = 70}, {
            name = 0x4cb7f4 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x4d08a6 "mode", has_arg = 1, flag = 0x0, val = 109}, {name = 0x4cb7f9 "dont-detach", has_arg = 0, 
            flag = 0x0, val = 110}, {name = 0x4cb805 "discard-status", has_arg = 0, flag = 0x0, val = 68}, {name = 0x4cb814 "clear-oidmaps", has_arg = 0, flag = 0x0, val = 67}, {
            name = 0x4cb822 "debug-assertions", has_arg = 0, flag = 0x0, val = 120}, {name = 0x4eac36 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x0, has_arg = 0, 
            flag = 0x0, val = 0}}
(gdb) 
gdb.txt (11,863 bytes)

t-ishii

2020-03-31 13:57

developer   ~0003286

Sorry for delay. gdb.txt looks like a stack trace of Pgpool-II parent process. What I wanted was the stack trace of pgpool child process which segfaults. To get the stack trace, you need to let the pgpool child process produce a core dump file, and take a stack trace from the core file. Can you please do that?

Xavok

2020-04-03 02:49

reporter   ~0003297

In the attachment

gdb.log (6,999 bytes)

t-ishii

2020-04-03 15:10

developer   ~0003298

Thanks. It's pretty strange. Maybe someone broke memory...

Xavok

2020-04-03 15:25

reporter   ~0003299

I have 2 identical virtual servers running multiple instances of pgpool on different ports, could there be a problem because of this?

Xavok

2020-04-03 15:28

reporter   ~0003300

For clarity, added a screen with running pgpool in the attachment

Снимок.JPG (176,068 bytes)
Снимок.JPG (176,068 bytes)

t-ishii

2020-04-03 15:35

developer   ~0003301

> I have 2 identical virtual servers running multiple instances of pgpool on different ports, could there be a problem because of this?
No I don't think so. Pgpool-II's bult-in regression test regularly runs multiple pgpool instances on a physical/virtual machine without problem. I have asked help from Muhammad Usama who is an authority of PCP subsystem.

Xavok

2020-04-15 20:58

reporter   ~0003327

Is there new information?

t-ishii

2020-04-16 14:07

developer   ~0003329

I have changed the person in charge from me to Usama, who is the authority in this area.

Muhammad Usama

2020-04-16 19:27

developer   ~0003330

Thanks, Xavok for providing the gdb stack trace that really helped to quickly pinpoint the issue.

I have pushed the fix to all affected branches.

https://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=75432c06bda37ff9c15e3e586624e21218a0a40c

cheers

Issue History

Date Modified Username Field Change
2020-03-20 13:06 Xavok New Issue
2020-03-20 13:06 Xavok File Added: pgpool.conf
2020-03-20 13:06 Xavok File Added: pgpool.log
2020-03-20 13:06 Xavok Tag Attached: pcp
2020-03-20 13:06 Xavok Tag Attached: pgpool
2020-03-20 17:47 t-ishii Note Added: 0003273
2020-03-20 17:48 t-ishii Assigned To => t-ishii
2020-03-20 17:48 t-ishii Status new => feedback
2020-03-20 17:48 t-ishii Description Updated View Revisions
2020-03-20 18:04 t-ishii Note Edited: 0003273 View Revisions
2020-03-21 19:08 Xavok File Added: gdb.txt
2020-03-21 19:08 Xavok Note Added: 0003274
2020-03-21 19:08 Xavok Status feedback => assigned
2020-03-31 13:57 t-ishii Note Added: 0003286
2020-03-31 13:58 t-ishii Status assigned => feedback
2020-04-03 02:49 Xavok File Added: gdb.log
2020-04-03 02:49 Xavok Note Added: 0003297
2020-04-03 02:49 Xavok Status feedback => assigned
2020-04-03 15:10 t-ishii Note Added: 0003298
2020-04-03 15:25 Xavok Note Added: 0003299
2020-04-03 15:28 Xavok File Added: Снимок.JPG
2020-04-03 15:28 Xavok Note Added: 0003300
2020-04-03 15:35 t-ishii Note Added: 0003301
2020-04-15 20:58 Xavok Note Added: 0003327
2020-04-16 14:06 t-ishii Assigned To t-ishii => Muhammad Usama
2020-04-16 14:07 t-ishii Note Added: 0003329
2020-04-16 19:27 Muhammad Usama Status assigned => resolved
2020-04-16 19:27 Muhammad Usama Resolution open => fixed
2020-04-16 19:27 Muhammad Usama Note Added: 0003330
2020-05-21 11:22 administrator Fixed in Version => 4.1.2
2020-05-21 11:22 administrator Target Version => 4.1.2
2020-05-21 11:30 administrator Status resolved => closed