<div dir="ltr">Thanks!<div>In the meantime, do you think that upgrading to 4.3 could solve the problem?</div><div><br></div><div>Thank you</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno mer 2 mar 2022 alle ore 14:51 Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp">ishii@sraoss.co.jp</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Ok, It seems the segfault occured in watchdog.<br>
<br>
Usama,<br>
Can you please take a look at this?<br>
<br>
Best reagards,<br>
--<br>
Tatsuo Ishii<br>
SRA OSS, Inc. Japan<br>
English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.jp</a><br>
<br>
From: shacky <<a href="mailto:shacky83@gmail.com" target="_blank">shacky83@gmail.com</a>><br>
Subject: Re: [pgpool-general: 8031] Re: PgPool-II 4.2.7: several segfaults on CentOS 8<br>
Date: Wed, 2 Mar 2022 12:57:50 +0100<br>
Message-ID: <CAPz3gm=<a href="mailto:wPi_FqSJpnN99gKFBtg0Lg-0x7RiqAPdytfqra8saYQ@mail.gmail.com" target="_blank">wPi_FqSJpnN99gKFBtg0Lg-0x7RiqAPdytfqra8saYQ@mail.gmail.com</a>><br>
<br>
> Here is the complete bt output:<br>
> ===================================================================================================<br>
> DEV root@proxy2-dev ~# gdb /usr/bin/pgpool<br>
> /root/pgpool-coredumps/core.pgpool.26.e120553488834d47bff3c72a21a9bba1.694540.1646129274000000<br>
> GNU gdb (GDB) Red Hat Enterprise Linux 8.2-16.el8<br>
> Copyright (C) 2018 Free Software Foundation, Inc.<br>
> License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html" rel="noreferrer" target="_blank">http://gnu.org/licenses/gpl.html</a><br>
>><br>
> This is free software: you are free to change and redistribute it.<br>
> There is NO WARRANTY, to the extent permitted by law.<br>
> Type "show copying" and "show warranty" for details.<br>
> This GDB was configured as "x86_64-redhat-linux-gnu".<br>
> Type "show configuration" for configuration details.<br>
> For bug reporting instructions, please see:<br>
> <<a href="http://www.gnu.org/software/gdb/bugs/" rel="noreferrer" target="_blank">http://www.gnu.org/software/gdb/bugs/</a>>.<br>
> Find the GDB manual and other documentation resources online at:<br>
> <<a href="http://www.gnu.org/software/gdb/documentation/" rel="noreferrer" target="_blank">http://www.gnu.org/software/gdb/documentation/</a>>.<br>
> <br>
> For help, type "help".<br>
> Type "apropos word" to search for commands related to "word"...<br>
> Reading symbols from /usr/bin/pgpool...Reading symbols from<br>
> /usr/lib/debug/usr/bin/pgpool-4.2.5-1pgdg.rhel8.x86_64.debug...done.<br>
> done.<br>
> <br>
> warning: Can't open file (null) during file-backed mapping note processing<br>
> [New LWP 694540]<br>
> [Thread debugging using libthread_db enabled]<br>
> Using host libthread_db library "/lib64/libthread_db.so.1".<br>
> Core was generated by `pgpool: watchdog '.<br>
> Program terminated with signal SIGSEGV, Segmentation fault.<br>
> #0 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120<br>
> 120 movdqu (%rax), %xmm4<br>
> (gdb) bt<br>
> #0 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120<br>
> #1 0x00007fe9b70ac0af in _IO_vfprintf_internal (s=s@entry=0x7ffdb575f7c0,<br>
> format=format@entry=0x55d3177fd2a8 "STATE MACHINE INVOKED WITH EVENT = %s<br>
> Current State = %s",<br>
> ap=ap@entry=0x7ffdb575f9b0) at vfprintf.c:1638<br>
> #2 0x00007fe9b7164bfd in ___vsnprintf_chk (s=s@entry=0x55d3177fd6b8 "STATE<br>
> MACHINE INVOKED WITH EVENT = it has found us again", maxlen=<optimized<br>
> out>, maxlen@entry=1024,<br>
> flags=flags@entry=1, slen=slen@entry=18446744073709551615,<br>
> format=format@entry=0x55d3177fd2a8 "STATE MACHINE INVOKED WITH EVENT = %s<br>
> Current State = %s",<br>
> args=args@entry=0x7ffdb575f9b0) at vsnprintf_chk.c:63<br>
> #3 0x000055d316dc9013 in vsnprintf (__ap=0x7ffdb575f9b0,<br>
> __fmt=0x55d3177fd2a8 "STATE MACHINE INVOKED WITH EVENT = %s Current State =<br>
> %s", __n=1024,<br>
> __s=0x55d3177fd6b8 "STATE MACHINE INVOKED WITH EVENT = it has found us<br>
> again") at /usr/include/bits/stdio2.h:80<br>
> #4 pvsnprintf (buf=0x55d3177fd6b8 "STATE MACHINE INVOKED WITH EVENT = it<br>
> has found us again", len=len@entry=1024,<br>
> fmt=fmt@entry=0x55d3177fd2a8 "STATE MACHINE INVOKED WITH EVENT = %s<br>
> Current State = %s", args=args@entry=0x7ffdb575f9b0) at utils/psprintf.c:126<br>
> #5 0x000055d316e11eab in appendStringInfoVA (str=str@entry=0x7ffdb575f990,<br>
> fmt=fmt@entry=0x55d3177fd2a8 "STATE MACHINE INVOKED WITH EVENT = %s Current<br>
> State = %s",<br>
> args=args@entry=0x7ffdb575f9b0) at stringinfo.c:138<br>
> #6 0x000055d316ddeabf in errmsg (fmt=fmt@entry=0x55d316f577d0 "STATE<br>
> MACHINE INVOKED WITH EVENT = %s Current State = %s") at<br>
> ../../src/utils/error/elog.c:665<br>
> #7 0x000055d316e30c4b in watchdog_state_machine<br>
> (event=event@entry=WD_EVENT_I_AM_APPEARING_FOUND,<br>
> wdNode=wdNode@entry=0x55d3177fbcf8, pkt=pkt@entry=0x0,<br>
> clusterCommand=clusterCommand@entry=0x0) at watchdog.c:5394<br>
> #8 0x000055d316e34a7b in cluster_service_message_processor<br>
> (pkt=0x55d317804898, wdNode=0x55d3177fbcf8) at watchdog.c:4030<br>
> #9 standard_packet_processor (wdNode=wdNode@entry=0x55d3177fbcf8,<br>
> pkt=pkt@entry=0x55d317804898) at watchdog.c:4202<br>
> #10 0x000055d316e312cd in watchdog_state_machine_nw_isolation<br>
> (clusterCommand=<optimized out>, pkt=<optimized out>, wdNode=<optimized<br>
> out>, event=<optimized out>)<br>
> at watchdog.c:6458<br>
> #11 watchdog_state_machine (event=event@entry=WD_EVENT_PACKET_RCV,<br>
> wdNode=wdNode@entry=0x55d3177fbcf8, pkt=pkt@entry=0x55d317804898,<br>
> clusterCommand=clusterCommand@entry=0x0)<br>
> at watchdog.c:5561<br>
> #12 0x000055d316e38514 in read_sockets (rmask=rmask@entry=0x7ffdb57604e0,<br>
> pending_fds_count=pending_fds_count@entry=1) at watchdog.c:8207<br>
> #13 0x000055d316e3b04f in watchdog_main () at watchdog.c:3500<br>
> #14 0x000055d316e3bb81 in fork_watchdog_child () at watchdog.c:1127<br>
> #15 initialize_watchdog () at watchdog.c:625<br>
> #16 0x000055d316d73088 in reaper () at main/pgpool_main.c:2428<br>
> #17 0x000055d316d79e28 in PgpoolMain (discard_status=<optimized out>,<br>
> clear_memcache_oidmaps=<optimized out>) at main/pgpool_main.c:473<br>
> #18 0x000055d316d7053c in main (argc=<optimized out>, argv=0x7ffdb576d1b8)<br>
> at main/main.c:365<br>
> ===================================================================================================<br>
> <br>
> Thank you!<br>
> <br>
> Il giorno mer 2 mar 2022 alle ore 05:05 Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>><br>
> ha scritto:<br>
> <br>
>> Can you please share the result of gdb command "bt"? Otherwise we<br>
>> cannot figure out which place of pgpool is related to the problem.<br>
>> --<br>
>> Tatsuo Ishii<br>
>> SRA OSS, Inc. Japan<br>
>> English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
>> Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.jp</a><br>
>><br>
>> > This is the coredump of the segmentation fault happened on proxy2-dev:<br>
>> ><br>
>> ><br>
>> ===================================================================================================<br>
>> > DEV root@proxy2-dev /v/l/s/coredump# gdb /usr/bin/pgpool<br>
>> ><br>
>> /root/pgpool-coredumps/core.pgpool.26.e120553488834d47bff3c72a21a9bba1.694540.1646129274000000<br>
>> > GNU gdb (GDB) Red Hat Enterprise Linux 8.2-16.el8<br>
>> > Copyright (C) 2018 Free Software Foundation, Inc.<br>
>> > License GPLv3+: GNU GPL version 3 or later <<br>
>> <a href="http://gnu.org/licenses/gpl.html" rel="noreferrer" target="_blank">http://gnu.org/licenses/gpl.html</a><br>
>> >><br>
>> > This is free software: you are free to change and redistribute it.<br>
>> > There is NO WARRANTY, to the extent permitted by law.<br>
>> > Type "show copying" and "show warranty" for details.<br>
>> > This GDB was configured as "x86_64-redhat-linux-gnu".<br>
>> > Type "show configuration" for configuration details.<br>
>> > For bug reporting instructions, please see:<br>
>> > <<a href="http://www.gnu.org/software/gdb/bugs/" rel="noreferrer" target="_blank">http://www.gnu.org/software/gdb/bugs/</a>>.<br>
>> > Find the GDB manual and other documentation resources online at:<br>
>> > <<a href="http://www.gnu.org/software/gdb/documentation/" rel="noreferrer" target="_blank">http://www.gnu.org/software/gdb/documentation/</a>>.<br>
>> ><br>
>> > For help, type "help".<br>
>> > Type "apropos word" to search for commands related to "word"...<br>
>> > Reading symbols from /usr/bin/pgpool...Reading symbols from<br>
>> > /usr/lib/debug/usr/bin/pgpool-4.2.5-1pgdg.rhel8.x86_64.debug...done.<br>
>> > done.<br>
>> ><br>
>> > warning: Can't open file (null) during file-backed mapping note<br>
>> processing<br>
>> > [New LWP 694540]<br>
>> > [Thread debugging using libthread_db enabled]<br>
>> > Using host libthread_db library "/lib64/libthread_db.so.1".<br>
>> > Core was generated by `pgpool: watchdog '.<br>
>> > Program terminated with signal SIGSEGV, Segmentation fault.<br>
>> > #0 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120<br>
>> > 120 movdqu (%rax), %xmm4<br>
>> ><br>
>> ===================================================================================================<br>
>> ><br>
>> > Thanks.<br>
>> ><br>
>> > Il giorno mar 1 mar 2022 alle ore 17:39 shacky <<a href="mailto:shacky83@gmail.com" target="_blank">shacky83@gmail.com</a>> ha<br>
>> > scritto:<br>
>> ><br>
>> >> Hi,<br>
>> >> I am experiencing many segfaults on PgPool-II 4.2.7 installed on two<br>
>> >> different environments (production and test).<br>
>> >><br>
>> >> Each environment is composed by three different PgPool-II nodes<br>
>> configured<br>
>> >> in a PgPool-II watchdog cluster, and I am experiencing segfaults on<br>
>> every 6<br>
>> >> PgPool-II nodes.<br>
>> >><br>
>> >> The 6 PgPool-II nodes are distributed on 4 different physical servers,<br>
>> so<br>
>> >> it's very unlikely that it's due to some hardware problems.<br>
>> >><br>
>> >> These are the snippets of the segfaults notifications:<br>
>> >><br>
>> >><br>
>> ===================================================================================================<br>
>> >> DEV root@proxy1-dev ~# dmesg|grep pgpool<br>
>> >> [ 15.970081] pgpool[834]: segfault at 8 ip 00007fb6069f5d8a sp<br>
>> >> 00007fff848e7448 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7fb60695c000+1bc000]<br>
>> >> [ 17.672642] pgpool[1336]: segfault at 8 ip 00007fb6069f5d8a sp<br>
>> >> 00007fff848e73e8 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7fb60695c000+1bc000]<br>
>> >><br>
>> >> DEV root@proxy2-dev ~# dmesg|grep pgpool<br>
>> >> [ 2109.530239] pgpool[848]: segfault at 8 ip 00007f372d024d8a sp<br>
>> >> 00007ffe5126c7d8 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7f372cf8b000+1bc000]<br>
>> >> [ 4752.698303] pgpool[22406]: segfault at 8 ip 00007fe9b70f3d8a sp<br>
>> >> 00007ffdb575f298 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7fe9b705a000+1bc000]<br>
>> >> [ 4761.219745] pgpool[22450]: segfault at 8 ip 00007fe9b70f3d8a sp<br>
>> >> 00007ffdb575f238 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7fe9b705a000+1bc000]<br>
>> >> [7089616.463274] pgpool[22568]: segfault at 8 ip 00007fe9b70f3d8a sp<br>
>> >> 00007ffdb575f208 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7fe9b705a000+1bc000]<br>
>> >> [7090394.285732] pgpool[661708]: segfault at 8 ip 00007fe9b70f3d8a sp<br>
>> >> 00007ffdb575f208 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7fe9b705a000+1bc000]<br>
>> >> [7090504.438435] pgpool[661929]: segfault at 8 ip 00007fe9b70f3d8a sp<br>
>> >> 00007ffdb575f208 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7fe9b705a000+1bc000]<br>
>> >> [7090528.565248] pgpool[661991]: segfault at 8 ip 00007fe9b70f3d8a sp<br>
>> >> 00007ffdb575f238 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7fe9b705a000+1bc000]<br>
>> >><br>
>> >> DEV root@proxy3-dev ~# dmesg|grep pgpool<br>
>> >> [ 2085.795853] pgpool[879]: segfault at 8 ip 00007f9f45e57d8a sp<br>
>> >> 00007ffc605d6678 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7f9f45dbe000+1bc000]<br>
>> >> [7088526.747095] pgpool[20105]: segfault at 8 ip 00007f9f45e57d8a sp<br>
>> >> 00007ffc605d65e8 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7f9f45dbe000+1bc000]<br>
>> >> [7088528.639860] pgpool[946407]: segfault at 8 ip 00007f9f45e57d8a sp<br>
>> >> 00007ffc605d6618 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7f9f45dbe000+1bc000]<br>
>> >> [7089324.627060] pgpool[946415]: segfault at 8 ip 00007f9f45e57d8a sp<br>
>> >> 00007ffc605d6618 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7f9f45dbe000+1bc000]<br>
>> >> [7089384.787587] pgpool[946506]: segfault at 8 ip 00007f9f45e57d8a sp<br>
>> >> 00007ffc605d6618 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7f9f45dbe000+1bc000]<br>
>> >> [7089580.047313] pgpool[946521]: segfault at 8 ip 00007f9f45e57d8a sp<br>
>> >> 00007ffc605d6618 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7f9f45dbe000+1bc000]<br>
>> >> [7089591.340077] pgpool[946547]: segfault at 8 ip 00007f9f45e57d8a sp<br>
>> >> 00007ffc605d6618 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7f9f45dbe000+1bc000]<br>
>> >> [7089597.788270] pgpool[946556]: segfault at 8 ip 00007f9f45e57d8a sp<br>
>> >> 00007ffc605d6618 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7f9f45dbe000+1bc000]<br>
>> >> [7089614.064323] pgpool[946565]: segfault at 8 ip 00007f9f45e57d8a sp<br>
>> >> 00007ffc605d6618 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7f9f45dbe000+1bc000]<br>
>> >> [7090492.104936] pgpool[946574]: segfault at 8 ip 00007f9f45e57d8a sp<br>
>> >> 00007ffc605d65e8 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7f9f45dbe000+1bc000]<br>
>> >> [7090575.965820] pgpool[946743]: segfault at 8 ip 00007f9f45e57d8a sp<br>
>> >> 00007ffc605d6618 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7f9f45dbe000+1bc000]<br>
>> >><br>
>> >> PROD root@proxy1-prod ~# dmesg|grep pgpool<br>
>> >> [ 6.631205] pgpool[814]: segfault at 8 ip 00007fe9729dbd8a sp<br>
>> >> 00007ffe7174ae88 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7fe972942000+1bc000]<br>
>> >> [ 17.054962] pgpool[1288]: segfault at 8 ip 00007fe9729dbd8a sp<br>
>> >> 00007ffe7174ae28 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7fe972942000+1bc000]<br>
>> >> [868124.710835] pgpool[1415]: segfault at 8 ip 00007fe9729dbd8a sp<br>
>> >> 00007ffe7174ae28 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7fe972942000+1bc000]<br>
>> >> [868127.082112] pgpool[1632535]: segfault at 8 ip 00007fe9729dbd8a sp<br>
>> >> 00007ffe7174ae28 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7fe972942000+1bc000]<br>
>> >><br>
>> >> PROD root@proxy2-prod ~# dmesg|grep pgpool<br>
>> >> [ 16.783629] pgpool[835]: segfault at 8 ip 00007f5f87877d8a sp<br>
>> >> 00007ffe3d47eb08 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7f5f877de000+1bc000]<br>
>> >> [ 21.012660] pgpool[1381]: segfault at 8 ip 00007f5f87877d8a sp<br>
>> >> 00007ffe3d47eaa8 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7f5f877de000+1bc000]<br>
>> >> [ 1978.102123] pgpool[1400]: segfault at 8 ip 00007f5f87877d8a sp<br>
>> >> 00007ffe3d47eaa8 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7f5f877de000+1bc000]<br>
>> >><br>
>> >> PROD root@proxy3-prod ~# dmesg|grep pgpool<br>
>> >> [ 13.567572] pgpool[826]: segfault at 8 ip 00007f4359c84d8a sp<br>
>> >> 00007fff04ddf008 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7f4359beb000+1bc000]<br>
>> >> [ 15.559884] pgpool[1384]: segfault at 8 ip 00007f4359c84d8a sp<br>
>> >> 00007fff04ddefa8 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7f4359beb000+1bc000]<br>
>> >> [1094174.197742] pgpool[1398]: segfault at 8 ip 00007f4359c84d8a sp<br>
>> >> 00007fff04ddef78 error 4 in <a href="http://libc-2.28.so" rel="noreferrer" target="_blank">libc-2.28.so</a>[7f4359beb000+1bc000]<br>
>> >><br>
>> >><br>
>> ===================================================================================================<br>
>> >><br>
>> >> The operating system is CentOS 8 (fully updated) with these PgPool-II<br>
>> >> packages installed:<br>
>> >><br>
>> >><br>
>> ===================================================================================================<br>
>> >> PROD root@proxy3 ~# rpm -qa|grep -i pgpool<br>
>> >> pgpool-II-pg12-4.2.7-1pgdg.rhel8.x86_64<br>
>> >> pgpool-II-release-4.2-1.noarch<br>
>> >><br>
>> >><br>
>> ===================================================================================================<br>
>> >><br>
>> >> Could you help me to understand what's going on, please?<br>
>> >><br>
>> >> Thank you very much!<br>
>> >> Bye<br>
>> >><br>
>><br>
</blockquote></div>