<div dir="ltr">Hello Tatsuo and Kawamoto,<div><br></div><div>We've been running PgPool built from commit <span style="background-color:rgba(29,28,29,0.04);color:rgb(224,30,90);font-family:Monaco,Menlo,Consolas,"Courier New",monospace;font-size:12px;font-variant-ligatures:none;white-space:pre-wrap">29734b57b66abcf7ba2c692015e77fc05b38eaa2</span> for about 96 hours now and we have not seen the problem occur again.</div><div>Hopefully this means the commit fixed the issue.</div><div>Thanks for your help.</div><div><br></div><div>Gautam</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 19, 2022 at 7:35 PM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp">ishii@sraoss.co.jp</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">As a side note:<br>
<br>
The commit tried to fix the problem. However I was not able to<br>
reproduce the problem and am not 100% sure the commit solves the<br>
problem.<br>
<br>
If you cannot build pgpool from the source code, you can wait until<br>
May 19th, 2022. On the day we will release Pgpool-II 4.3.2, which<br>
includes the commit.<br>
<br>
Best reagards,<br>
--<br>
Tatsuo Ishii<br>
SRA OSS, Inc. Japan<br>
English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.jp</a><br>
<br>
> Hello,<br>
> <br>
> Sorry for the late reply.<br>
> We changed the way of shared memory access where Seg fault orrured to a safer way.<br>
> <a href="https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commit;h=f1710d86b61de476372a0086b977647156af415c" rel="noreferrer" target="_blank">https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commit;h=f1710d86b61de476372a0086b977647156af415c</a><br>
> <br>
> If possible, could you please use the corrent V4_3_STABLE code to check the seg fault no<br>
> longer occurs? <br>
> <br>
> Best regards.<br>
> <br>
> On Thu, 14 Apr 2022 11:23:31 -0700<br>
> Gautam Bellary <<a href="mailto:gautam@pulsasensors.com" target="_blank">gautam@pulsasensors.com</a>> wrote:<br>
> <br>
>> Hi PgPool Team,<br>
>> <br>
>> Is there any update on this?<br>
>> Please let us know if there are any more details we can provide.<br>
>> We continue to observe these segfaults regularly.<br>
>> <br>
>> Thanks,<br>
>> Gautam<br>
>> <br>
>> On Wed, Apr 6, 2022 at 6:53 AM Bo Peng <<a href="mailto:pengbo@sraoss.co.jp" target="_blank">pengbo@sraoss.co.jp</a>> wrote:<br>
>> <br>
>> > Hello,<br>
>> ><br>
>> > Thank you for reporting this issue.<br>
>> ><br>
>> > We are looking into this issue.<br>
>> > We will let you know when we can figure out how to fix it.<br>
>> ><br>
>> > Regards,<br>
>> ><br>
>> > On Tue, 5 Apr 2022 10:16:09 -0700<br>
>> > Gautam Bellary <<a href="mailto:gautam@pulsasensors.com" target="_blank">gautam@pulsasensors.com</a>> wrote:<br>
>> ><br>
>> > > Hello PgPool Team,<br>
>> > ><br>
>> > > We've been experiencing regular segfaults from PgPool-II 4.3.1 that<br>
>> > appear<br>
>> > > to have started recently after we began using this version. Our logs<br>
>> > > include three different cases of segfaults, ("segfault at 14", "segfault<br>
>> > at<br>
>> > > 24" and "segfault at 0" in the logs) and we're seeing each segfault<br>
>> > several<br>
>> > > times a day on the master/leader instance in our production and test<br>
>> > > clusters. While these segfaults do not always have an immediate impact on<br>
>> > > connections between our servers and the database, we have observed an<br>
>> > > increasing number of "bad connection" issues over time, which we did not<br>
>> > > observe before upgrading to PgPool 4.3.1 and experiencing the segfaults.<br>
>> > ><br>
>> > > Details about our environment and the log lines for each segfault are<br>
>> > > included below, and we've attached the core dump backtraces from gdb for<br>
>> > > two of the three segfaults ("segfault at 14" and "segfault at 24"),<br>
>> > > including `bt` and `bt full`. We'll add details for "segfault at 0" here<br>
>> > or<br>
>> > > in a new thread when we're able to capture a coredump for it.<br>
>> > ><br>
>> > > Thanks,<br>
>> > > Gautam<br>
>> > ><br>
>> > > *Environment details (ubuntu-focal-20.04-amd64):*<br>
>> > ><br>
>> > > - Cluster contains 3 PgPool nodes and 3 PSQL nodes, all on AWS EC2<br>
>> > > instances, pgpool.conf attached.<br>
>> > > - $ uname -a<br>
>> > > Linux ip-172-30-166-230 5.4.0-1038-aws #40-Ubuntu SMP Fri Feb 5<br>
>> > 23:50:40<br>
>> > > UTC 2021 x86_64 x86_64 x86_64 GNU/Linux<br>
>> > > - # SHOW POOL_VERSION;<br>
>> > > pool_version<br>
>> > > -----------------------<br>
>> > > 4.3.1 (tamahomeboshi)<br>
>> > ><br>
>> > > *Log lines for segfaults:*<br>
>> > > ISSUE 1 ("segfault at 14") - backtrace attached as<br>
>> > > "segfault_at_14_backtrace_20220405.txt":<br>
>> > > [350739.877169] pgpool[1157084]: segfault at 14 ip 000055b950530f25 sp<br>
>> > > 00007ffc9e560f50 error 6 in pgpool[55b950503000+d9000]<br>
>> > > [350739.877178] Code: 00 00 89 ef be 06 00 00 00 c7 44 24 50 01 00 00 00<br>
>> > e8<br>
>> > > ff 3e fd ff 85 c0 0f 88 a6 0f 00 00 89 ef e8 30 d0 03 00 48 8b 44 24 08<br>
>> > > <c7> 40 14 00 00 00 00 4c 8d a4 24 20 01 00 00 48 8d 35 65 19 23 00<br>
>> > ><br>
>> > > ISSUE 2 ("segfault at 24") - backtrace attached as<br>
>> > > "segfault_at_24_backtrace_20220405.txt":<br>
>> > > [410790.718731] pgpool[1354761]: segfault at 24 ip 000055b9505308fa sp<br>
>> > > 00007ffc9e560f50 error 6 in pgpool[55b950503000+d9000]<br>
>> > > [410790.718740] Code: 80 78 4c 01 75 0a 83 78 50 00 0f 84 cd 05 00 00 31<br>
>> > f6<br>
>> > > 48 8d 3d 0d ac 0b 00 e8 72 c8 02 00 e8 7d 42 fd ff 89 c7 e8 d6 89 fd ff<br>
>> > > <c7> 40 24 00 00 00 00 48 8b 05 38 89 1e 00 80 78 4c 01 0f 84 20 01<br>
>> > ><br>
>> > > ISSUE 3 ("segfault at 0") - backtrace unavailable at this time:<br>
>> > > [ 4932.453010] pgpool[53241]: segfault at 0 ip 000055ae84d76163 sp<br>
>> > > 00007ffd94e8c970 error 4 in pgpool[55ae84d2c000+d9000]<br>
>> > > [ 4932.453018] Code: 01 c5 41 8b 45 10 85 c0 0f 8f f1 00 00 00 89 35 d3<br>
>> > 60<br>
>> > > 1e 00 48 8d 35 d0 60 1e 00 48 8d 7e fc 89 0d c6 60 1e 00 e8 4d bd ff ff<br>
>> > > <8b> 38 4c 8d 68 18 83 ef 18 48 63 ff 48 89 3b e8 99 21 02 00 48 8b<br>
>> ><br>
>> ><br>
>> > --<br>
>> > Bo Peng <<a href="mailto:pengbo@sraoss.co.jp" target="_blank">pengbo@sraoss.co.jp</a>><br>
>> > SRA OSS, Inc. Japan<br>
>> > <a href="http://www.sraoss.co.jp/" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/</a><br>
>> ><br>
> <br>
> <br>
> -- <br>
> KAWAMOTO Masaya <<a href="mailto:kawamoto@sraoss.co.jp" target="_blank">kawamoto@sraoss.co.jp</a>><br>
> SRA OSS, Inc. Japan<br>
> _______________________________________________<br>
> pgpool-general mailing list<br>
> <a href="mailto:pgpool-general@pgpool.net" target="_blank">pgpool-general@pgpool.net</a><br>
> <a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br>
</blockquote></div>