[pgpool-general: 8103] Re: Segfaults on PgPool-II 4.3.1

Tue Apr 26 03:23:50 JST 2022

Hello Tatsuo and Kawamoto,

We've been running PgPool built from commit
29734b57b66abcf7ba2c692015e77fc05b38eaa2 for about 96 hours now and we have
not seen the problem occur again.
Hopefully this means the commit fixed the issue.
Thanks for your help.

Gautam

On Tue, Apr 19, 2022 at 7:35 PM Tatsuo Ishii <ishii at sraoss.co.jp> wrote:

> As a side note:
>
> The commit tried to fix the problem. However I was not able to
> reproduce the problem and am not 100% sure the commit solves the
> problem.
>
> If you cannot build pgpool from the source code, you can wait until
> May 19th, 2022. On the day we will release Pgpool-II 4.3.2, which
> includes the commit.
>
> Best reagards,
> --
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese:http://www.sraoss.co.jp
>
> > Hello,
> >
> > Sorry for the late reply.
> > We changed the way of shared memory access  where Seg fault orrured to a
> safer way.
> >
> https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commit;h=f1710d86b61de476372a0086b977647156af415c
> >
> > If possible, could you please use the corrent V4_3_STABLE code to check
> the seg fault no
> > longer occurs?
> >
> > Best regards.
> >
> > On Thu, 14 Apr 2022 11:23:31 -0700
> > Gautam Bellary <gautam at pulsasensors.com> wrote:
> >
> >> Hi PgPool Team,
> >>
> >> Is there any update on this?
> >> Please let us know if there are any more details we can provide.
> >> We continue to observe these segfaults regularly.
> >>
> >> Thanks,
> >> Gautam
> >>
> >> On Wed, Apr 6, 2022 at 6:53 AM Bo Peng <pengbo at sraoss.co.jp> wrote:
> >>
> >> > Hello,
> >> >
> >> > Thank you for reporting this issue.
> >> >
> >> > We are looking into this issue.
> >> > We will let you know when we can figure out how to fix it.
> >> >
> >> > Regards,
> >> >
> >> > On Tue, 5 Apr 2022 10:16:09 -0700
> >> > Gautam Bellary <gautam at pulsasensors.com> wrote:
> >> >
> >> > > Hello PgPool Team,
> >> > >
> >> > > We've been experiencing regular segfaults from PgPool-II 4.3.1 that
> >> > appear
> >> > > to have started recently after we began using this version. Our logs
> >> > > include three different cases of segfaults, ("segfault at 14",
> "segfault
> >> > at
> >> > > 24" and "segfault at 0" in the logs) and we're seeing each segfault
> >> > several
> >> > > times a day on the master/leader instance in our production and test
> >> > > clusters. While these segfaults do not always have an immediate
> impact on
> >> > > connections between our servers and the database, we have observed
> an
> >> > > increasing number of "bad connection" issues over time, which we
> did not
> >> > > observe before upgrading to PgPool 4.3.1 and experiencing the
> segfaults.
> >> > >
> >> > > Details about our environment and the log lines for each segfault
> are
> >> > > included below, and we've attached the core dump backtraces from
> gdb for
> >> > > two of the three segfaults ("segfault at 14" and "segfault at 24"),
> >> > > including `bt` and `bt full`. We'll add details for "segfault at 0"
> here
> >> > or
> >> > > in a new thread when we're able to capture a coredump for it.
> >> > >
> >> > > Thanks,
> >> > > Gautam
> >> > >
> >> > > *Environment details (ubuntu-focal-20.04-amd64):*
> >> > >
> >> > >    - Cluster contains 3 PgPool nodes and 3 PSQL nodes, all on AWS
> EC2
> >> > >    instances, pgpool.conf attached.
> >> > >    - $ uname -a
> >> > >    Linux ip-172-30-166-230 5.4.0-1038-aws #40-Ubuntu SMP Fri Feb 5
> >> > 23:50:40
> >> > >    UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> >> > >    - # SHOW POOL_VERSION;
> >> > >         pool_version
> >> > >    -----------------------
> >> > >     4.3.1 (tamahomeboshi)
> >> > >
> >> > > *Log lines for segfaults:*
> >> > > ISSUE 1 ("segfault at 14") - backtrace attached as
> >> > > "segfault_at_14_backtrace_20220405.txt":
> >> > > [350739.877169] pgpool[1157084]: segfault at 14 ip 000055b950530f25
> sp
> >> > > 00007ffc9e560f50 error 6 in pgpool[55b950503000+d9000]
> >> > > [350739.877178] Code: 00 00 89 ef be 06 00 00 00 c7 44 24 50 01 00
> 00 00
> >> > e8
> >> > > ff 3e fd ff 85 c0 0f 88 a6 0f 00 00 89 ef e8 30 d0 03 00 48 8b 44
> 24 08
> >> > > <c7> 40 14 00 00 00 00 4c 8d a4 24 20 01 00 00 48 8d 35 65 19 23 00
> >> > >
> >> > > ISSUE 2 ("segfault at 24") - backtrace attached as
> >> > > "segfault_at_24_backtrace_20220405.txt":
> >> > > [410790.718731] pgpool[1354761]: segfault at 24 ip 000055b9505308fa
> sp
> >> > > 00007ffc9e560f50 error 6 in pgpool[55b950503000+d9000]
> >> > > [410790.718740] Code: 80 78 4c 01 75 0a 83 78 50 00 0f 84 cd 05 00
> 00 31
> >> > f6
> >> > > 48 8d 3d 0d ac 0b 00 e8 72 c8 02 00 e8 7d 42 fd ff 89 c7 e8 d6 89
> fd ff
> >> > > <c7> 40 24 00 00 00 00 48 8b 05 38 89 1e 00 80 78 4c 01 0f 84 20 01
> >> > >
> >> > > ISSUE 3 ("segfault at 0") - backtrace unavailable at this time:
> >> > > [ 4932.453010] pgpool[53241]: segfault at 0 ip 000055ae84d76163 sp
> >> > > 00007ffd94e8c970 error 4 in pgpool[55ae84d2c000+d9000]
> >> > > [ 4932.453018] Code: 01 c5 41 8b 45 10 85 c0 0f 8f f1 00 00 00 89
> 35 d3
> >> > 60
> >> > > 1e 00 48 8d 35 d0 60 1e 00 48 8d 7e fc 89 0d c6 60 1e 00 e8 4d bd
> ff ff
> >> > > <8b> 38 4c 8d 68 18 83 ef 18 48 63 ff 48 89 3b e8 99 21 02 00 48 8b
> >> >
> >> >
> >> > --
> >> > Bo Peng <pengbo at sraoss.co.jp>
> >> > SRA OSS, Inc. Japan
> >> > http://www.sraoss.co.jp/
> >> >
> >
> >
> > --
> > KAWAMOTO Masaya <kawamoto at sraoss.co.jp>
> > SRA OSS, Inc. Japan
> > _______________________________________________
> > pgpool-general mailing list
> > pgpool-general at pgpool.net
> > http://www.pgpool.net/mailman/listinfo/pgpool-general
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20220425/76f9272e/attachment.htm>