[pgpool-general: 4974] Re: PGPool Child Processes constantly dying

[Tatsuo Ishii]

Which version of pgpool-II? Is it possible to get core file?  The file
is automatically created when a process dies because of segfault. You
need to set corelimite size to unlimited. e.g. "ulimit -c unlimited"
before starting pgpool-II.

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp

> Thanks for that Vlad - i wasn't aware of that option :-)
> 
> I have attached some of the log. In it i can see process 42151 at
> 12:27:46 dies as the result of a segmentation fault (there are a few
> but i chose this one as an example).
> 
> if anyone has some time and can see anything it would be much
> appreciated. I am looking through it myself.
> 
> thanks,
> /steven
> 
> On Sat, Sep 3, 2016 at 5:23 AM, Vlad Novikov <xou.slackware at gmail.com> wrote:
>> Hello Steven,
>>
>> Did you try enabling debugging mode (e.g. by running pgpool -n -d
>> 1>logfile.log 2>&1 &) and adjusting the logging level in pgpool.conf? That
>> should provide more information about why the process is dying.
>>
>> Regards,
>> Vlad
>>
>> On Fri, Sep 2, 2016 at 1:54 AM, Steven Livingstone <steven at livz.org> wrote:
>>>
>>> Hi. I have a scenario where i have a connection pooling client
>>> (PGBouncer) making requests to PGPool and the child processes spun up
>>> by PGPool create segmentation faults, increasing in number over time
>>> until the client just gets errors.
>>>
>>> I am new to PGPool debugging and gdb in general, so any help much
>>> appreciated! I have turned off just about every advanced option in
>>> PGPool.conf over the last week but still can't get to the bottom of
>>> this.
>>>
>>> I did managed to create a backtrace of *something* by attaching to one
>>> of the child processes but in attaching to it, it doesn't die any more
>>> :-S As soon as i detach, it dies and there is a segfault as shown
>>> below (this is a small section ... i just get this constantly over
>>> time). .... i suspect i am missing something in how to trace properly.
>>>
>>> Sep  2 08:51:10 data-ha-node-0 pgpool[12573]: [7332-1] 2016-09-02
>>> 08:51:10: pid 12573: WARNING:  child process with pid: 31577 was
>>> terminated by segmentation fault
>>> Sep  2 08:51:10 data-ha-node-0 pgpool[12573]: [7333-1] 2016-09-02
>>> 08:51:10: pid 12573: LOG:  fork a new child process with pid: 31595
>>> Sep  2 08:51:10 data-ha-node-0 pgpool[31589]: [7331-1] 2016-09-02
>>> 08:51:10: pid 31589: LOG:  new connection received
>>> Sep  2 08:51:10 data-ha-node-0 pgpool[31589]: [7331-2] 2016-09-02
>>> 08:51:10: pid 31589: DETAIL:  connecting host=10.0.1.8 port=53526
>>> Sep  2 08:51:11 data-ha-node-0 kernel: [69483.710126] pgpool[31589]:
>>> segfault at ac ip 00000000004354a5 sp 00007fffffff91d0 error 4 in
>>> pgpool[400000+132000]
>>> Sep  2 08:51:11 data-ha-node-0 pgpool[12573]: [7334-1] 2016-09-02
>>> 08:51:11: pid 12573: WARNING:  child process with pid: 31589 was
>>> terminated by segmentation fault
>>> Sep  2 08:51:11 data-ha-node-0 pgpool[12573]: [7335-1] 2016-09-02
>>> 08:51:11: pid 12573: LOG:  fork a new child process with pid: 31601
>>> Sep  2 08:51:11 data-ha-node-0 pgpool[31595]: [7333-1] 2016-09-02
>>> 08:51:11: pid 31595: LOG:  new connection received
>>> Sep  2 08:51:11 data-ha-node-0 pgpool[31595]: [7333-2] 2016-09-02
>>> 08:51:11: pid 31595: DETAIL:  connecting host=10.0.1.8 port=53528
>>> Sep  2 08:51:11 data-ha-node-0 kernel: [69484.288433] pgpool[31595]:
>>> segfault at ac ip 00000000004354a5 sp 00007fffffff91d0 error 4 in
>>> pgpool[400000+132000]
>>> Sep  2 08:51:12 data-ha-node-0 pgpool[12573]: [7336-1] 2016-09-02
>>> 08:51:12: pid 12573: WARNING:  child process with pid: 31595 was
>>> terminated by segmentation fault
>>> Sep  2 08:51:12 data-ha-node-0 pgpool[12573]: [7337-1] 2016-09-02
>>> 08:51:12: pid 12573: LOG:  fork a new child process with pid: 31605
>>> Sep  2 08:51:12 data-ha-node-0 pgpool[31586]: [7329-1] 2016-09-02
>>> 08:51:12: pid 31586: LOG:  new connection received
>>>
>>>
>>>
>>>
>>>
>>> (gdb) bt
>>> #0  0x00007ffff717ac33 in select () from /lib/x86_64-linux-gnu/libc.so.6
>>> #1  0x0000000000426529 in wait_for_new_connections
>>> (saddr=0x7fffffff95c0, timeout=0x7fffffff9440, fds=0x7686c0) at
>>> protocol/child.c:1854
>>> #2  do_child (fds=fds at entry=0x7686c0) at protocol/child.c:295
>>> #3  0x00000000004076f5 in fork_a_child (fds=0x7686c0, id=1) at
>>> main/pgpool_main.c:678
>>> #4  0x00000000004080cc in reaper () at main/pgpool_main.c:2263
>>> #5  0x000000000040c2a9 in PgpoolMain
>>> (discard_status=discard_status at entry=0 '\000',
>>> clear_memcache_oidmaps=clear_memcache_oidmaps at entry=0 '\000') at
>>> main/pgpool_main.c:429
>>> #6  0x0000000000405f13 in main (argc=<optimized out>, argv=<optimized
>>> out>) at main/main.c:310
>>>
>>> Any ideas?
>>>
>>> /steven
>>> _______________________________________________
>>> pgpool-general mailing list
>>> pgpool-general at pgpool.net
>>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>>
>>