[pgpool-general: 4966] Re: PGPool Child Processes constantly dying

[Steven Livingstone]

Thanks for that Vlad - i wasn't aware of that option :-)

I have attached some of the log. In it i can see process 42151 at
12:27:46 dies as the result of a segmentation fault (there are a few
but i chose this one as an example).

if anyone has some time and can see anything it would be much
appreciated. I am looking through it myself.

thanks,
/steven

On Sat, Sep 3, 2016 at 5:23 AM, Vlad Novikov <xou.slackware at gmail.com> wrote:
> Hello Steven,
>
> Did you try enabling debugging mode (e.g. by running pgpool -n -d
> 1>logfile.log 2>&1 &) and adjusting the logging level in pgpool.conf? That
> should provide more information about why the process is dying.
>
> Regards,
> Vlad
>
> On Fri, Sep 2, 2016 at 1:54 AM, Steven Livingstone <steven at livz.org> wrote:
>>
>> Hi. I have a scenario where i have a connection pooling client
>> (PGBouncer) making requests to PGPool and the child processes spun up
>> by PGPool create segmentation faults, increasing in number over time
>> until the client just gets errors.
>>
>> I am new to PGPool debugging and gdb in general, so any help much
>> appreciated! I have turned off just about every advanced option in
>> PGPool.conf over the last week but still can't get to the bottom of
>> this.
>>
>> I did managed to create a backtrace of *something* by attaching to one
>> of the child processes but in attaching to it, it doesn't die any more
>> :-S As soon as i detach, it dies and there is a segfault as shown
>> below (this is a small section ... i just get this constantly over
>> time). .... i suspect i am missing something in how to trace properly.
>>
>> Sep  2 08:51:10 data-ha-node-0 pgpool[12573]: [7332-1] 2016-09-02
>> 08:51:10: pid 12573: WARNING:  child process with pid: 31577 was
>> terminated by segmentation fault
>> Sep  2 08:51:10 data-ha-node-0 pgpool[12573]: [7333-1] 2016-09-02
>> 08:51:10: pid 12573: LOG:  fork a new child process with pid: 31595
>> Sep  2 08:51:10 data-ha-node-0 pgpool[31589]: [7331-1] 2016-09-02
>> 08:51:10: pid 31589: LOG:  new connection received
>> Sep  2 08:51:10 data-ha-node-0 pgpool[31589]: [7331-2] 2016-09-02
>> 08:51:10: pid 31589: DETAIL:  connecting host=10.0.1.8 port=53526
>> Sep  2 08:51:11 data-ha-node-0 kernel: [69483.710126] pgpool[31589]:
>> segfault at ac ip 00000000004354a5 sp 00007fffffff91d0 error 4 in
>> pgpool[400000+132000]
>> Sep  2 08:51:11 data-ha-node-0 pgpool[12573]: [7334-1] 2016-09-02
>> 08:51:11: pid 12573: WARNING:  child process with pid: 31589 was
>> terminated by segmentation fault
>> Sep  2 08:51:11 data-ha-node-0 pgpool[12573]: [7335-1] 2016-09-02
>> 08:51:11: pid 12573: LOG:  fork a new child process with pid: 31601
>> Sep  2 08:51:11 data-ha-node-0 pgpool[31595]: [7333-1] 2016-09-02
>> 08:51:11: pid 31595: LOG:  new connection received
>> Sep  2 08:51:11 data-ha-node-0 pgpool[31595]: [7333-2] 2016-09-02
>> 08:51:11: pid 31595: DETAIL:  connecting host=10.0.1.8 port=53528
>> Sep  2 08:51:11 data-ha-node-0 kernel: [69484.288433] pgpool[31595]:
>> segfault at ac ip 00000000004354a5 sp 00007fffffff91d0 error 4 in
>> pgpool[400000+132000]
>> Sep  2 08:51:12 data-ha-node-0 pgpool[12573]: [7336-1] 2016-09-02
>> 08:51:12: pid 12573: WARNING:  child process with pid: 31595 was
>> terminated by segmentation fault
>> Sep  2 08:51:12 data-ha-node-0 pgpool[12573]: [7337-1] 2016-09-02
>> 08:51:12: pid 12573: LOG:  fork a new child process with pid: 31605
>> Sep  2 08:51:12 data-ha-node-0 pgpool[31586]: [7329-1] 2016-09-02
>> 08:51:12: pid 31586: LOG:  new connection received
>>
>>
>>
>>
>>
>> (gdb) bt
>> #0  0x00007ffff717ac33 in select () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x0000000000426529 in wait_for_new_connections
>> (saddr=0x7fffffff95c0, timeout=0x7fffffff9440, fds=0x7686c0) at
>> protocol/child.c:1854
>> #2  do_child (fds=fds at entry=0x7686c0) at protocol/child.c:295
>> #3  0x00000000004076f5 in fork_a_child (fds=0x7686c0, id=1) at
>> main/pgpool_main.c:678
>> #4  0x00000000004080cc in reaper () at main/pgpool_main.c:2263
>> #5  0x000000000040c2a9 in PgpoolMain
>> (discard_status=discard_status at entry=0 '\000',
>> clear_memcache_oidmaps=clear_memcache_oidmaps at entry=0 '\000') at
>> main/pgpool_main.c:429
>> #6  0x0000000000405f13 in main (argc=<optimized out>, argv=<optimized
>> out>) at main/main.c:310
>>
>> Any ideas?
>>
>> /steven
>> _______________________________________________
>> pgpool-general mailing list
>> pgpool-general at pgpool.net
>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log.rtf
Type: application/rtf
Size: 169862 bytes
Desc: not available
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20160903/4b92657a/attachment-0001.rtf>