[pgpool-general: 4963] PGPool Child Processes constantly dying

[Steven Livingstone]

Hi. I have a scenario where i have a connection pooling client
(PGBouncer) making requests to PGPool and the child processes spun up
by PGPool create segmentation faults, increasing in number over time
until the client just gets errors.

I am new to PGPool debugging and gdb in general, so any help much
appreciated! I have turned off just about every advanced option in
PGPool.conf over the last week but still can't get to the bottom of
this.

I did managed to create a backtrace of *something* by attaching to one
of the child processes but in attaching to it, it doesn't die any more
:-S As soon as i detach, it dies and there is a segfault as shown
below (this is a small section ... i just get this constantly over
time). .... i suspect i am missing something in how to trace properly.

Sep  2 08:51:10 data-ha-node-0 pgpool[12573]: [7332-1] 2016-09-02
08:51:10: pid 12573: WARNING:  child process with pid: 31577 was
terminated by segmentation fault
Sep  2 08:51:10 data-ha-node-0 pgpool[12573]: [7333-1] 2016-09-02
08:51:10: pid 12573: LOG:  fork a new child process with pid: 31595
Sep  2 08:51:10 data-ha-node-0 pgpool[31589]: [7331-1] 2016-09-02
08:51:10: pid 31589: LOG:  new connection received
Sep  2 08:51:10 data-ha-node-0 pgpool[31589]: [7331-2] 2016-09-02
08:51:10: pid 31589: DETAIL:  connecting host=10.0.1.8 port=53526
Sep  2 08:51:11 data-ha-node-0 kernel: [69483.710126] pgpool[31589]:
segfault at ac ip 00000000004354a5 sp 00007fffffff91d0 error 4 in
pgpool[400000+132000]
Sep  2 08:51:11 data-ha-node-0 pgpool[12573]: [7334-1] 2016-09-02
08:51:11: pid 12573: WARNING:  child process with pid: 31589 was
terminated by segmentation fault
Sep  2 08:51:11 data-ha-node-0 pgpool[12573]: [7335-1] 2016-09-02
08:51:11: pid 12573: LOG:  fork a new child process with pid: 31601
Sep  2 08:51:11 data-ha-node-0 pgpool[31595]: [7333-1] 2016-09-02
08:51:11: pid 31595: LOG:  new connection received
Sep  2 08:51:11 data-ha-node-0 pgpool[31595]: [7333-2] 2016-09-02
08:51:11: pid 31595: DETAIL:  connecting host=10.0.1.8 port=53528
Sep  2 08:51:11 data-ha-node-0 kernel: [69484.288433] pgpool[31595]:
segfault at ac ip 00000000004354a5 sp 00007fffffff91d0 error 4 in
pgpool[400000+132000]
Sep  2 08:51:12 data-ha-node-0 pgpool[12573]: [7336-1] 2016-09-02
08:51:12: pid 12573: WARNING:  child process with pid: 31595 was
terminated by segmentation fault
Sep  2 08:51:12 data-ha-node-0 pgpool[12573]: [7337-1] 2016-09-02
08:51:12: pid 12573: LOG:  fork a new child process with pid: 31605
Sep  2 08:51:12 data-ha-node-0 pgpool[31586]: [7329-1] 2016-09-02
08:51:12: pid 31586: LOG:  new connection received





(gdb) bt
#0  0x00007ffff717ac33 in select () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x0000000000426529 in wait_for_new_connections
(saddr=0x7fffffff95c0, timeout=0x7fffffff9440, fds=0x7686c0) at
protocol/child.c:1854
#2  do_child (fds=fds at entry=0x7686c0) at protocol/child.c:295
#3  0x00000000004076f5 in fork_a_child (fds=0x7686c0, id=1) at
main/pgpool_main.c:678
#4  0x00000000004080cc in reaper () at main/pgpool_main.c:2263
#5  0x000000000040c2a9 in PgpoolMain
(discard_status=discard_status at entry=0 '\000',
clear_memcache_oidmaps=clear_memcache_oidmaps at entry=0 '\000') at
main/pgpool_main.c:429
#6  0x0000000000405f13 in main (argc=<optimized out>, argv=<optimized
out>) at main/main.c:310

Any ideas?

/steven