View Issue Details

IDProjectCategoryView StatusLast Update
0000300Pgpool-IIBugpublic2017-05-16 06:21
ReporterdrrtuyAssigned Tot-ishii 
PrioritynormalSeveritymajorReproducibilitysometimes
Status resolvedResolutionopen 
PlatformX86_64OSLinuxOS VersionCentOS 7.1
Product Version3.6.1 
Target VersionFixed in Version3.6.4 
Summary0000300: pgpool-II doesn't handle lost socket file
DescriptionHere are the acronyms used in the schema:

ruby application - RB
pgpool II 3.6.1 PGPOOL
postgres 9.5.4 Master PGM
postgres 9.5.4 Replica PGR

                                 ----> PGM (ip 10.69.64.164)
RB ----> PGPOOL (ip 10.69.64.118)
                                 ----> PGS


We faced a situation when PGPOOL - PGM connection hangs in active state forever. Here is the debug info from PGM side: https://pastebin.com/aYmjjy4p (state of the backend, backtrace of the backend, list of the process fd).

It occures when pgpool somehow lost socket file descriptors of its connection with PGM but the TCP connection is still presented in the system.
Here are the debug info from the pgpool side: https://pastebin.com/vDRBLZ97. It contains ss output that states the connection is alive, list of fds for the pgpool process along with strace of the process.
Steps To ReproduceI can't surely reproduce the bug and I'm looking for advice.
TagsNo tags attached.

Activities

t-ishii

2017-04-14 10:22

developer   ~0001420

Which version of Pgpool-II? You said 3.6.1 but "Product Version" is 3.6.2.
Anyway, I think some of issues similar this have been solved in 3.6 stable tree (will be 3.6.3). Can you please try it out?

drrtuy

2017-04-14 13:06

reporter   ~0001421

Thx for the replay Tatsuo. The actual version is 3.6.1 but I couldn't choose it from the dropdown list, so I took to closest.
I will upgrade my prod servers next week and message you back.
While upgrading I wonder if it possible to turn on a verbose debug on a particular forked pgpool proccess using gdb if debug logging disabled globally? It would be helpfull for frontend protocol violation investigation which seems to me the case of the situation.

t-ishii

2017-04-17 09:21

developer   ~0001423

Actually 3.6.1 can be chosen from the list. I have set the "Product Version" to 3.6.1.

> While upgrading I wonder if it possible to turn on a verbose debug on a particular forked pgpool proccess using gdb if debug logging disabled globally?

Yes. All the configuration variables are stored in global pool_config struct. Each configuration value is stored as the struct member.In your case you want to change log_min_messages:

p g_pool_config.log_min_messages
$2 = 19
(gdb) set pool_config.log_min_messages = 10
(gdb) p g_pool_config.log_min_messages
$3 = 10
(gdb) c

The value you want to change to can be found in src/include/utils/elog.h:

/* Error level codes */
#define DEBUG5 10 /* Debugging messages, in categories of
                                 * decreasing detail. */
#define DEBUG4 11
:
:

drrtuy

2017-05-15 23:58

reporter   ~0001509

The problem has been solved with the set of patches you told Tatsuo, so the issue should be closed.

Issue History

Date Modified Username Field Change
2017-04-13 17:18 drrtuy New Issue
2017-04-14 10:22 t-ishii Assigned To => t-ishii
2017-04-14 10:22 t-ishii Status new => feedback
2017-04-14 10:22 t-ishii Note Added: 0001420
2017-04-14 13:06 drrtuy Note Added: 0001421
2017-04-14 13:06 drrtuy Status feedback => assigned
2017-04-17 09:15 t-ishii Status assigned => feedback
2017-04-17 09:15 t-ishii Product Version 3.6.2 => 3.6.1
2017-04-17 09:21 t-ishii Note Added: 0001423
2017-05-15 23:58 drrtuy Note Added: 0001509
2017-05-15 23:58 drrtuy Status feedback => assigned
2017-05-16 05:54 t-ishii Status assigned => resolved
2017-05-16 06:21 administrator Fixed in Version => 3.6.4