View Issue Details

IDProjectCategoryView StatusLast Update
0000421Pgpool-IIBugpublic2018-08-11 12:51
ReportercleitondomazakAssigned Tot-ishii 
PrioritynormalSeveritymajorReproducibilityalways
Status resolvedResolutionopen 
Product Version3.7.5 
Target VersionFixed in Version 
Summary0000421: pgPool does not connect when Master node is down
DescriptionWhen I do a failover test in PostgreSQL Master instance, executing a service stop, the pgpool simply does not connect, and I need restart the pgpool and then I can connect again, even while the PostgreSQL Master instance is down.

This is the pgPool logs after postgresql master has been stopped:

pgpool[17956]: [5-1] 2018-08-09 15:01:11: pid 17956: LOG: reading and processing packets
pgpool[17741]: 2018-08-09 15:01:11: pid 17956: LOG: reading and processing packets
pgpool[17741]: 2018-08-09 15:01:11: pid 17956: DETAIL: postmaster on DB node 0 was shutdown by administrative command
pgpool[17741]: 2018-08-09 15:01:11: pid 17956: LOG: received degenerate backend request for node_id: 0 from pid [17956]
pgpool[17956]: [5-2] 2018-08-09 15:01:11: pid 17956: DETAIL: postmaster on DB node 0 was shutdown by administrative command
pgpool[17741]: 2018-08-09 15:01:11: pid 17741: LOG: Pgpool-II parent process has received failover request
pgpool[17741]: 2018-08-09 15:01:11: pid 17741: LOG: starting degeneration. shutdown host 10.61.16.31(5432)
pgpool[17956]: [6-1] 2018-08-09 15:01:11: pid 17956: LOG: received degenerate backend request for node_id: 0 from pid [17956]
pgpool[17741]: [625-1] 2018-08-09 15:01:11: pid 17741: LOG: Pgpool-II parent process has received failover request
pgpool[17741]: [626-1] 2018-08-09 15:01:11: pid 17741: LOG: starting degeneration. shutdown host 10.61.16.31(5432)
pgpool[17741]: [627-1] 2018-08-09 15:01:11: pid 17741: LOG: Restart all children
pgpool[17741]: 2018-08-09 15:01:11: pid 17741: LOG: Restart all children
pgpool[17741]: 2018-08-09 15:01:11: pid 17741: LOG: find_primary_node_repeatedly: waiting for finding a primary node
pgpool[17741]: 2018-08-09 15:01:11: pid 17741: LOG: find_primary_node: checking backend no 0
pgpool[17741]: 2018-08-09 15:01:11: pid 17741: LOG: find_primary_node: checking backend no 1
pgpool[17741]: [628-1] 2018-08-09 15:01:11: pid 17741: LOG: find_primary_node_repeatedly: waiting for finding a primary node
pgpool[17741]: [629-1] 2018-08-09 15:01:11: pid 17741: LOG: find_primary_node: checking backend no 0
pgpool[17741]: [630-1] 2018-08-09 15:01:11: pid 17741: LOG: find_primary_node: checking backend no 1
pgpool[17741]: [631-1] 2018-08-09 15:01:12: pid 17741: LOG: find_primary_node: checking backend no 0
pgpool[17741]: 2018-08-09 15:01:12: pid 17741: LOG: find_primary_node: checking backend no 0
pgpool[17741]: [632-1] 2018-08-09 15:01:12: pid 17741: LOG: find_primary_node: checking backend no 1
pgpool[17741]: 2018-08-09 15:01:12: pid 17741: LOG: find_primary_node: checking backend no 1
pgpool[18824]: [620-1] 2018-08-09 15:01:12: pid 18824: LOG: failed to connect to PostgreSQL server on "10.61.16.31:5432", getsockopt() detected error "Connection refused"
pgpool[17741]: 2018-08-09 15:01:12: pid 18824: LOG: failed to connect to PostgreSQL server on "10.61.16.31:5432", getsockopt() detected error "Connection refused"
pgpool[18824]: [621-1] 2018-08-09 15:01:12: pid 18824: ERROR: failed to make persistent db connection
pgpool[18824]: [621-2] 2018-08-09 15:01:12: pid 18824: DETAIL: connection to host:"10.61.16.31:5432" failed
pgpool[17741]: 2018-08-09 15:01:12: pid 18824: ERROR: failed to make persistent db connection
pgpool[17741]: 2018-08-09 15:01:12: pid 18824: DETAIL: connection to host:"10.61.16.31:5432" failed
Steps To Reproduce#PostgreSQL Master
service postgresql stop

# Try connect on pgPool
psql -p 5433

And then the screen is locked until I execute the pgpool restart
Additional InformationSO: Ubuntu Server 16.04
pgPool 3.7.5

Tagsfailover, load balancing, master slave

Activities

t-ishii

2018-08-10 11:51

developer   ~0002140

I suspect that Pgpool-II is looking for the new primary node but it does not exist. Maybe you don't set failover script to promote standby?

cleitondomazak

2018-08-10 17:14

reporter   ~0002141

But when I do the pgpool restart, even with master down, I can connect through pgpool on Slave and the read is OK, as is expected.

About the promote standby, I decided to not promote automatically, my application allows me handling this manually.

My concern is about any failure that can occur in pgPool > PostgreSQL Master communication, and the node status is set to Down, and I can't keep the connections with Standby at least.

Or it's an expected behavior?

t-ishii

2018-08-10 17:37

developer   ~0002142

> But when I do the pgpool restart, even with master down, I can connect through pgpool on Slave and the read is OK, as is expected.
Probably Pgpool-II already knows the master is down by looking at pool_status file, if Pgpool-II started without -D option (disreagrding pool_status).

I think you could just shorten the time for Pgpool-II looking for the primary node by setting search_primary_node_timeout to, say, 10 (seconds).
After 10 seconds passed, Pgpool-II should give up looking for the primary and start to accept connections.

cleitondomazak

2018-08-10 18:20

reporter   ~0002143

First, thank so much for your fast help.

I changed the search_primary_node_timeout parameter, and works perfectly now, and "solved" the other problem that now I can see clearly :). When I restarted the pgPool the pcp port wasn't up immediately, and now I realize that pcp port waits for the search_primary_node_timeout.

t-ishii

2018-08-11 12:51

developer   ~0002144

Glad to hear that,. I will change the issue status to "resolved".

Issue History

Date Modified Username Field Change
2018-08-10 00:45 cleitondomazak New Issue
2018-08-10 00:45 cleitondomazak Tag Attached: failover
2018-08-10 00:45 cleitondomazak Tag Attached: load balancing
2018-08-10 00:45 cleitondomazak Tag Attached: master slave
2018-08-10 11:51 t-ishii Note Added: 0002140
2018-08-10 11:52 t-ishii Assigned To => t-ishii
2018-08-10 11:52 t-ishii Status new => feedback
2018-08-10 17:14 cleitondomazak Note Added: 0002141
2018-08-10 17:14 cleitondomazak Status feedback => assigned
2018-08-10 17:37 t-ishii Note Added: 0002142
2018-08-10 17:42 t-ishii Status assigned => feedback
2018-08-10 18:20 cleitondomazak Note Added: 0002143
2018-08-10 18:20 cleitondomazak Status feedback => assigned
2018-08-11 12:51 t-ishii Note Added: 0002144
2018-08-11 12:51 t-ishii Status assigned => resolved