View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000421 | Pgpool-II | Bug | public | 2018-08-10 00:45 | 2018-08-11 12:51 |
| Reporter | cleitondomazak | Assigned To | t-ishii | ||
| Priority | normal | Severity | major | Reproducibility | always |
| Status | resolved | Resolution | open | ||
| Product Version | 3.7.5 | ||||
| Summary | 0000421: pgPool does not connect when Master node is down | ||||
| Description | When I do a failover test in PostgreSQL Master instance, executing a service stop, the pgpool simply does not connect, and I need restart the pgpool and then I can connect again, even while the PostgreSQL Master instance is down. This is the pgPool logs after postgresql master has been stopped: pgpool[17956]: [5-1] 2018-08-09 15:01:11: pid 17956: LOG: reading and processing packets pgpool[17741]: 2018-08-09 15:01:11: pid 17956: LOG: reading and processing packets pgpool[17741]: 2018-08-09 15:01:11: pid 17956: DETAIL: postmaster on DB node 0 was shutdown by administrative command pgpool[17741]: 2018-08-09 15:01:11: pid 17956: LOG: received degenerate backend request for node_id: 0 from pid [17956] pgpool[17956]: [5-2] 2018-08-09 15:01:11: pid 17956: DETAIL: postmaster on DB node 0 was shutdown by administrative command pgpool[17741]: 2018-08-09 15:01:11: pid 17741: LOG: Pgpool-II parent process has received failover request pgpool[17741]: 2018-08-09 15:01:11: pid 17741: LOG: starting degeneration. shutdown host 10.61.16.31(5432) pgpool[17956]: [6-1] 2018-08-09 15:01:11: pid 17956: LOG: received degenerate backend request for node_id: 0 from pid [17956] pgpool[17741]: [625-1] 2018-08-09 15:01:11: pid 17741: LOG: Pgpool-II parent process has received failover request pgpool[17741]: [626-1] 2018-08-09 15:01:11: pid 17741: LOG: starting degeneration. shutdown host 10.61.16.31(5432) pgpool[17741]: [627-1] 2018-08-09 15:01:11: pid 17741: LOG: Restart all children pgpool[17741]: 2018-08-09 15:01:11: pid 17741: LOG: Restart all children pgpool[17741]: 2018-08-09 15:01:11: pid 17741: LOG: find_primary_node_repeatedly: waiting for finding a primary node pgpool[17741]: 2018-08-09 15:01:11: pid 17741: LOG: find_primary_node: checking backend no 0 pgpool[17741]: 2018-08-09 15:01:11: pid 17741: LOG: find_primary_node: checking backend no 1 pgpool[17741]: [628-1] 2018-08-09 15:01:11: pid 17741: LOG: find_primary_node_repeatedly: waiting for finding a primary node pgpool[17741]: [629-1] 2018-08-09 15:01:11: pid 17741: LOG: find_primary_node: checking backend no 0 pgpool[17741]: [630-1] 2018-08-09 15:01:11: pid 17741: LOG: find_primary_node: checking backend no 1 pgpool[17741]: [631-1] 2018-08-09 15:01:12: pid 17741: LOG: find_primary_node: checking backend no 0 pgpool[17741]: 2018-08-09 15:01:12: pid 17741: LOG: find_primary_node: checking backend no 0 pgpool[17741]: [632-1] 2018-08-09 15:01:12: pid 17741: LOG: find_primary_node: checking backend no 1 pgpool[17741]: 2018-08-09 15:01:12: pid 17741: LOG: find_primary_node: checking backend no 1 pgpool[18824]: [620-1] 2018-08-09 15:01:12: pid 18824: LOG: failed to connect to PostgreSQL server on "10.61.16.31:5432", getsockopt() detected error "Connection refused" pgpool[17741]: 2018-08-09 15:01:12: pid 18824: LOG: failed to connect to PostgreSQL server on "10.61.16.31:5432", getsockopt() detected error "Connection refused" pgpool[18824]: [621-1] 2018-08-09 15:01:12: pid 18824: ERROR: failed to make persistent db connection pgpool[18824]: [621-2] 2018-08-09 15:01:12: pid 18824: DETAIL: connection to host:"10.61.16.31:5432" failed pgpool[17741]: 2018-08-09 15:01:12: pid 18824: ERROR: failed to make persistent db connection pgpool[17741]: 2018-08-09 15:01:12: pid 18824: DETAIL: connection to host:"10.61.16.31:5432" failed | ||||
| Steps To Reproduce | #PostgreSQL Master service postgresql stop # Try connect on pgPool psql -p 5433 And then the screen is locked until I execute the pgpool restart | ||||
| Additional Information | SO: Ubuntu Server 16.04 pgPool 3.7.5 | ||||
| Tags | failover, load balancing, master slave | ||||
|
|
I suspect that Pgpool-II is looking for the new primary node but it does not exist. Maybe you don't set failover script to promote standby? |
|
|
But when I do the pgpool restart, even with master down, I can connect through pgpool on Slave and the read is OK, as is expected. About the promote standby, I decided to not promote automatically, my application allows me handling this manually. My concern is about any failure that can occur in pgPool > PostgreSQL Master communication, and the node status is set to Down, and I can't keep the connections with Standby at least. Or it's an expected behavior? |
|
|
> But when I do the pgpool restart, even with master down, I can connect through pgpool on Slave and the read is OK, as is expected. Probably Pgpool-II already knows the master is down by looking at pool_status file, if Pgpool-II started without -D option (disreagrding pool_status). I think you could just shorten the time for Pgpool-II looking for the primary node by setting search_primary_node_timeout to, say, 10 (seconds). After 10 seconds passed, Pgpool-II should give up looking for the primary and start to accept connections. |
|
|
First, thank so much for your fast help. I changed the search_primary_node_timeout parameter, and works perfectly now, and "solved" the other problem that now I can see clearly :). When I restarted the pgPool the pcp port wasn't up immediately, and now I realize that pcp port waits for the search_primary_node_timeout. |
|
|
Glad to hear that,. I will change the issue status to "resolved". |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2018-08-10 00:45 | cleitondomazak | New Issue | |
| 2018-08-10 00:45 | cleitondomazak | Tag Attached: failover | |
| 2018-08-10 00:45 | cleitondomazak | Tag Attached: load balancing | |
| 2018-08-10 00:45 | cleitondomazak | Tag Attached: master slave | |
| 2018-08-10 11:51 | t-ishii | Note Added: 0002140 | |
| 2018-08-10 11:52 | t-ishii | Assigned To | => t-ishii |
| 2018-08-10 11:52 | t-ishii | Status | new => feedback |
| 2018-08-10 17:14 | cleitondomazak | Note Added: 0002141 | |
| 2018-08-10 17:14 | cleitondomazak | Status | feedback => assigned |
| 2018-08-10 17:37 | t-ishii | Note Added: 0002142 | |
| 2018-08-10 17:42 | t-ishii | Status | assigned => feedback |
| 2018-08-10 18:20 | cleitondomazak | Note Added: 0002143 | |
| 2018-08-10 18:20 | cleitondomazak | Status | feedback => assigned |
| 2018-08-11 12:51 | t-ishii | Note Added: 0002144 | |
| 2018-08-11 12:51 | t-ishii | Status | assigned => resolved |
