View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000474 | Pgpool-II | Bug | public | 2019-03-15 13:05 | 2019-05-21 16:14 |
| Reporter | pensnarik | Assigned To | pengbo | ||
| Priority | high | Severity | major | Reproducibility | random |
| Status | closed | Resolution | open | ||
| Platform | Docker container | OS | Debian Jessie | ||
| Product Version | 4.0.2 | ||||
| Summary | 0000474: pgpool goes to restarting loop after receiving failover request | ||||
| Description | One of our pgpool instances running inside Docker container suddenly went to restarting loop. We have the following messages in our container log: 11:44:19: pid 192: LOG: received degenerate backend request for node_id: 1 from pid [192] 11:44:19: pid 192: FATAL: failed to create a backend connection 11:44:19: pid 192: DETAIL: executing failover on backend 11:44:19: pid 56: LOG: starting degeneration. shutdown host xxx2-service(5432) 11:44:19: pid 192: WARNING: failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known" 11:44:19: pid 56: LOG: Pgpool-II parent process has received failover request 11:44:19: pid 56: LOG: execute command: echo ">>> Failover - that will initialize new primary node search" 11:44:19: pid 56: LOG: Do not restart children because we are switching over node id 1 host: xxx2-service port: 5432 and we are in streaming replication mode >>> Failover - that will initialize new primary node search 11:44:19: pid 56: LOG: find_primary_node_repeatedly: waiting for finding a primary node 11:44:29: pid 56: LOG: find_primary_node: standby node is 2 11:44:29: pid 56: LOG: find_primary_node: primary node is 1 11:44:29: pid 56: LOG: find_primary_node: standby node is 0 failover done. shutdown host xxx2-service(5432) 11:44:29: pid 56: LOG: failover done. shutdown host xxx2-service(5432) 11:44:29: pid 56: LOG: failover: set new master node: 0 11:44:29: pid 56: LOG: failover: set new primary node: 1 11:44:30: pid 212: LOG: restart request received in pcp child process 11:44:30: pid 56: LOG: PCP child 212 exits with status 0 in failover() 11:44:30: pid 233: LOG: failback event detected 11:44:30: pid 56: LOG: fork a new PCP child pid 232 in failover() 11:44:30: pid 233: DETAIL: restarting myself 11:44:31: pid 158: DETAIL: restarting myself 11:44:31: pid 158: LOG: failback event detected 11:44:32: pid 190: LOG: failback event detected 11:44:32: pid 190: DETAIL: restarting myself Then goes many "failback event detected" and "restarting myself" messages. Symptoms are the same as in the bug https://www.pgpool.net/mantisbt/view.php?id=322 which is closed for now. | ||||
| Steps To Reproduce | 1. Run pgpool 2. Cordon one of the slave nodes We have pgpool scaled to 2 nodes and the weird thing is the problem occurred only on one of them (despite the fact the container are absolutely identical). | ||||
| Additional Information | Configuration parameters values, which might be related to the issue: backend_flag0 = 'ALLOW_TO_FAILOVER' num_init_children = 32 max_pool = 4 child_life_time = 300 child_max_connections = 0 connection_life_time = 0 client_idle_limit = 0 connection_cache = on replication_mode = off replicate_select = off replication_stop_on_mismatch = off failover_if_affected_tuples_mismatch = off load_balance_mode = off master_slave_mode = off master_slave_sub_mode = 'stream' delay_threshold = 0 failover_command = '' failback_command = '' failover_on_backend_error = on detach_false_primary = off use_watchdog = off failover_when_quorum_exists = on failover_require_consensus = on allow_multiple_failover_requests_from_node = off | ||||
| Tags | failover | ||||
|
|
Could you show pgpool.conf? |
|
|
Here is my configuration file. |
|
|
Did you use raw mode? I can see only one backend settings in your pgpool.conf. And you have to set a different port to the backend port from pgpool port. Here is an configuration example: http://www.pgpool.net/docs/latest/en/html/example-cluster.html |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2019-03-15 13:05 | pensnarik | New Issue | |
| 2019-03-15 13:05 | pensnarik | Tag Attached: failover | |
| 2019-03-17 21:30 | pengbo | Note Added: 0002432 | |
| 2019-03-18 15:41 | pensnarik | File Added: pgpool-4.0.2.conf | |
| 2019-03-18 15:41 | pensnarik | Note Added: 0002434 | |
| 2019-03-31 22:43 | pengbo | Note Added: 0002488 | |
| 2019-03-31 22:43 | pengbo | Assigned To | => pengbo |
| 2019-03-31 22:43 | pengbo | Status | new => assigned |
| 2019-04-07 23:11 | pengbo | Status | assigned => feedback |
| 2019-05-21 16:14 | administrator | Status | feedback => closed |
