View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0000285pgpool-HABugpublic2017-02-06 15:192017-02-08 13:42
ReporterAshok Tiwari Assigned To 
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionopen 
PlatformlinuxOScentos 7 
Summary0000285: after failover standby not become primary
Descriptionpostgres=# show pool_nodes;
 node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay
---------+-----------------+------+--------+-----------+---------+------------+-------------------+-------------------
 0 | postgres_test_1 | 5432 | down | 0.500000 | standby | 1 | false | 0
 1 | postgres_test_2 | 5432 | up | 0.500000 | standby | 0 | true | 0
(2 rows)
Steps To Reproduceafter node 0 close both server become standby
Additional Information2016-08-22 13:37:34: pid 33206: LOG: waiting for watchdog to initialize
2016-08-22 13:37:34: pid 33208: LOG: setting the local watchdog node name to "<SERVER2>_9000"
2016-08-22 13:37:34: pid 33208: LOG: watchdog cluster configured with 1 remote nodes
2016-08-22 13:37:34: pid 33208: LOG: watchdog remote node:0 on <IP>:9000
2016-08-22 13:37:34: pid 33208: LOG: interface monitoring is disabled in watchdog
2016-08-22 13:37:34: pid 33208: LOG: IPC socket path: "/tmp/.s.PGPOOLWD_CMD.9000"
2016-08-22 13:37:34: pid 33208: LOG: new outbond connection to <IP>:9000
2016-08-22 13:37:34: pid 33208: LOG: watchdog node state changed from [LOADING] to [INITIALIZING]
2016-08-22 13:37:34: pid 33208: LOG: new watchdog node connection is received from "<IP2>:44211"
2016-08-22 13:37:35: pid 33208: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY]
2016-08-22 13:37:35: pid 33208: LOG: successfully joined the watchdog cluster as standby node
2016-08-22 13:37:35: pid 33208: DETAIL: our join coordinator request is accepted by cluster leader node "<SERVER>_9999"
2016-08-22 13:37:35: pid 33206: LOG: watchdog process is initialized
2016-08-22 13:37:35: pid 33208: LOG: new IPC connection received
2016-08-22 13:37:35: pid 33206: LOG: Setting up socket for 0.0.0.0:9999
2016-08-22 13:37:35: pid 33206: LOG: Setting up socket for :::9999
2016-08-22 13:37:35: pid 33210: LOG: 2 watchdog nodes are configured for lifecheck
2016-08-22 13:37:35: pid 33210: LOG: watchdog nodes ID:0 Name:"<SERVER2>_9000"
2016-08-22 13:37:35: pid 33210: DETAIL: Host:"<IP1>" WD Port:9000 pgpool-II port:9999
2016-08-22 13:37:35: pid 33210: LOG: watchdog nodes ID:1 Name:"<SERVER>_9999"
2016-08-22 13:37:35: pid 33210: DETAIL: Host:"<IP2>" WD Port:9000 pgpool-II port:9999
2016-08-22 13:37:35: pid 33206: LOG: pgpool-II successfully started. version 3.5.3 (ekieboshi)
2016-08-22 13:37:35: pid 33206: LOG: find_primary_node: checking backend no 0
2016-08-22 13:37:35: pid 33206: LOG: find_primary_node: checking backend no 1
2016-08-22 13:37:35: pid 33206: LOG: find_primary_node: primary node id is 1
2016-08-22 13:37:36: pid 33214: LOG: set SO_REUSEPORT option to the socket
2016-08-22 13:37:36: pid 33214: LOG: creating socket for sending heartbeat
2016-08-22 13:37:36: pid 33214: DETAIL: set SO_REUSEPORT
2016-08-22 13:37:36: pid 33212: LOG: set SO_REUSEPORT option to the socket
2016-08-22 13:37:36: pid 33212: LOG: creating watchdog heartbeat receive socket.
2016-08-22 13:37:36: pid 33212: DETAIL: set SO_REUSEPORT
2016-08-22 13:37:56: pid 33208: LOG: read from socket failed, remote end closed the connection
2016-08-22 13:37:56: pid 33208: LOG: read from socket failed, remote end closed the connection
2016-08-22 13:37:59: pid 33221: LOG: connection closed.
2016-08-22 13:37:59: pid 33221: DETAIL: retry to create new connection pool
2016-08-22 13:37:59: pid 33221: LOG: failed to connect to PostgreSQL server on "<IP2>:5432", getsockopt() detected error "Connection refused"
2016-08-22 13:37:59: pid 33221: LOG: received degenerate backend request for node_id: 1 from pid [33221]
2016-08-22 13:37:59: pid 33208: LOG: new IPC connection received
2016-08-22 13:37:59: pid 33221: FATAL: failed to create a backend connection
2016-08-22 13:37:59: pid 33221: DETAIL: executing failover on backend
2016-08-22 13:37:59: pid 33206: LOG: starting degeneration. shutdown host <IP2>(5432)
2016-08-22 13:37:59: pid 33206: LOG: Restart all children
2016-08-22 13:37:59: pid 33211: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33213: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33215: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33216: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33217: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33208: LOG: new IPC connection received
2016-08-22 13:37:59: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:37:59: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:37:59: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:37:59: pid 33244: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33218: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33243: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33219: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33242: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33220: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33241: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33228: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33229: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33234: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33230: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33233: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33231: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33232: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33235: LOG: child process received shutdown request signal 3
2016-08-22 13:37:59: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:37:59: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:37:59: pid 33208: LOG: new IPC connection received
2016-08-22 13:37:59: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:37:59: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:37:59: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:37:59: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:37:59: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:37:59: pid 33208: LOG: new IPC connection received
2016-08-22 13:37:59: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:37:59: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:37:59: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:37:59: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:37:59: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:00: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:00: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:00: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:00: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:00: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:00: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:00: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:00: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:00: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:00: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:00: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:00: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:01: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:01: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:01: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:01: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:01: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:01: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:01: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:01: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:01: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:01: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:01: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:01: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:02: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:02: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:02: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:02: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:02: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:02: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:02: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:02: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:02: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:02: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:02: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:02: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:07: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:07: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:07: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:07: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:07: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:07: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:08: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:08: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:08: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:08: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:08: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:08: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:08: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:08: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:08: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:08: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:08: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:08: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:09: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:09: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:09: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:09: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:09: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:09: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:09: pid 33206: WARNING: timeout wating for unlock
2016-08-22 13:38:09: pid 33206: LOG: find_primary_node_repeatedly: waiting for finding a primary node
2016-08-22 13:38:09: pid 33206: LOG: find_primary_node: checking backend no 0
2016-08-22 13:38:09: pid 33206: LOG: find_primary_node: checking backend no 1
2016-08-22 13:38:10: pid 33206: LOG: find_primary_node: checking backend no 0
2016-08-22 13:38:10: pid 33206: LOG: find_primary_node: checking backend no 1
2016-08-22 13:38:11: pid 33206: LOG: find_primary_node: checking backend no 0
2016-08-22 13:38:11: pid 33206: LOG: find_primary_node: checking backend no 1
2016-08-22 13:38:12: pid 33206: LOG: find_primary_node: checking backend no 0
2016-08-22 13:38:12: pid 33206: LOG: find_primary_node: checking backend no 1
2016-08-22 13:38:13: pid 33206: LOG: find_primary_node: checking backend no 0
2016-08-22 13:38:13: pid 33206: LOG: find_primary_node: checking backend no 1
2016-08-22 13:38:14: pid 33206: LOG: find_primary_node: checking backend no 0
2016-08-22 13:38:14: pid 33206: LOG: find_primary_node: checking backend no 1
2016-08-22 13:38:15: pid 33206: LOG: find_primary_node: checking backend no 0
2016-08-22 13:38:15: pid 33206: LOG: find_primary_node: checking backend no 1
2016-08-22 13:38:16: pid 33206: LOG: find_primary_node: checking backend no 0
2016-08-22 13:38:18: pid 33206: LOG: find_primary_node: checking backend no 1
2016-08-22 13:38:19: pid 33206: LOG: failover: no follow backends are degenerated
2016-08-22 13:38:19: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:19: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:19: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:19: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:19: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:19: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:20: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:20: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:20: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:20: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:20: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:20: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:20: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:20: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:20: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:20: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:20: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:20: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:21: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:21: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:21: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:21: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:21: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:21: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:21: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:21: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:21: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:21: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:21: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:21: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:22: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:22: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:22: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:22: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:22: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:22: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:22: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:22: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:22: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:22: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:22: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:22: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:23: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:23: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:23: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:23: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:23: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:23: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:23: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:23: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:23: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:23: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:25: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:25: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:25: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:25: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:25: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:25: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:26: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:26: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:26: pid 33208: LOG: failed to process sync request from IPC socket
2016-08-22 13:38:26: pid 33208: DETAIL: failed to forward the request to master watchdog node "<SERVER>_9999"
2016-08-22 13:38:26: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:26: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:26: pid 33210: LOG: informing the node status change to watchdog
2016-08-22 13:38:26: pid 33210: DETAIL: node id :1 status = "NODE DEAD" message:"No heartbeat signal from node"
2016-08-22 13:38:26: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:26: pid 33208: LOG: received node status change ipc message
2016-08-22 13:38:26: pid 33208: DETAIL: No heartbeat signal from node
2016-08-22 13:38:26: pid 33208: LOG: remote node "<SERVER>_9999" is lost
2016-08-22 13:38:26: pid 33208: LOG: watchdog cluster has lost the coordinator node
2016-08-22 13:38:26: pid 33208: LOG: watchdog node state changed from [STANDBY] to [JOINING]
2016-08-22 13:38:26: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:26: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:26: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:26: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:27: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:27: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:27: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:27: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:27: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:27: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:27: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:27: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:28: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:28: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:28: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:28: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:28: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:28: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:28: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:28: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:29: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:29: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:29: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:29: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:29: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:29: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:29: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:29: pid 33206: DETAIL: issue command to watchdog returned NULL
2016-08-22 13:38:30: pid 33206: WARNING: timeout wating for unlock
2016-08-22 13:38:30: pid 33206: LOG: failover: set new primary node: -1
2016-08-22 13:38:30: pid 33206: LOG: failover: set new master node: 0
2016-08-22 13:38:30: pid 33246: LOG: worker process received restart request
2016-08-22 13:38:30: pid 33208: LOG: new IPC connection received
2016-08-22 13:38:30: pid 33208: LOG: processing sync request from IPC socket
2016-08-22 13:38:30: pid 33206: WARNING: watchdog failed to send failover command
2016-08-22 13:38:30: pid 33206: DETAIL: issue command to watchdog returned NULL
failover done. shutdown host <IP2>(5432)2016-08-22 13:38:30: pid 33206: LOG: failover done. shutdown host <IP2>(5432)
2016-08-22 13:38:31: pid 33245: LOG: restart request received in pcp child process
2016-08-22 13:38:31: pid 33206: LOG: PCP child 33245 exits with status 0 in failover()
2016-08-22 13:38:31: pid 33206: LOG: fork a new PCP child pid 33360 in failover()
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33211 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33211 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33213 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33213 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33215 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33215 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33216 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33216 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33217 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33217 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33218 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33218 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33219 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33219 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33220 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33220 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33221 exits with status 256
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33221 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33222 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33222 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33223 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33223 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33224 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33224 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33225 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33225 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33226 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33226 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33227 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33227 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33228 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33228 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33229 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33229 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33230 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33230 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33231 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33231 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33232 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33232 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33233 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33233 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33234 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33234 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33235 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33235 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33236 exits with status 0
2016-08-22 13:38:31: pid 33206: LOG: child process with pid: 33236 exited with success and will not be restarted
2016-08-22 13:38:31: pid 33206: LOG: worker child process with pid: 33246 exits with status 256
2016-08-22 13:38:31: pid 33206: LOG: fork a new worker child process with pid: 33361
2016-08-22 13:38:32: pid 33208: LOG: watchdog node state changed from [JOINING] to [INITIALIZING]
2016-08-22 13:38:33: pid 33208: LOG: I am the only alive node in the watchdog cluster
2016-08-22 13:38:33: pid 33208: HINT: skiping stand for coordinator state
2016-08-22 13:38:33: pid 33208: LOG: watchdog node state changed from [INITIALIZING] to [MASTER]
2016-08-22 13:38:33: pid 33208: LOG: I am announcing my self as master/coordinator watchdog node
2016-08-22 13:38:38: pid 33208: LOG: I am the cluster leader node
2016-08-22 13:38:38: pid 33208: DETAIL: our declare coordinator message is accepted by all nodes
2016-08-22 13:38:38: pid 33208: LOG: I am the cluster leader node. Starting escalation process
2016-08-22 13:38:38: pid 33208: LOG: escalation process started with PID:33374
2016-08-22 13:38:38: pid 33374: LOG: watchdog: escalation started
2016-08-22 13:38:38: pid 33208: LOG: watchdog escalation process with pid: 33374 exit with SUCCESS.

TagsNo tags attached.

Activities

t-ishii

2017-02-08 13:41

developer   ~0001338

Last edited: 2017-02-08 13:42

 Can you please change "project" and submit report again? Apparently this is not related to pgpool-HA.

In the mean time I will close this issue.

Issue History

Date Modified Username Field Change
2017-02-06 15:19 Ashok Tiwari New Issue
2017-02-08 13:41 t-ishii Note Added: 0001338
2017-02-08 13:42 t-ishii Note Edited: 0001338
2017-02-08 13:42 t-ishii Status new => closed