View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000658 | Pgpool-II | Bug | public | 2020-11-11 22:52 | 2020-12-02 15:28 |
| Reporter | papegaaij | Assigned To | pengbo | ||
| Priority | normal | Severity | major | Reproducibility | random |
| Status | assigned | Resolution | open | ||
| Product Version | 4.1.4 | ||||
| Summary | 0000658: LOST node cannot rejoin cluster due to life-check | ||||
| Description | Sometimes a node that is LOST is unable to join the cluster because the master refuses it to join. Restarting the master fixed the problem. I've got the feeling the node cannot be marked as healthy by the life-check because it is not allowed to join, but it is refused because it marked LOST by the life-check and needs to be marked healthy first. It seems to happen when a node leaves the cluster right after joining it. After this, it cannot join again. I've attached the logging from both nodes (172.29.30.2 is the master, 172.29.30.3 is the failing node) and the configuration from the master (the config for the other node is identical). | ||||
| Tags | No tags attached. | ||||
|
|
|
|
|
I've done some more investigation of the logs, and you can clearly see that the message in the 172.29.30.3.log: 2020-11-11 14:12:48: pid 14: LOG: creating watchdog heartbeat receive socket. This indicates to me that the lost node does not create its heartbeat socket until it has joined the cluster, but it cannot join the cluster because it is marked lost. |
|
|
I don't know what happened to the attachments, but they seem to be gone. Do you want me to upload them again? |
|
|
Sorry for late response. If it's possible could you upload your log files again? I have looked into this issue, but I could not reproduce it. Could you share the steps to reproduce this issue? |
|
|
I think it must be race condition somewhere. So far I've managed to trigger it once. It happened while I was testing various restart scenarios. So I was stopping and starting nodes in the cluster. I don't think there's a reliable way of reproducing the problem, but the logs do show the problem exists. 172.29.30.3-2.log (163,736 bytes)
2020-11-11 13:48:21: pid 178: LOG: forked new pcp worker, pid=186 socket=7 2020-11-11 13:48:21: pid 178: LOG: PCP process with pid: 186 exit with SUCCESS. 2020-11-11 13:48:21: pid 178: LOG: PCP process with pid: 186 exits with status 0 2020-11-11 13:48:21: pid 178: LOG: forked new pcp worker, pid=192 socket=7 2020-11-11 13:48:21: pid 192: LOG: received degenerate backend request for node_id: 2 from pid [192] 2020-11-11 13:48:21: pid 11: LOG: new IPC connection received 2020-11-11 13:48:21: pid 11: LOG: watchdog received the failover command from local pgpool-II on IPC interface 2020-11-11 13:48:21: pid 11: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface 2020-11-11 13:48:21: pid 11: LOG: The failover request does not need quorum to hold 2020-11-11 13:48:21: pid 11: DETAIL: proceeding with the failover 2020-11-11 13:48:21: pid 11: HINT: REQ_DETAIL_CONFIRMED 2020-11-11 13:48:21: pid 1: LOG: Pgpool-II parent process has received failover request 2020-11-11 13:48:21: pid 11: LOG: new IPC connection received 2020-11-11 13:48:21: pid 11: LOG: received the failover indication from Pgpool-II on IPC interface 2020-11-11 13:48:21: pid 11: LOG: watchdog is informed of failover start by the main process 2020-11-11 13:48:21: pid 1: LOG: starting degeneration. shutdown host 172.29.30.3(5432) 2020-11-11 13:48:21: pid 1: LOG: Do not restart children because we are switching over node id 2 host: 172.29.30.3 port: 5432 and we are in streaming replication mode 2020-11-11 13:48:21: pid 1: LOG: execute command: /etc/pgpool2/pgpool_failover.sh 2 172.29.30.3 5432 /var/lib/postgresql/data/ 0 172.29.30.1 0 1 5432 /var/lib/postgresql/data/ 172.29.30.2 5432 failover.sh: Standby node is down. Skipping failover. 2020-11-11 13:48:21: pid 1: LOG: failover: set new primary node: 1 2020-11-11 13:48:21: pid 1: LOG: failover: set new master node: 0 2020-11-11 13:48:21: pid 11: LOG: new IPC connection received 2020-11-11 13:48:21: pid 11: LOG: received the failover indication from Pgpool-II on IPC interface 2020-11-11 13:48:21: pid 11: LOG: watchdog is informed of failover end by the main process failover done. shutdown host 172.29.30.3(5432)2020-11-11 13:48:21: pid 1: LOG: failover done. shutdown host 172.29.30.3(5432) 2020-11-11 13:48:21: pid 179: LOG: worker process received restart request 2020-11-11 13:48:21: pid 178: LOG: PCP process with pid: 192 exit with SUCCESS. 2020-11-11 13:48:21: pid 178: LOG: PCP process with pid: 192 exits with status 0 2020-11-11 13:48:21: pid 154: LOG: reading and processing packets 2020-11-11 13:48:21: pid 154: DETAIL: postmaster on DB node 2 was shutdown by administrative command 2020-11-11 13:48:22: pid 53: LOG: failed to connect to PostgreSQL server on "172.29.30.3:5432", getsockopt() detected error "Connection refused" 2020-11-11 13:48:22: pid 53: ERROR: failed to make persistent db connection 2020-11-11 13:48:22: pid 53: DETAIL: connection to host:"172.29.30.3:5432" failed 2020-11-11 13:48:22: pid 178: LOG: restart request received in pcp child process 2020-11-11 13:48:22: pid 1: LOG: PCP child 178 exits with status 0 in failover() 2020-11-11 13:48:22: pid 1: LOG: fork a new PCP child pid 195 in failover() 2020-11-11 13:48:22: pid 11: LOG: Watchdog is shutting down 2020-11-11 13:48:22: pid 196: LOG: watchdog: de-escalation started Cleaning old run data... 2020-11-11 13:48:59: pid 1: LOG: Backend status file /var/log/postgresql/pgpool_status does not exist 2020-11-11 13:48:59: pid 1: LOG: memory cache initialized 2020-11-11 13:48:59: pid 1: DETAIL: memcache blocks :64 2020-11-11 13:48:59: pid 1: LOG: pool_discard_oid_maps: discarded memqcache oid maps 2020-11-11 13:48:59: pid 1: LOG: waiting for watchdog to initialize 2020-11-11 13:48:59: pid 11: LOG: setting the local watchdog node name to "172.29.30.3:5432 Linux 6497bd770d79" 2020-11-11 13:48:59: pid 11: LOG: watchdog cluster is configured with 2 remote nodes 2020-11-11 13:48:59: pid 11: LOG: watchdog remote node:0 on 172.29.30.1:9009 2020-11-11 13:48:59: pid 11: LOG: watchdog remote node:1 on 172.29.30.2:9009 2020-11-11 13:48:59: pid 11: LOG: interface monitoring is disabled in watchdog 2020-11-11 13:48:59: pid 11: LOG: watchdog node state changed from [DEAD] to [LOADING] 2020-11-11 13:48:59: pid 11: LOG: new outbound connection to 172.29.30.1:9009 2020-11-11 13:48:59: pid 11: LOG: new outbound connection to 172.29.30.2:9009 2020-11-11 13:48:59: pid 11: LOG: watchdog node state changed from [LOADING] to [INITIALIZING] 2020-11-11 13:48:59: pid 11: LOG: new watchdog node connection is received from "172.29.30.1:25289" 2020-11-11 13:48:59: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:48:59: pid 11: LOG: new watchdog node connection is received from "172.29.30.2:46814" 2020-11-11 13:48:59: pid 11: LOG: new node joined the cluster hostname:"172.29.30.1" port:9009 pgpool_port:5432 2020-11-11 13:48:59: pid 11: DETAIL: Pgpool-II version:"4.1.4" watchdog messaging version: 1.1 2020-11-11 13:48:59: pid 11: LOG: new node joined the cluster hostname:"172.29.30.2" port:9009 pgpool_port:5432 2020-11-11 13:48:59: pid 11: DETAIL: Pgpool-II version:"4.1.4" watchdog messaging version: 1.1 2020-11-11 13:48:59: pid 11: LOG: read from socket failed, remote end closed the connection 2020-11-11 13:48:59: pid 11: LOG: client socket of 172.29.30.2:5432 Linux 41a0ba327eeb is closed 2020-11-11 13:48:59: pid 11: LOG: remote node "172.29.30.2:5432 Linux 41a0ba327eeb" is reporting that it has lost us 2020-11-11 13:48:59: pid 11: LOG: read from socket failed, remote end closed the connection 2020-11-11 13:48:59: pid 11: LOG: outbound socket of 172.29.30.2:5432 Linux 41a0ba327eeb is closed 2020-11-11 13:48:59: pid 11: LOG: remote node "172.29.30.2:5432 Linux 41a0ba327eeb" is not reachable 2020-11-11 13:48:59: pid 11: DETAIL: marking the node as lost 2020-11-11 13:48:59: pid 11: LOG: remote node "172.29.30.2:5432 Linux 41a0ba327eeb" is lost 2020-11-11 13:48:59: pid 11: LOG: watchdog cluster has lost the coordinator node 2020-11-11 13:48:59: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:49:00: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] 2020-11-11 13:49:00: pid 11: LOG: our stand for coordinator request is rejected by node "172.29.30.1:5432 Linux df4d4f6cdb3f" 2020-11-11 13:49:00: pid 11: DETAIL: we might be in partial network isolation and cluster already have a valid master 2020-11-11 13:49:00: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:49:00: pid 11: LOG: watchdog node state changed from [STANDING FOR MASTER] to [NETWORK ISOLATION] 2020-11-11 13:49:01: pid 11: LOG: read from socket failed, remote end closed the connection 2020-11-11 13:49:01: pid 11: LOG: client socket of 172.29.30.1:5432 Linux df4d4f6cdb3f is closed 2020-11-11 13:49:01: pid 11: LOG: remote node "172.29.30.1:5432 Linux df4d4f6cdb3f" is reporting that it has lost us 2020-11-11 13:49:01: pid 11: LOG: read from socket failed, remote end closed the connection 2020-11-11 13:49:01: pid 11: LOG: outbound socket of 172.29.30.1:5432 Linux df4d4f6cdb3f is closed 2020-11-11 13:49:01: pid 11: LOG: remote node "172.29.30.1:5432 Linux df4d4f6cdb3f" is not reachable 2020-11-11 13:49:01: pid 11: DETAIL: marking the node as lost 2020-11-11 13:49:01: pid 11: LOG: remote node "172.29.30.1:5432 Linux df4d4f6cdb3f" is lost 2020-11-11 13:49:10: pid 11: LOG: new watchdog node connection is received from "172.29.30.2:15583" 2020-11-11 13:49:10: pid 11: LOG: trying again to join the cluster 2020-11-11 13:49:10: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:49:10: pid 11: LOG: new node joined the cluster hostname:"172.29.30.2" port:9009 pgpool_port:5432 2020-11-11 13:49:10: pid 11: DETAIL: Pgpool-II version:"4.1.4" watchdog messaging version: 1.1 2020-11-11 13:49:10: pid 11: LOG: The newly joined node:"172.29.30.2:5432 Linux 41a0ba327eeb" had left the cluster because it was lost 2020-11-11 13:49:10: pid 11: DETAIL: lost reason was "NOT REACHABLE" and startup time diff = 0 2020-11-11 13:49:10: pid 11: LOG: new watchdog node connection is received from "172.29.30.1:54985" 2020-11-11 13:49:10: pid 11: LOG: new node joined the cluster hostname:"172.29.30.1" port:9009 pgpool_port:5432 2020-11-11 13:49:10: pid 11: DETAIL: Pgpool-II version:"4.1.4" watchdog messaging version: 1.1 2020-11-11 13:49:10: pid 11: LOG: The newly joined node:"172.29.30.1:5432 Linux df4d4f6cdb3f" had left the cluster because it was lost 2020-11-11 13:49:10: pid 11: DETAIL: lost reason was "NOT REACHABLE" and startup time diff = 0 2020-11-11 13:49:10: pid 11: LOG: new outbound connection to 172.29.30.1:9009 2020-11-11 13:49:10: pid 11: LOG: new outbound connection to 172.29.30.2:9009 2020-11-11 13:49:14: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:49:15: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] 2020-11-11 13:49:15: pid 11: LOG: our stand for coordinator request is rejected by node "172.29.30.1:5432 Linux df4d4f6cdb3f" 2020-11-11 13:49:15: pid 11: DETAIL: we might be in partial network isolation and cluster already have a valid master 2020-11-11 13:49:15: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:49:15: pid 11: LOG: watchdog node state changed from [STANDING FOR MASTER] to [NETWORK ISOLATION] 2020-11-11 13:49:25: pid 11: LOG: trying again to join the cluster 2020-11-11 13:49:25: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:49:25: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:49:25: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:49:26: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:49:26: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:49:26: pid 11: HINT: rejoining the cluster. 2020-11-11 13:49:26: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:49:26: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:49:26: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:49:36: pid 11: LOG: trying again to join the cluster 2020-11-11 13:49:36: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:49:36: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:49:37: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:49:37: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:49:38: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:49:38: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:49:38: pid 11: HINT: rejoining the cluster. 2020-11-11 13:49:38: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:49:38: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:49:38: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:49:48: pid 11: LOG: trying again to join the cluster 2020-11-11 13:49:48: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:49:48: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:49:48: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:49:48: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:49:49: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:49:49: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:49:49: pid 11: HINT: rejoining the cluster. 2020-11-11 13:49:49: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:49:49: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:49:49: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:49:59: pid 11: LOG: trying again to join the cluster 2020-11-11 13:49:59: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:49:59: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:49:59: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:49:59: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:50:00: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:50:00: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:50:00: pid 11: HINT: rejoining the cluster. 2020-11-11 13:50:00: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:50:00: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:50:00: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:50:10: pid 11: LOG: trying again to join the cluster 2020-11-11 13:50:10: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:50:10: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:50:10: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:50:10: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:50:11: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:50:11: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:50:11: pid 11: HINT: rejoining the cluster. 2020-11-11 13:50:11: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:50:11: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:50:11: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:50:21: pid 11: LOG: trying again to join the cluster 2020-11-11 13:50:21: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:50:21: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:50:21: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:50:21: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:50:22: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:50:22: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:50:22: pid 11: HINT: rejoining the cluster. 2020-11-11 13:50:22: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:50:22: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:50:22: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:50:32: pid 11: LOG: trying again to join the cluster 2020-11-11 13:50:32: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:50:32: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:50:32: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:50:32: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:50:33: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:50:33: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:50:33: pid 11: HINT: rejoining the cluster. 2020-11-11 13:50:33: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:50:33: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:50:33: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:50:43: pid 11: LOG: trying again to join the cluster 2020-11-11 13:50:43: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:50:43: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:50:43: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:50:43: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:50:44: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:50:44: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:50:44: pid 11: HINT: rejoining the cluster. 2020-11-11 13:50:44: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:50:44: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:50:44: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:50:54: pid 11: LOG: trying again to join the cluster 2020-11-11 13:50:54: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:50:54: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:50:54: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:50:54: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:50:55: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:50:55: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:50:55: pid 11: HINT: rejoining the cluster. 2020-11-11 13:50:55: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:50:55: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:50:55: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:51:05: pid 11: LOG: trying again to join the cluster 2020-11-11 13:51:05: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:51:05: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:51:05: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:51:05: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:51:06: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:51:06: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:51:06: pid 11: HINT: rejoining the cluster. 2020-11-11 13:51:06: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:51:06: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:51:06: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:51:16: pid 11: LOG: trying again to join the cluster 2020-11-11 13:51:16: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:51:16: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:51:16: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:51:16: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:51:17: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:51:17: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:51:17: pid 11: HINT: rejoining the cluster. 2020-11-11 13:51:17: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:51:17: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:51:17: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:51:27: pid 11: LOG: trying again to join the cluster 2020-11-11 13:51:27: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:51:27: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:51:27: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:51:27: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:51:28: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:51:28: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:51:28: pid 11: HINT: rejoining the cluster. 2020-11-11 13:51:28: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:51:28: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:51:28: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:51:38: pid 11: LOG: trying again to join the cluster 2020-11-11 13:51:38: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:51:38: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:51:38: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:51:38: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:51:39: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:51:39: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:51:39: pid 11: HINT: rejoining the cluster. 2020-11-11 13:51:39: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:51:39: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:51:39: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:51:49: pid 11: LOG: trying again to join the cluster 2020-11-11 13:51:49: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:51:49: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:51:49: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:51:49: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:51:50: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:51:50: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:51:50: pid 11: HINT: rejoining the cluster. 2020-11-11 13:51:50: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:51:50: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:51:50: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:52:00: pid 11: LOG: trying again to join the cluster 2020-11-11 13:52:00: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:52:00: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:52:00: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:52:00: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:52:01: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:52:01: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:52:01: pid 11: HINT: rejoining the cluster. 2020-11-11 13:52:01: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:52:01: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:52:01: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:52:11: pid 11: LOG: trying again to join the cluster 2020-11-11 13:52:11: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:52:11: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:52:11: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:52:11: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:52:12: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:52:12: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:52:12: pid 11: HINT: rejoining the cluster. 2020-11-11 13:52:12: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:52:12: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:52:12: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:52:22: pid 11: LOG: trying again to join the cluster 2020-11-11 13:52:22: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:52:22: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:52:22: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:52:22: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:52:23: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:52:23: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:52:23: pid 11: HINT: rejoining the cluster. 2020-11-11 13:52:23: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:52:23: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:52:23: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:52:33: pid 11: LOG: trying again to join the cluster 2020-11-11 13:52:33: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:52:33: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:52:33: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:52:33: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:52:34: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:52:34: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:52:34: pid 11: HINT: rejoining the cluster. 2020-11-11 13:52:34: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:52:34: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:52:34: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:52:44: pid 11: LOG: trying again to join the cluster 2020-11-11 13:52:44: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:52:44: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:52:44: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:52:44: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:52:45: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:52:45: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:52:45: pid 11: HINT: rejoining the cluster. 2020-11-11 13:52:45: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:52:45: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:52:45: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:52:55: pid 11: LOG: trying again to join the cluster 2020-11-11 13:52:55: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:52:55: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:52:55: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:52:55: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:52:56: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:52:56: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:52:56: pid 11: HINT: rejoining the cluster. 2020-11-11 13:52:56: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:52:56: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:52:56: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:53:06: pid 11: LOG: trying again to join the cluster 2020-11-11 13:53:06: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:53:06: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:53:06: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:53:06: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:53:07: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:53:07: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:53:07: pid 11: HINT: rejoining the cluster. 2020-11-11 13:53:07: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:53:07: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:53:07: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:53:17: pid 11: LOG: trying again to join the cluster 2020-11-11 13:53:17: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:53:17: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:53:17: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:53:17: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:53:18: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:53:18: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:53:18: pid 11: HINT: rejoining the cluster. 2020-11-11 13:53:18: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:53:18: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:53:18: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:53:28: pid 11: LOG: trying again to join the cluster 2020-11-11 13:53:28: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:53:28: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:53:28: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:53:28: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:53:29: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:53:29: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:53:29: pid 11: HINT: rejoining the cluster. 2020-11-11 13:53:29: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:53:29: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:53:29: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:53:39: pid 11: LOG: trying again to join the cluster 2020-11-11 13:53:39: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:53:39: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:53:39: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:53:39: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:53:40: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:53:40: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:53:40: pid 11: HINT: rejoining the cluster. 2020-11-11 13:53:40: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:53:40: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:53:40: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:53:50: pid 11: LOG: trying again to join the cluster 2020-11-11 13:53:50: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:53:50: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:53:50: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:53:50: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:53:51: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:53:51: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:53:51: pid 11: HINT: rejoining the cluster. 2020-11-11 13:53:51: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:53:51: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:53:51: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:54:01: pid 11: LOG: trying again to join the cluster 2020-11-11 13:54:01: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:54:01: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:54:01: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:54:01: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:54:02: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:54:02: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:54:02: pid 11: HINT: rejoining the cluster. 2020-11-11 13:54:02: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:54:02: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:54:02: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:54:12: pid 11: LOG: trying again to join the cluster 2020-11-11 13:54:12: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:54:12: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:54:12: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:54:12: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:54:13: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:54:13: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:54:13: pid 11: HINT: rejoining the cluster. 2020-11-11 13:54:13: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:54:13: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:54:13: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:54:23: pid 11: LOG: trying again to join the cluster 2020-11-11 13:54:23: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:54:23: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:54:23: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:54:23: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:54:24: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:54:24: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:54:24: pid 11: HINT: rejoining the cluster. 2020-11-11 13:54:24: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:54:24: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:54:24: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:54:34: pid 11: LOG: trying again to join the cluster 2020-11-11 13:54:34: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:54:34: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:54:34: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:54:34: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:54:35: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:54:35: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:54:35: pid 11: HINT: rejoining the cluster. 2020-11-11 13:54:35: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:54:35: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:54:35: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:54:45: pid 11: LOG: trying again to join the cluster 2020-11-11 13:54:45: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:54:45: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:54:45: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:54:45: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:54:46: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:54:46: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:54:46: pid 11: HINT: rejoining the cluster. 2020-11-11 13:54:46: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:54:46: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:54:46: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:54:56: pid 11: LOG: trying again to join the cluster 2020-11-11 13:54:56: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:54:56: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:54:56: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:54:56: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:54:57: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:54:57: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:54:57: pid 11: HINT: rejoining the cluster. 2020-11-11 13:54:57: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:54:57: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:54:57: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:55:07: pid 11: LOG: trying again to join the cluster 2020-11-11 13:55:07: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:55:07: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:55:07: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:55:07: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:55:08: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:55:08: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:55:08: pid 11: HINT: rejoining the cluster. 2020-11-11 13:55:08: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:55:08: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:55:08: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:55:18: pid 11: LOG: trying again to join the cluster 2020-11-11 13:55:18: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:55:18: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:55:18: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:55:18: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:55:19: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:55:19: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:55:19: pid 11: HINT: rejoining the cluster. 2020-11-11 13:55:19: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:55:19: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:55:19: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:55:29: pid 11: LOG: trying again to join the cluster 2020-11-11 13:55:29: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:55:29: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:55:29: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:55:29: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:55:30: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:55:30: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:55:30: pid 11: HINT: rejoining the cluster. 2020-11-11 13:55:30: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:55:30: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:55:30: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:55:40: pid 11: LOG: trying again to join the cluster 2020-11-11 13:55:40: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:55:40: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:55:40: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:55:40: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:55:41: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:55:41: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:55:41: pid 11: HINT: rejoining the cluster. 2020-11-11 13:55:41: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:55:41: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:55:41: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:55:51: pid 11: LOG: trying again to join the cluster 2020-11-11 13:55:51: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:55:51: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:55:51: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:55:51: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:55:52: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:55:52: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:55:52: pid 11: HINT: rejoining the cluster. 2020-11-11 13:55:52: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:55:52: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:55:52: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:56:02: pid 11: LOG: trying again to join the cluster 2020-11-11 13:56:02: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:56:02: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:56:02: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:56:02: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:56:03: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:56:03: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:56:03: pid 11: HINT: rejoining the cluster. 2020-11-11 13:56:03: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:56:03: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:56:03: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:56:13: pid 11: LOG: trying again to join the cluster 2020-11-11 13:56:13: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:56:13: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:56:13: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:56:13: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:56:14: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:56:14: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:56:14: pid 11: HINT: rejoining the cluster. 2020-11-11 13:56:14: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:56:14: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:56:14: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:56:24: pid 11: LOG: trying again to join the cluster 2020-11-11 13:56:24: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:56:24: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:56:24: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:56:24: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:56:25: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:56:25: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:56:25: pid 11: HINT: rejoining the cluster. 2020-11-11 13:56:25: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:56:25: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:56:25: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:56:35: pid 11: LOG: trying again to join the cluster 2020-11-11 13:56:35: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:56:35: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:56:35: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:56:35: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:56:36: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:56:36: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:56:36: pid 11: HINT: rejoining the cluster. 2020-11-11 13:56:36: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:56:36: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:56:36: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:56:46: pid 11: LOG: trying again to join the cluster 2020-11-11 13:56:46: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:56:46: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:56:46: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:56:46: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:56:47: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:56:47: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:56:47: pid 11: HINT: rejoining the cluster. 2020-11-11 13:56:47: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:56:47: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:56:47: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:56:57: pid 11: LOG: trying again to join the cluster 2020-11-11 13:56:57: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:56:57: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:56:57: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:56:57: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:56:58: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:56:58: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:56:58: pid 11: HINT: rejoining the cluster. 2020-11-11 13:56:58: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:56:58: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:56:58: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:57:08: pid 11: LOG: trying again to join the cluster 2020-11-11 13:57:08: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:57:08: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:57:08: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:57:08: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:57:09: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:57:09: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:57:09: pid 11: HINT: rejoining the cluster. 2020-11-11 13:57:09: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:57:09: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:57:09: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:57:19: pid 11: LOG: trying again to join the cluster 2020-11-11 13:57:19: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:57:19: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:57:19: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:57:19: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:57:20: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:57:20: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:57:20: pid 11: HINT: rejoining the cluster. 2020-11-11 13:57:20: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:57:20: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:57:20: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:57:30: pid 11: LOG: trying again to join the cluster 2020-11-11 13:57:30: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:57:30: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:57:30: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:57:30: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:57:31: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:57:31: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:57:31: pid 11: HINT: rejoining the cluster. 2020-11-11 13:57:31: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:57:31: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:57:31: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:57:41: pid 11: LOG: trying again to join the cluster 2020-11-11 13:57:41: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:57:41: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:57:41: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:57:41: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:57:42: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:57:42: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:57:42: pid 11: HINT: rejoining the cluster. 2020-11-11 13:57:42: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:57:42: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:57:42: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:57:52: pid 11: LOG: trying again to join the cluster 2020-11-11 13:57:52: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:57:52: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:57:52: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:57:52: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:57:53: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:57:53: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:57:53: pid 11: HINT: rejoining the cluster. 2020-11-11 13:57:53: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:57:53: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:57:53: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:58:03: pid 11: LOG: trying again to join the cluster 2020-11-11 13:58:03: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:58:03: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:58:03: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:58:03: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:58:04: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:58:04: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:58:04: pid 11: HINT: rejoining the cluster. 2020-11-11 13:58:04: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:58:04: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:58:04: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:58:14: pid 11: LOG: trying again to join the cluster 2020-11-11 13:58:14: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:58:14: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:58:14: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:58:14: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:58:15: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:58:15: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:58:15: pid 11: HINT: rejoining the cluster. 2020-11-11 13:58:15: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:58:15: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:58:15: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:58:25: pid 11: LOG: trying again to join the cluster 2020-11-11 13:58:25: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:58:25: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:58:25: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:58:25: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:58:26: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:58:26: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:58:26: pid 11: HINT: rejoining the cluster. 2020-11-11 13:58:26: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:58:26: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:58:26: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:58:36: pid 11: LOG: trying again to join the cluster 2020-11-11 13:58:36: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:58:36: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:58:36: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:58:36: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:58:37: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:58:37: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:58:37: pid 11: HINT: rejoining the cluster. 2020-11-11 13:58:37: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:58:37: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:58:37: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:58:47: pid 11: LOG: trying again to join the cluster 2020-11-11 13:58:47: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:58:47: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:58:47: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:58:47: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:58:48: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:58:48: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:58:48: pid 11: HINT: rejoining the cluster. 2020-11-11 13:58:48: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:58:48: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:58:48: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:58:58: pid 11: LOG: trying again to join the cluster 2020-11-11 13:58:58: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:58:58: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:58:58: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:58:58: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:58:59: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:58:59: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:58:59: pid 11: HINT: rejoining the cluster. 2020-11-11 13:58:59: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:58:59: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:58:59: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:59:09: pid 11: LOG: trying again to join the cluster 2020-11-11 13:59:09: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:59:09: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:59:09: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:59:09: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:59:10: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:59:10: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:59:10: pid 11: HINT: rejoining the cluster. 2020-11-11 13:59:10: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:59:10: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:59:10: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:59:20: pid 11: LOG: trying again to join the cluster 2020-11-11 13:59:20: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:59:20: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:59:20: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:59:20: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:59:21: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:59:21: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:59:21: pid 11: HINT: rejoining the cluster. 2020-11-11 13:59:21: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:59:21: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:59:21: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:59:31: pid 11: LOG: trying again to join the cluster 2020-11-11 13:59:31: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:59:31: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:59:31: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:59:31: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:59:32: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:59:32: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:59:32: pid 11: HINT: rejoining the cluster. 2020-11-11 13:59:32: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:59:32: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:59:32: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:59:42: pid 11: LOG: trying again to join the cluster 2020-11-11 13:59:42: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:59:42: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:59:42: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:59:42: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:59:43: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:59:43: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:59:43: pid 11: HINT: rejoining the cluster. 2020-11-11 13:59:43: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:59:43: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:59:43: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 13:59:53: pid 11: LOG: trying again to join the cluster 2020-11-11 13:59:53: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 13:59:53: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 13:59:53: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 13:59:53: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 13:59:54: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 13:59:54: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 13:59:54: pid 11: HINT: rejoining the cluster. 2020-11-11 13:59:54: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 13:59:54: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 13:59:54: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:00:04: pid 11: LOG: trying again to join the cluster 2020-11-11 14:00:04: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:00:04: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:00:04: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:00:04: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:00:05: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:00:05: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:00:05: pid 11: HINT: rejoining the cluster. 2020-11-11 14:00:05: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:00:05: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:00:05: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:00:15: pid 11: LOG: trying again to join the cluster 2020-11-11 14:00:15: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:00:15: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:00:15: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:00:15: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:00:16: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:00:16: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:00:16: pid 11: HINT: rejoining the cluster. 2020-11-11 14:00:16: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:00:16: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:00:16: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:00:26: pid 11: LOG: trying again to join the cluster 2020-11-11 14:00:26: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:00:26: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:00:26: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:00:26: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:00:27: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:00:27: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:00:27: pid 11: HINT: rejoining the cluster. 2020-11-11 14:00:27: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:00:27: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:00:27: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:00:37: pid 11: LOG: trying again to join the cluster 2020-11-11 14:00:37: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:00:37: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:00:37: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:00:37: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:00:38: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:00:38: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:00:38: pid 11: HINT: rejoining the cluster. 2020-11-11 14:00:38: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:00:38: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:00:38: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:00:48: pid 11: LOG: trying again to join the cluster 2020-11-11 14:00:48: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:00:48: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:00:48: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:00:48: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:00:49: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:00:49: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:00:49: pid 11: HINT: rejoining the cluster. 2020-11-11 14:00:49: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:00:49: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:00:49: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:01:00: pid 11: LOG: trying again to join the cluster 2020-11-11 14:01:00: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:01:00: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:01:00: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:01:00: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:01:01: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:01:01: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:01:01: pid 11: HINT: rejoining the cluster. 2020-11-11 14:01:01: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:01:01: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:01:01: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:01:11: pid 11: LOG: trying again to join the cluster 2020-11-11 14:01:11: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:01:11: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:01:11: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:01:11: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:01:12: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:01:12: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:01:12: pid 11: HINT: rejoining the cluster. 2020-11-11 14:01:12: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:01:12: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:01:12: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:01:22: pid 11: LOG: trying again to join the cluster 2020-11-11 14:01:22: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:01:22: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:01:22: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:01:22: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:01:23: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:01:23: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:01:23: pid 11: HINT: rejoining the cluster. 2020-11-11 14:01:23: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:01:23: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:01:23: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:01:33: pid 11: LOG: trying again to join the cluster 2020-11-11 14:01:33: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:01:33: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:01:33: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:01:33: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:01:34: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:01:34: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:01:34: pid 11: HINT: rejoining the cluster. 2020-11-11 14:01:34: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:01:34: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:01:34: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:01:44: pid 11: LOG: trying again to join the cluster 2020-11-11 14:01:44: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:01:44: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:01:44: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:01:44: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:01:45: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:01:45: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:01:45: pid 11: HINT: rejoining the cluster. 2020-11-11 14:01:45: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:01:45: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:01:45: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:01:55: pid 11: LOG: trying again to join the cluster 2020-11-11 14:01:55: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:01:55: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:01:55: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:01:55: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:01:56: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:01:56: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:01:56: pid 11: HINT: rejoining the cluster. 2020-11-11 14:01:56: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:01:56: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:01:56: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:02:06: pid 11: LOG: trying again to join the cluster 2020-11-11 14:02:06: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:02:06: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:02:06: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:02:06: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:02:07: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:02:07: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:02:07: pid 11: HINT: rejoining the cluster. 2020-11-11 14:02:07: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:02:07: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:02:07: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:02:17: pid 11: LOG: trying again to join the cluster 2020-11-11 14:02:17: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:02:17: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:02:17: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:02:17: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:02:18: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:02:18: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:02:18: pid 11: HINT: rejoining the cluster. 2020-11-11 14:02:18: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:02:18: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:02:18: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:02:28: pid 11: LOG: trying again to join the cluster 2020-11-11 14:02:28: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:02:28: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:02:28: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:02:28: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:02:29: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:02:29: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:02:29: pid 11: HINT: rejoining the cluster. 2020-11-11 14:02:29: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:02:29: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:02:29: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:02:39: pid 11: LOG: trying again to join the cluster 2020-11-11 14:02:39: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:02:39: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:02:39: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:02:39: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:02:40: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:02:40: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:02:40: pid 11: HINT: rejoining the cluster. 2020-11-11 14:02:40: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:02:40: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:02:40: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:02:50: pid 11: LOG: trying again to join the cluster 2020-11-11 14:02:50: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:02:50: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:02:50: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:02:50: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:02:51: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:02:51: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:02:51: pid 11: HINT: rejoining the cluster. 2020-11-11 14:02:51: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:02:51: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:02:51: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:03:01: pid 11: LOG: trying again to join the cluster 2020-11-11 14:03:01: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:03:01: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:03:01: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:03:01: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:03:02: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:03:02: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:03:02: pid 11: HINT: rejoining the cluster. 2020-11-11 14:03:02: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:03:02: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:03:02: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:03:12: pid 11: LOG: trying again to join the cluster 2020-11-11 14:03:12: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:03:12: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:03:12: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:03:12: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:03:13: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:03:13: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:03:13: pid 11: HINT: rejoining the cluster. 2020-11-11 14:03:13: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:03:13: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:03:13: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:03:23: pid 11: LOG: trying again to join the cluster 2020-11-11 14:03:23: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:03:23: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:03:23: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:03:23: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:03:24: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:03:24: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:03:24: pid 11: HINT: rejoining the cluster. 2020-11-11 14:03:24: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:03:24: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:03:24: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:03:34: pid 11: LOG: trying again to join the cluster 2020-11-11 14:03:34: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:03:34: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:03:34: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:03:34: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:03:35: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:03:35: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:03:35: pid 11: HINT: rejoining the cluster. 2020-11-11 14:03:35: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:03:35: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:03:35: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:03:45: pid 11: LOG: trying again to join the cluster 2020-11-11 14:03:45: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:03:45: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:03:45: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:03:45: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:03:46: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:03:46: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:03:46: pid 11: HINT: rejoining the cluster. 2020-11-11 14:03:46: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:03:46: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:03:46: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:03:56: pid 11: LOG: trying again to join the cluster 2020-11-11 14:03:56: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:03:56: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:03:56: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:03:56: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:03:57: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:03:57: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:03:57: pid 11: HINT: rejoining the cluster. 2020-11-11 14:03:57: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:03:57: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:03:57: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:04:07: pid 11: LOG: trying again to join the cluster 2020-11-11 14:04:07: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:04:07: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:04:07: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:04:07: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:04:08: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:04:08: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:04:08: pid 11: HINT: rejoining the cluster. 2020-11-11 14:04:08: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:04:08: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:04:08: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:04:18: pid 11: LOG: trying again to join the cluster 2020-11-11 14:04:18: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:04:18: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:04:18: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:04:18: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:04:19: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:04:19: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:04:19: pid 11: HINT: rejoining the cluster. 2020-11-11 14:04:19: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:04:19: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:04:19: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:04:29: pid 11: LOG: trying again to join the cluster 2020-11-11 14:04:29: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:04:29: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:04:29: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:04:29: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:04:30: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:04:30: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:04:30: pid 11: HINT: rejoining the cluster. 2020-11-11 14:04:30: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:04:30: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:04:30: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:04:40: pid 11: LOG: trying again to join the cluster 2020-11-11 14:04:40: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:04:40: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:04:40: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:04:40: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:04:41: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:04:41: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:04:41: pid 11: HINT: rejoining the cluster. 2020-11-11 14:04:41: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:04:41: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:04:41: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:04:51: pid 11: LOG: trying again to join the cluster 2020-11-11 14:04:51: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:04:51: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:04:51: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:04:51: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:04:52: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:04:52: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:04:52: pid 11: HINT: rejoining the cluster. 2020-11-11 14:04:52: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:04:52: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:04:52: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:05:02: pid 11: LOG: trying again to join the cluster 2020-11-11 14:05:02: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:05:02: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:05:02: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:05:02: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:05:03: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:05:03: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:05:03: pid 11: HINT: rejoining the cluster. 2020-11-11 14:05:03: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:05:03: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:05:03: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:05:13: pid 11: LOG: trying again to join the cluster 2020-11-11 14:05:13: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:05:13: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:05:13: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:05:13: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:05:14: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:05:14: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:05:14: pid 11: HINT: rejoining the cluster. 2020-11-11 14:05:14: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:05:14: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:05:14: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:05:24: pid 11: LOG: trying again to join the cluster 2020-11-11 14:05:24: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:05:24: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:05:24: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:05:24: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:05:25: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:05:25: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:05:25: pid 11: HINT: rejoining the cluster. 2020-11-11 14:05:25: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:05:25: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:05:25: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:05:35: pid 11: LOG: trying again to join the cluster 2020-11-11 14:05:35: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:05:35: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:05:35: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:05:35: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:05:36: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:05:36: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:05:36: pid 11: HINT: rejoining the cluster. 2020-11-11 14:05:36: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:05:36: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:05:36: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:05:46: pid 11: LOG: trying again to join the cluster 2020-11-11 14:05:46: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:05:46: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:05:46: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:05:46: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:05:47: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:05:47: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:05:47: pid 11: HINT: rejoining the cluster. 2020-11-11 14:05:47: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:05:47: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:05:47: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:05:57: pid 11: LOG: trying again to join the cluster 2020-11-11 14:05:57: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:05:57: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:05:57: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:05:57: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:05:58: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:05:58: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:05:58: pid 11: HINT: rejoining the cluster. 2020-11-11 14:05:58: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:05:58: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:05:58: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:06:08: pid 11: LOG: trying again to join the cluster 2020-11-11 14:06:08: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:06:08: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:06:08: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:06:08: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:06:09: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:06:09: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:06:09: pid 11: HINT: rejoining the cluster. 2020-11-11 14:06:09: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:06:09: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:06:09: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:06:19: pid 11: LOG: trying again to join the cluster 2020-11-11 14:06:19: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:06:19: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:06:19: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:06:19: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:06:20: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:06:20: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:06:20: pid 11: HINT: rejoining the cluster. 2020-11-11 14:06:20: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:06:20: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:06:20: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:06:30: pid 11: LOG: trying again to join the cluster 2020-11-11 14:06:30: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:06:30: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:06:30: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:06:30: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:06:31: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:06:31: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:06:31: pid 11: HINT: rejoining the cluster. 2020-11-11 14:06:31: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:06:31: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:06:31: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:06:41: pid 11: LOG: trying again to join the cluster 2020-11-11 14:06:41: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:06:41: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:06:41: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:06:41: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:06:42: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:06:42: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:06:42: pid 11: HINT: rejoining the cluster. 2020-11-11 14:06:42: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:06:42: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:06:42: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:06:52: pid 11: LOG: trying again to join the cluster 2020-11-11 14:06:52: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:06:52: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:06:52: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:06:52: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:06:53: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:06:53: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:06:53: pid 11: HINT: rejoining the cluster. 2020-11-11 14:06:53: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:06:53: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:06:53: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:07:03: pid 11: LOG: trying again to join the cluster 2020-11-11 14:07:03: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:07:03: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:07:03: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:07:03: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:07:04: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:07:04: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:07:04: pid 11: HINT: rejoining the cluster. 2020-11-11 14:07:04: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:07:04: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:07:04: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:07:14: pid 11: LOG: trying again to join the cluster 2020-11-11 14:07:14: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:07:14: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:07:14: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:07:14: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:07:15: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:07:15: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:07:15: pid 11: HINT: rejoining the cluster. 2020-11-11 14:07:15: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:07:15: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:07:15: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:07:25: pid 11: LOG: trying again to join the cluster 2020-11-11 14:07:25: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:07:25: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:07:25: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:07:25: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:07:26: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:07:26: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:07:26: pid 11: HINT: rejoining the cluster. 2020-11-11 14:07:26: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:07:26: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:07:26: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:07:36: pid 11: LOG: trying again to join the cluster 2020-11-11 14:07:36: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:07:36: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:07:36: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:07:36: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:07:37: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:07:37: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:07:37: pid 11: HINT: rejoining the cluster. 2020-11-11 14:07:37: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:07:37: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:07:37: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:07:47: pid 11: LOG: trying again to join the cluster 2020-11-11 14:07:47: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:07:47: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:07:47: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:07:47: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:07:48: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:07:48: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:07:48: pid 11: HINT: rejoining the cluster. 2020-11-11 14:07:48: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:07:48: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:07:48: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:07:58: pid 11: LOG: trying again to join the cluster 2020-11-11 14:07:58: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:07:58: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:07:58: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:07:58: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:07:59: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:07:59: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:07:59: pid 11: HINT: rejoining the cluster. 2020-11-11 14:07:59: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:07:59: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:07:59: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:08:09: pid 11: LOG: trying again to join the cluster 2020-11-11 14:08:09: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:08:09: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:08:09: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:08:09: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:08:10: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:08:10: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:08:10: pid 11: HINT: rejoining the cluster. 2020-11-11 14:08:10: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:08:10: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:08:10: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:08:20: pid 11: LOG: trying again to join the cluster 2020-11-11 14:08:20: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:08:20: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:08:20: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:08:20: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:08:21: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:08:21: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:08:21: pid 11: HINT: rejoining the cluster. 2020-11-11 14:08:21: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:08:21: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:08:21: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:08:31: pid 11: LOG: trying again to join the cluster 2020-11-11 14:08:31: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:08:31: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:08:31: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:08:31: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:08:32: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:08:32: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:08:32: pid 11: HINT: rejoining the cluster. 2020-11-11 14:08:32: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:08:32: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:08:32: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:08:42: pid 11: LOG: trying again to join the cluster 2020-11-11 14:08:42: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:08:42: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:08:42: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:08:42: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:08:43: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:08:43: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:08:43: pid 11: HINT: rejoining the cluster. 2020-11-11 14:08:43: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:08:43: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:08:43: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:08:53: pid 11: LOG: trying again to join the cluster 2020-11-11 14:08:53: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:08:53: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:08:53: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:08:53: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:08:54: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:08:54: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:08:54: pid 11: HINT: rejoining the cluster. 2020-11-11 14:08:54: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:08:54: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:08:54: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:09:04: pid 11: LOG: trying again to join the cluster 2020-11-11 14:09:04: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:09:04: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:09:04: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:09:04: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:09:05: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:09:05: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:09:05: pid 11: HINT: rejoining the cluster. 2020-11-11 14:09:05: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:09:05: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:09:05: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:09:15: pid 11: LOG: trying again to join the cluster 2020-11-11 14:09:15: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:09:15: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:09:15: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:09:15: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:09:16: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:09:16: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:09:16: pid 11: HINT: rejoining the cluster. 2020-11-11 14:09:16: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:09:16: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:09:16: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:09:26: pid 11: LOG: trying again to join the cluster 2020-11-11 14:09:26: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:09:26: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:09:26: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:09:26: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:09:27: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:09:27: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:09:27: pid 11: HINT: rejoining the cluster. 2020-11-11 14:09:27: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:09:27: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:09:27: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:09:37: pid 11: LOG: trying again to join the cluster 2020-11-11 14:09:37: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:09:37: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:09:37: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:09:37: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:09:38: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:09:38: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:09:38: pid 11: HINT: rejoining the cluster. 2020-11-11 14:09:38: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:09:38: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:09:38: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:09:48: pid 11: LOG: trying again to join the cluster 2020-11-11 14:09:48: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:09:48: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:09:48: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:09:48: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:09:49: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:09:49: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:09:49: pid 11: HINT: rejoining the cluster. 2020-11-11 14:09:49: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:09:49: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:09:49: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:09:59: pid 11: LOG: trying again to join the cluster 2020-11-11 14:09:59: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:09:59: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:09:59: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:09:59: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:10:00: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:10:00: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:10:00: pid 11: HINT: rejoining the cluster. 2020-11-11 14:10:00: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:10:00: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:10:00: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:10:10: pid 11: LOG: trying again to join the cluster 2020-11-11 14:10:10: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:10:10: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:10:10: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:10:10: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:10:11: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:10:11: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:10:11: pid 11: HINT: rejoining the cluster. 2020-11-11 14:10:11: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:10:11: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:10:11: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:10:21: pid 11: LOG: trying again to join the cluster 2020-11-11 14:10:21: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:10:21: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:10:21: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:10:21: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:10:22: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:10:22: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:10:22: pid 11: HINT: rejoining the cluster. 2020-11-11 14:10:22: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:10:22: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:10:22: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:10:32: pid 11: LOG: trying again to join the cluster 2020-11-11 14:10:32: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:10:32: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:10:32: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:10:32: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:10:33: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:10:33: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:10:33: pid 11: HINT: rejoining the cluster. 2020-11-11 14:10:33: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:10:33: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:10:33: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:10:43: pid 11: LOG: trying again to join the cluster 2020-11-11 14:10:43: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:10:43: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:10:43: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:10:43: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:10:44: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:10:44: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:10:44: pid 11: HINT: rejoining the cluster. 2020-11-11 14:10:44: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:10:44: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:10:44: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:10:54: pid 11: LOG: trying again to join the cluster 2020-11-11 14:10:54: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:10:54: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:10:54: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:10:54: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:10:55: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:10:55: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:10:55: pid 11: HINT: rejoining the cluster. 2020-11-11 14:10:55: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:10:55: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:10:55: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:11:05: pid 11: LOG: trying again to join the cluster 2020-11-11 14:11:05: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:11:05: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:11:05: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:11:05: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:11:06: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:11:06: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:11:06: pid 11: HINT: rejoining the cluster. 2020-11-11 14:11:06: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:11:06: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:11:06: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:11:16: pid 11: LOG: trying again to join the cluster 2020-11-11 14:11:16: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:11:16: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:11:16: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:11:16: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:11:17: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:11:17: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:11:17: pid 11: HINT: rejoining the cluster. 2020-11-11 14:11:17: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:11:17: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:11:17: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:11:27: pid 11: LOG: trying again to join the cluster 2020-11-11 14:11:27: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:11:27: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:11:27: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:11:27: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:11:28: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:11:28: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:11:28: pid 11: HINT: rejoining the cluster. 2020-11-11 14:11:28: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:11:28: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:11:28: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:11:38: pid 11: LOG: trying again to join the cluster 2020-11-11 14:11:38: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:11:38: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:11:38: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:11:38: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:11:39: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:11:39: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:11:39: pid 11: HINT: rejoining the cluster. 2020-11-11 14:11:39: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:11:39: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:11:39: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:11:49: pid 11: LOG: trying again to join the cluster 2020-11-11 14:11:49: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:11:49: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:11:49: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:11:49: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:11:50: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:11:50: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:11:50: pid 11: HINT: rejoining the cluster. 2020-11-11 14:11:50: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:11:50: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:11:50: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:12:00: pid 11: LOG: trying again to join the cluster 2020-11-11 14:12:00: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:12:00: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:12:00: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:12:00: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:12:01: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:12:01: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:12:01: pid 11: HINT: rejoining the cluster. 2020-11-11 14:12:01: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:12:01: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:12:01: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:12:11: pid 11: LOG: trying again to join the cluster 2020-11-11 14:12:11: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:12:11: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:12:11: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:12:11: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:12:12: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:12:12: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:12:12: pid 11: HINT: rejoining the cluster. 2020-11-11 14:12:12: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:12:12: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:12:12: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:12:22: pid 11: LOG: trying again to join the cluster 2020-11-11 14:12:22: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:12:22: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:12:22: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:12:22: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:12:23: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:12:23: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:12:23: pid 11: HINT: rejoining the cluster. 2020-11-11 14:12:23: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:12:23: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:12:23: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:12:33: pid 11: LOG: trying again to join the cluster 2020-11-11 14:12:33: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:12:33: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:12:33: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:12:33: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:12:34: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:12:34: pid 11: NOTICE: our join coordinator is rejected by node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:12:34: pid 11: HINT: rejoining the cluster. 2020-11-11 14:12:34: pid 11: LOG: master node "172.29.30.2:5432 Linux 41a0ba327eeb" thinks we are lost, and "172.29.30.2:5432 Linux 41a0ba327eeb" is not letting us join 2020-11-11 14:12:34: pid 11: HINT: please verify the watchdog life-check and network is working properly 2020-11-11 14:12:34: pid 11: LOG: watchdog node state changed from [STANDBY] to [NETWORK ISOLATION] 2020-11-11 14:12:38: pid 11: LOG: read from socket failed, remote end closed the connection 2020-11-11 14:12:38: pid 11: LOG: outbound socket of 172.29.30.2:5432 Linux 41a0ba327eeb is closed 2020-11-11 14:12:38: pid 11: LOG: read from socket failed, remote end closed the connection 2020-11-11 14:12:38: pid 11: LOG: client socket of 172.29.30.2:5432 Linux 41a0ba327eeb is closed 2020-11-11 14:12:38: pid 11: LOG: remote node "172.29.30.2:5432 Linux 41a0ba327eeb" is not reachable 2020-11-11 14:12:38: pid 11: DETAIL: marking the node as lost 2020-11-11 14:12:38: pid 11: LOG: remote node "172.29.30.2:5432 Linux 41a0ba327eeb" is lost 2020-11-11 14:12:38: pid 11: LOG: watchdog cluster has lost the coordinator node 2020-11-11 14:12:38: pid 11: LOG: removing the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" from watchdog cluster master 2020-11-11 14:12:41: pid 11: LOG: new watchdog node connection is received from "172.29.30.2:6794" 2020-11-11 14:12:41: pid 11: LOG: new node joined the cluster hostname:"172.29.30.2" port:9009 pgpool_port:5432 2020-11-11 14:12:41: pid 11: DETAIL: Pgpool-II version:"4.1.4" watchdog messaging version: 1.1 2020-11-11 14:12:41: pid 11: LOG: The newly joined node:"172.29.30.2:5432 Linux 41a0ba327eeb" had left the cluster because it was lost 2020-11-11 14:12:41: pid 11: DETAIL: lost reason was "NOT REACHABLE" and startup time diff = 11103 2020-11-11 14:12:45: pid 11: LOG: trying again to join the cluster 2020-11-11 14:12:45: pid 11: LOG: watchdog node state changed from [NETWORK ISOLATION] to [JOINING] 2020-11-11 14:12:45: pid 11: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2020-11-11 14:12:46: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [PARTICIPATING IN ELECTION] 2020-11-11 14:12:46: pid 11: LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [INITIALIZING] 2020-11-11 14:12:46: pid 11: LOG: setting the remote node "172.29.30.2:5432 Linux 41a0ba327eeb" as watchdog cluster master 2020-11-11 14:12:47: pid 11: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2020-11-11 14:12:47: pid 11: LOG: successfully joined the watchdog cluster as standby node 2020-11-11 14:12:47: pid 11: DETAIL: our join coordinator request is accepted by cluster leader node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:12:47: pid 1: LOG: watchdog process is initialized 2020-11-11 14:12:47: pid 1: DETAIL: watchdog messaging data version: 1.1 2020-11-11 14:12:47: pid 1: LOG: Pgpool-II parent process received watchdog quorum change signal from watchdog 2020-11-11 14:12:47: pid 11: LOG: new IPC connection received 2020-11-11 14:12:47: pid 11: LOG: new IPC connection received 2020-11-11 14:12:47: pid 1: LOG: watchdog cluster now holds the quorum 2020-11-11 14:12:47: pid 1: DETAIL: updating the state of quarantine backend nodes 2020-11-11 14:12:47: pid 13: LOG: 3 watchdog nodes are configured for lifecheck 2020-11-11 14:12:47: pid 13: LOG: watchdog nodes ID:0 Name:"172.29.30.3:5432 Linux 6497bd770d79" 2020-11-11 14:12:47: pid 13: DETAIL: Host:"172.29.30.3" WD Port:9009 pgpool-II port:5432 2020-11-11 14:12:47: pid 13: LOG: watchdog nodes ID:1 Name:"172.29.30.1:5432 Linux df4d4f6cdb3f" 2020-11-11 14:12:47: pid 13: DETAIL: Host:"172.29.30.1" WD Port:9009 pgpool-II port:5432 2020-11-11 14:12:47: pid 13: LOG: watchdog nodes ID:2 Name:"172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:12:47: pid 13: DETAIL: Host:"172.29.30.2" WD Port:9009 pgpool-II port:5432 2020-11-11 14:12:47: pid 11: LOG: new IPC connection received 2020-11-11 14:12:47: pid 11: LOG: new IPC connection received 2020-11-11 14:12:47: pid 1: LOG: we have joined the watchdog cluster as STANDBY node 2020-11-11 14:12:47: pid 1: DETAIL: syncing the backend states from the MASTER watchdog node 2020-11-11 14:12:47: pid 11: LOG: new IPC connection received 2020-11-11 14:12:47: pid 11: LOG: received the get data request from local pgpool-II on IPC interface 2020-11-11 14:12:47: pid 11: LOG: get data request from local pgpool-II node received on IPC interface is forwarded to master watchdog node "172.29.30.2:5432 Linux 41a0ba327eeb" 2020-11-11 14:12:47: pid 11: DETAIL: waiting for the reply... 2020-11-11 14:12:47: pid 1: LOG: master watchdog node "172.29.30.2:5432 Linux 41a0ba327eeb" returned status for 3 backend nodes 2020-11-11 14:12:47: pid 1: LOG: Setting up socket for 0.0.0.0:5432 2020-11-11 14:12:47: pid 1: LOG: Setting up socket for :::5432 2020-11-11 14:12:47: pid 50: LOG: PCP process: 50 started 2020-11-11 14:12:47: pid 1: LOG: pgpool-II successfully started. version 4.1.4 (karasukiboshi) 2020-11-11 14:12:47: pid 1: LOG: node status[0]: 0 2020-11-11 14:12:47: pid 1: LOG: node status[1]: 0 2020-11-11 14:12:47: pid 1: LOG: node status[2]: 0 2020-11-11 14:12:48: pid 14: LOG: set SO_REUSEPORT option to the socket 2020-11-11 14:12:48: pid 14: LOG: creating watchdog heartbeat receive socket. 2020-11-11 14:12:48: pid 14: DETAIL: set SO_REUSEPORT 2020-11-11 14:12:48: pid 15: LOG: set SO_REUSEPORT option to the socket 2020-11-11 14:12:48: pid 15: LOG: creating socket for sending heartbeat 2020-11-11 14:12:48: pid 15: DETAIL: set SO_REUSEPORT 2020-11-11 14:12:48: pid 16: LOG: set SO_REUSEPORT option to the socket 2020-11-11 14:12:48: pid 16: LOG: creating watchdog heartbeat receive socket. 2020-11-11 14:12:48: pid 16: DETAIL: set SO_REUSEPORT 2020-11-11 14:12:48: pid 17: LOG: set SO_REUSEPORT option to the socket 2020-11-11 14:12:48: pid 17: LOG: creating socket for sending heartbeat 2020-11-11 14:12:48: pid 17: DETAIL: set SO_REUSEPORT 2020-11-11 14:12:49: pid 11: LOG: new outbound connection to 172.29.30.2:9009 2020-11-11 14:12:51: pid 11: LOG: remote node "172.29.30.1:5432 Linux df4d4f6cdb3f" is reporting that it has found us again 2020-11-11 14:14:27: pid 13: LOG: watchdog: lifecheck started 2020-11-11 14:15:00: pid 44: LOG: new connection received 2020-11-11 14:15:00: pid 44: DETAIL: connecting host=tkh-app.tkh_network port=45594 2020-11-11 14:15:00: pid 44: LOG: pool_reuse_block: blockid: 0 2020-11-11 14:15:00: pid 44: CONTEXT: while searching system catalog, When relcache is missed |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2020-11-11 22:52 | papegaaij | New Issue | |
| 2020-11-11 22:52 | papegaaij | File Added: 172.29.30.2.log | |
| 2020-11-11 22:52 | papegaaij | File Added: 172.29.30.3.log | |
| 2020-11-11 22:52 | papegaaij | File Added: pgpool.conf | |
| 2020-11-11 23:16 | papegaaij | Note Added: 0003596 | |
| 2020-11-12 13:41 | pengbo | Assigned To | => pengbo |
| 2020-11-12 13:41 | pengbo | Status | new => assigned |
| 2020-11-12 13:41 | pengbo | Description Updated | |
| 2020-12-01 20:00 | papegaaij | Note Added: 0003628 | |
| 2020-12-02 15:18 | pengbo | Note Added: 0003632 | |
| 2020-12-02 15:19 | pengbo | Status | assigned => feedback |
| 2020-12-02 15:28 | papegaaij | File Added: 172.29.30.2-2.log | |
| 2020-12-02 15:28 | papegaaij | File Added: 172.29.30.3-2.log | |
| 2020-12-02 15:28 | papegaaij | File Added: pgpool-2.conf | |
| 2020-12-02 15:28 | papegaaij | Note Added: 0003633 | |
| 2020-12-02 15:28 | papegaaij | Status | feedback => assigned |
