Dec 2 01:51:56 postgredbprod01 pgpool[21449]: [1-1] 2020-12-02 01:51:56: pid 21449: LOG: memory cache initialized Dec 2 01:51:56 postgredbprod01 pgpool[21449]: [1-2] 2020-12-02 01:51:56: pid 21449: DETAIL: memcache blocks :64 Dec 2 01:51:56 postgredbprod01 pgpool[21449]: [2-1] 2020-12-02 01:51:56: pid 21449: LOG: pool_discard_oid_maps: discarded memqcache oid maps Dec 2 01:51:56 postgredbprod01 pgpool[21449]: [3-1] 2020-12-02 01:51:56: pid 21449: LOG: waiting for watchdog to initialize Dec 2 01:51:56 postgredbprod01 pgpool[21451]: [3-1] 2020-12-02 01:51:56: pid 21451: LOG: setting the local watchdog node name to "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 01:51:56 postgredbprod01 pgpool[21451]: [4-1] 2020-12-02 01:51:56: pid 21451: LOG: watchdog cluster is configured with 1 remote nodes Dec 2 01:51:56 postgredbprod01 pgpool[21451]: [5-1] 2020-12-02 01:51:56: pid 21451: LOG: watchdog remote node:0 on postgredbprod02:9000 Dec 2 01:51:56 postgredbprod01 pgpool[21451]: [6-1] 2020-12-02 01:51:56: pid 21451: LOG: interface monitoring is disabled in watchdog Dec 2 01:51:56 postgredbprod01 pgpool[21451]: [7-1] 2020-12-02 01:51:56: pid 21451: LOG: watchdog node state changed from [DEAD] to [LOADING] Dec 2 01:52:01 postgredbprod01 pgpool[21451]: [8-1] 2020-12-02 01:52:01: pid 21451: LOG: watchdog node state changed from [LOADING] to [JOINING] Dec 2 01:52:05 postgredbprod01 pgpool[21451]: [9-1] 2020-12-02 01:52:05: pid 21451: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Dec 2 01:52:06 postgredbprod01 pgpool[21451]: [10-1] 2020-12-02 01:52:06: pid 21451: LOG: I am the only alive node in the watchdog cluster Dec 2 01:52:06 postgredbprod01 pgpool[21451]: [10-2] 2020-12-02 01:52:06: pid 21451: HINT: skipping stand for coordinator state Dec 2 01:52:06 postgredbprod01 pgpool[21451]: [11-1] 2020-12-02 01:52:06: pid 21451: LOG: watchdog node state changed from [INITIALIZING] to [MASTER] Dec 2 01:52:06 postgredbprod01 pgpool[21451]: [12-1] 2020-12-02 01:52:06: pid 21451: LOG: I am announcing my self as master/coordinator watchdog node Dec 2 01:52:10 postgredbprod01 pgpool[21451]: [13-1] 2020-12-02 01:52:10: pid 21451: LOG: I am the cluster leader node Dec 2 01:52:10 postgredbprod01 pgpool[21451]: [13-2] 2020-12-02 01:52:10: pid 21451: DETAIL: our declare coordinator message is accepted by all nodes Dec 2 01:52:10 postgredbprod01 pgpool[21451]: [14-1] 2020-12-02 01:52:10: pid 21451: LOG: setting the local node "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" as watchdog cluster master Dec 2 01:52:10 postgredbprod01 pgpool[21451]: [15-1] 2020-12-02 01:52:10: pid 21451: LOG: I am the cluster leader node but we do not have enough nodes in cluster Dec 2 01:52:10 postgredbprod01 pgpool[21451]: [15-2] 2020-12-02 01:52:10: pid 21451: DETAIL: waiting for the quorum to start escalation process Dec 2 01:52:10 postgredbprod01 pgpool[21449]: [4-1] 2020-12-02 01:52:10: pid 21449: LOG: watchdog process is initialized Dec 2 01:52:10 postgredbprod01 pgpool[21449]: [4-2] 2020-12-02 01:52:10: pid 21449: DETAIL: watchdog messaging data version: 1.1 Dec 2 01:52:10 postgredbprod01 pgpool[21451]: [16-1] 2020-12-02 01:52:10: pid 21451: LOG: new IPC connection received Dec 2 01:52:10 postgredbprod01 pgpool[21451]: [17-1] 2020-12-02 01:52:10: pid 21451: LOG: new IPC connection received Dec 2 01:52:10 postgredbprod01 pgpool[21463]: [5-1] 2020-12-02 01:52:10: pid 21463: LOG: 2 watchdog nodes are configured for lifecheck Dec 2 01:52:10 postgredbprod01 pgpool[21449]: [5-1] 2020-12-02 01:52:10: pid 21449: LOG: Setting up socket for 0.0.0.0:9999 Dec 2 01:52:10 postgredbprod01 pgpool[21463]: [6-1] 2020-12-02 01:52:10: pid 21463: LOG: watchdog nodes ID:0 Name:"postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 01:52:10 postgredbprod01 pgpool[21463]: [6-2] 2020-12-02 01:52:10: pid 21463: DETAIL: Host:"postgredbprod01" WD Port:9000 pgpool-II port:9999 Dec 2 01:52:10 postgredbprod01 pgpool[21449]: [6-1] 2020-12-02 01:52:10: pid 21449: LOG: Setting up socket for :::9999 Dec 2 01:52:10 postgredbprod01 pgpool[21463]: [7-1] 2020-12-02 01:52:10: pid 21463: LOG: watchdog nodes ID:1 Name:"Not_Set" Dec 2 01:52:10 postgredbprod01 pgpool[21463]: [7-2] 2020-12-02 01:52:10: pid 21463: DETAIL: Host:"postgredbprod02" WD Port:9000 pgpool-II port:9999 Dec 2 01:52:10 postgredbprod01 pgpool[21463]: [8-1] 2020-12-02 01:52:10: pid 21463: LOG: watchdog lifecheck trusted server "postgredbprod02" added for the availability check Dec 2 01:52:11 postgredbprod01 pgpool[21449]: [7-1] 2020-12-02 01:52:11: pid 21449: LOG: find_primary_node_repeatedly: waiting for finding a primary node Dec 2 01:52:11 postgredbprod01 pgpool[21449]: [8-1] 2020-12-02 01:52:11: pid 21449: LOG: find_primary_node: primary node is 0 Dec 2 01:52:11 postgredbprod01 pgpool[24467]: [9-1] 2020-12-02 01:52:11: pid 24467: LOG: PCP process: 24467 started Dec 2 01:52:11 postgredbprod01 pgpool[21449]: [9-1] 2020-12-02 01:52:11: pid 21449: LOG: pgpool-II successfully started. version 4.1.5 (karasukiboshi) Dec 2 01:52:11 postgredbprod01 pgpool[21449]: [10-1] 2020-12-02 01:52:11: pid 21449: LOG: node status[0]: 1 Dec 2 01:52:11 postgredbprod01 pgpool[21449]: [11-1] 2020-12-02 01:52:11: pid 21449: LOG: node status[1]: 0 Dec 2 01:52:11 postgredbprod01 pgpool[21465]: [8-1] 2020-12-02 01:52:11: pid 21465: LOG: failed to create watchdog heartbeat receive socket. Dec 2 01:52:11 postgredbprod01 pgpool[21465]: [8-2] 2020-12-02 01:52:11: pid 21465: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege Dec 2 01:52:11 postgredbprod01 pgpool[21465]: [9-1] 2020-12-02 01:52:11: pid 21465: LOG: set SO_REUSEPORT option to the socket Dec 2 01:52:11 postgredbprod01 pgpool[21465]: [10-1] 2020-12-02 01:52:11: pid 21465: LOG: creating watchdog heartbeat receive socket. Dec 2 01:52:11 postgredbprod01 pgpool[21465]: [10-2] 2020-12-02 01:52:11: pid 21465: DETAIL: set SO_REUSEPORT Dec 2 01:52:11 postgredbprod01 pgpool[21466]: [8-1] 2020-12-02 01:52:11: pid 21466: LOG: creating socket for sending heartbeat Dec 2 01:52:11 postgredbprod01 pgpool[21466]: [8-2] 2020-12-02 01:52:11: pid 21466: DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege Dec 2 01:52:11 postgredbprod01 pgpool[21466]: [9-1] 2020-12-02 01:52:11: pid 21466: LOG: set SO_REUSEPORT option to the socket Dec 2 01:52:11 postgredbprod01 pgpool[21466]: [10-1] 2020-12-02 01:52:11: pid 21466: LOG: creating socket for sending heartbeat Dec 2 01:52:11 postgredbprod01 pgpool[21466]: [10-2] 2020-12-02 01:52:11: pid 21466: DETAIL: set SO_REUSEPORT Dec 2 01:52:49 postgredbprod01 pgpool[24514]: [1-1] 2020-12-02 01:52:49: pid 24514: LOG: stop request sent to pgpool. waiting for termination... Dec 2 01:52:49 postgredbprod01 pgpool[21451]: [18-1] 2020-12-02 01:52:49: pid 21451: LOG: Watchdog is shutting down Dec 2 02:13:08 postgredbprod01 pgpool[25291]: [1-1] 2020-12-02 02:13:08: pid 25291: LOG: memory cache initialized Dec 2 02:13:08 postgredbprod01 pgpool[25291]: [1-2] 2020-12-02 02:13:08: pid 25291: DETAIL: memcache blocks :64 Dec 2 02:13:08 postgredbprod01 pgpool[25291]: [2-1] 2020-12-02 02:13:08: pid 25291: LOG: pool_discard_oid_maps: discarded memqcache oid maps Dec 2 02:13:08 postgredbprod01 pgpool[25291]: [3-1] 2020-12-02 02:13:08: pid 25291: LOG: waiting for watchdog to initialize Dec 2 02:13:08 postgredbprod01 pgpool[25293]: [3-1] 2020-12-02 02:13:08: pid 25293: LOG: setting the local watchdog node name to "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 02:13:08 postgredbprod01 pgpool[25293]: [4-1] 2020-12-02 02:13:08: pid 25293: LOG: watchdog cluster is configured with 1 remote nodes Dec 2 02:13:08 postgredbprod01 pgpool[25293]: [5-1] 2020-12-02 02:13:08: pid 25293: LOG: watchdog remote node:0 on postgredbprod02:9000 Dec 2 02:13:08 postgredbprod01 pgpool[25293]: [6-1] 2020-12-02 02:13:08: pid 25293: LOG: interface monitoring is disabled in watchdog Dec 2 02:13:08 postgredbprod01 pgpool[25293]: [7-1] 2020-12-02 02:13:08: pid 25293: LOG: watchdog node state changed from [DEAD] to [LOADING] Dec 2 02:13:14 postgredbprod01 pgpool[25293]: [8-1] 2020-12-02 02:13:14: pid 25293: LOG: watchdog node state changed from [LOADING] to [JOINING] Dec 2 02:13:18 postgredbprod01 pgpool[25293]: [9-1] 2020-12-02 02:13:18: pid 25293: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Dec 2 02:13:19 postgredbprod01 pgpool[25293]: [10-1] 2020-12-02 02:13:19: pid 25293: LOG: I am the only alive node in the watchdog cluster Dec 2 02:13:19 postgredbprod01 pgpool[25293]: [10-2] 2020-12-02 02:13:19: pid 25293: HINT: skipping stand for coordinator state Dec 2 02:13:19 postgredbprod01 pgpool[25293]: [11-1] 2020-12-02 02:13:19: pid 25293: LOG: watchdog node state changed from [INITIALIZING] to [MASTER] Dec 2 02:13:19 postgredbprod01 pgpool[25293]: [12-1] 2020-12-02 02:13:19: pid 25293: LOG: I am announcing my self as master/coordinator watchdog node Dec 2 02:13:23 postgredbprod01 pgpool[25293]: [13-1] 2020-12-02 02:13:23: pid 25293: LOG: I am the cluster leader node Dec 2 02:13:23 postgredbprod01 pgpool[25293]: [13-2] 2020-12-02 02:13:23: pid 25293: DETAIL: our declare coordinator message is accepted by all nodes Dec 2 02:13:23 postgredbprod01 pgpool[25293]: [14-1] 2020-12-02 02:13:23: pid 25293: LOG: setting the local node "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" as watchdog cluster master Dec 2 02:13:23 postgredbprod01 pgpool[25293]: [15-1] 2020-12-02 02:13:23: pid 25293: LOG: I am the cluster leader node but we do not have enough nodes in cluster Dec 2 02:13:23 postgredbprod01 pgpool[25293]: [15-2] 2020-12-02 02:13:23: pid 25293: DETAIL: waiting for the quorum to start escalation process Dec 2 02:13:23 postgredbprod01 pgpool[25291]: [4-1] 2020-12-02 02:13:23: pid 25291: LOG: watchdog process is initialized Dec 2 02:13:23 postgredbprod01 pgpool[25291]: [4-2] 2020-12-02 02:13:23: pid 25291: DETAIL: watchdog messaging data version: 1.1 Dec 2 02:13:23 postgredbprod01 pgpool[25293]: [16-1] 2020-12-02 02:13:23: pid 25293: LOG: new IPC connection received Dec 2 02:13:23 postgredbprod01 pgpool[25293]: [17-1] 2020-12-02 02:13:23: pid 25293: LOG: new IPC connection received Dec 2 02:13:23 postgredbprod01 pgpool[25291]: [5-1] 2020-12-02 02:13:23: pid 25291: LOG: Setting up socket for 0.0.0.0:9999 Dec 2 02:13:23 postgredbprod01 pgpool[25291]: [6-1] 2020-12-02 02:13:23: pid 25291: LOG: Setting up socket for :::9999 Dec 2 02:13:23 postgredbprod01 pgpool[25303]: [5-1] 2020-12-02 02:13:23: pid 25303: LOG: 2 watchdog nodes are configured for lifecheck Dec 2 02:13:23 postgredbprod01 pgpool[25303]: [6-1] 2020-12-02 02:13:23: pid 25303: LOG: watchdog nodes ID:0 Name:"postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 02:13:23 postgredbprod01 pgpool[25303]: [6-2] 2020-12-02 02:13:23: pid 25303: DETAIL: Host:"postgredbprod01" WD Port:9000 pgpool-II port:9999 Dec 2 02:13:23 postgredbprod01 pgpool[25303]: [7-1] 2020-12-02 02:13:23: pid 25303: LOG: watchdog nodes ID:1 Name:"Not_Set" Dec 2 02:13:23 postgredbprod01 pgpool[25303]: [7-2] 2020-12-02 02:13:23: pid 25303: DETAIL: Host:"postgredbprod02" WD Port:9000 pgpool-II port:9999 Dec 2 02:13:23 postgredbprod01 pgpool[25303]: [8-1] 2020-12-02 02:13:23: pid 25303: LOG: watchdog lifecheck trusted server "postgredbprod02" added for the availability check Dec 2 02:13:23 postgredbprod01 pgpool[25291]: [7-1] 2020-12-02 02:13:23: pid 25291: LOG: find_primary_node_repeatedly: waiting for finding a primary node Dec 2 02:13:23 postgredbprod01 pgpool[25291]: [8-1] 2020-12-02 02:13:23: pid 25291: LOG: find_primary_node: primary node is 0 Dec 2 02:13:23 postgredbprod01 pgpool[28308]: [9-1] 2020-12-02 02:13:23: pid 28308: LOG: PCP process: 28308 started Dec 2 02:13:23 postgredbprod01 pgpool[25291]: [9-1] 2020-12-02 02:13:23: pid 25291: LOG: pgpool-II successfully started. version 4.1.5 (karasukiboshi) Dec 2 02:13:23 postgredbprod01 pgpool[25291]: [10-1] 2020-12-02 02:13:23: pid 25291: LOG: node status[0]: 1 Dec 2 02:13:23 postgredbprod01 pgpool[25291]: [11-1] 2020-12-02 02:13:23: pid 25291: LOG: node status[1]: 0 Dec 2 02:13:24 postgredbprod01 pgpool[25305]: [8-1] 2020-12-02 02:13:24: pid 25305: LOG: failed to create watchdog heartbeat receive socket. Dec 2 02:13:24 postgredbprod01 pgpool[25305]: [8-2] 2020-12-02 02:13:24: pid 25305: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege Dec 2 02:13:24 postgredbprod01 pgpool[25305]: [9-1] 2020-12-02 02:13:24: pid 25305: LOG: set SO_REUSEPORT option to the socket Dec 2 02:13:24 postgredbprod01 pgpool[25305]: [10-1] 2020-12-02 02:13:24: pid 25305: LOG: creating watchdog heartbeat receive socket. Dec 2 02:13:24 postgredbprod01 pgpool[25305]: [10-2] 2020-12-02 02:13:24: pid 25305: DETAIL: set SO_REUSEPORT Dec 2 02:13:24 postgredbprod01 pgpool[25307]: [8-1] 2020-12-02 02:13:24: pid 25307: LOG: creating socket for sending heartbeat Dec 2 02:13:24 postgredbprod01 pgpool[25307]: [8-2] 2020-12-02 02:13:24: pid 25307: DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege Dec 2 02:13:24 postgredbprod01 pgpool[25307]: [9-1] 2020-12-02 02:13:24: pid 25307: LOG: set SO_REUSEPORT option to the socket Dec 2 02:13:24 postgredbprod01 pgpool[25307]: [10-1] 2020-12-02 02:13:24: pid 25307: LOG: creating socket for sending heartbeat Dec 2 02:13:24 postgredbprod01 pgpool[25307]: [10-2] 2020-12-02 02:13:24: pid 25307: DETAIL: set SO_REUSEPORT Dec 2 02:18:25 postgredbprod01 pgpool[28308]: [10-1] 2020-12-02 02:18:25: pid 28308: LOG: forked new pcp worker, pid=28590 socket=8 Dec 2 02:18:25 postgredbprod01 pgpool[25293]: [18-1] 2020-12-02 02:18:25: pid 25293: LOG: new IPC connection received Dec 2 02:18:25 postgredbprod01 pgpool[28308]: [11-1] 2020-12-02 02:18:25: pid 28308: LOG: PCP process with pid: 28590 exit with SUCCESS. Dec 2 02:18:25 postgredbprod01 pgpool[28308]: [12-1] 2020-12-02 02:18:25: pid 28308: LOG: PCP process with pid: 28590 exits with status 0 Dec 2 02:18:45 postgredbprod01 pgpool[28308]: [13-1] 2020-12-02 02:18:45: pid 28308: LOG: forked new pcp worker, pid=28611 socket=8 Dec 2 02:18:45 postgredbprod01 pgpool[25293]: [19-1] 2020-12-02 02:18:45: pid 25293: LOG: new IPC connection received Dec 2 02:18:45 postgredbprod01 pgpool[28308]: [14-1] 2020-12-02 02:18:45: pid 28308: LOG: PCP process with pid: 28611 exit with SUCCESS. Dec 2 02:18:45 postgredbprod01 pgpool[28308]: [15-1] 2020-12-02 02:18:45: pid 28308: LOG: PCP process with pid: 28611 exits with status 0 Dec 2 02:20:14 postgredbprod01 pgpool[28308]: [16-1] 2020-12-02 02:20:14: pid 28308: LOG: forked new pcp worker, pid=28693 socket=8 Dec 2 02:20:14 postgredbprod01 pgpool[25293]: [20-1] 2020-12-02 02:20:14: pid 25293: LOG: new IPC connection received Dec 2 02:20:14 postgredbprod01 pgpool[28308]: [17-1] 2020-12-02 02:20:14: pid 28308: LOG: PCP process with pid: 28693 exit with SUCCESS. Dec 2 02:20:14 postgredbprod01 pgpool[28308]: [18-1] 2020-12-02 02:20:14: pid 28308: LOG: PCP process with pid: 28693 exits with status 0 Dec 2 02:27:09 postgredbprod01 pgpool[29092]: [1-1] 2020-12-02 02:27:09: pid 29092: LOG: stop request sent to pgpool. waiting for termination... Dec 2 02:27:09 postgredbprod01 pgpool[25293]: [21-1] 2020-12-02 02:27:09: pid 25293: LOG: Watchdog is shutting down Dec 2 02:41:19 postgredbprod01 pgpool[29910]: [1-1] 2020-12-02 02:41:19: pid 29910: LOG: memory cache initialized Dec 2 02:41:19 postgredbprod01 pgpool[29910]: [1-2] 2020-12-02 02:41:19: pid 29910: DETAIL: memcache blocks :64 Dec 2 02:41:19 postgredbprod01 pgpool[29910]: [2-1] 2020-12-02 02:41:19: pid 29910: LOG: pool_discard_oid_maps: discarded memqcache oid maps Dec 2 02:41:19 postgredbprod01 pgpool[29910]: [3-1] 2020-12-02 02:41:19: pid 29910: LOG: waiting for watchdog to initialize Dec 2 02:41:19 postgredbprod01 pgpool[29912]: [3-1] 2020-12-02 02:41:19: pid 29912: LOG: setting the local watchdog node name to "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 02:41:19 postgredbprod01 pgpool[29912]: [4-1] 2020-12-02 02:41:19: pid 29912: LOG: watchdog cluster is configured with 1 remote nodes Dec 2 02:41:19 postgredbprod01 pgpool[29912]: [5-1] 2020-12-02 02:41:19: pid 29912: LOG: watchdog remote node:0 on postgredbprod02:9000 Dec 2 02:41:19 postgredbprod01 pgpool[29912]: [6-1] 2020-12-02 02:41:19: pid 29912: LOG: interface monitoring is disabled in watchdog Dec 2 02:41:19 postgredbprod01 pgpool[29912]: [7-1] 2020-12-02 02:41:19: pid 29912: LOG: watchdog node state changed from [DEAD] to [LOADING] Dec 2 02:41:24 postgredbprod01 pgpool[29912]: [8-1] 2020-12-02 02:41:24: pid 29912: LOG: watchdog node state changed from [LOADING] to [JOINING] Dec 2 02:41:28 postgredbprod01 pgpool[29912]: [9-1] 2020-12-02 02:41:28: pid 29912: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Dec 2 02:41:29 postgredbprod01 pgpool[29912]: [10-1] 2020-12-02 02:41:29: pid 29912: LOG: I am the only alive node in the watchdog cluster Dec 2 02:41:29 postgredbprod01 pgpool[29912]: [10-2] 2020-12-02 02:41:29: pid 29912: HINT: skipping stand for coordinator state Dec 2 02:41:29 postgredbprod01 pgpool[29912]: [11-1] 2020-12-02 02:41:29: pid 29912: LOG: watchdog node state changed from [INITIALIZING] to [MASTER] Dec 2 02:41:29 postgredbprod01 pgpool[29912]: [12-1] 2020-12-02 02:41:29: pid 29912: LOG: I am announcing my self as master/coordinator watchdog node Dec 2 02:41:33 postgredbprod01 pgpool[29912]: [13-1] 2020-12-02 02:41:33: pid 29912: LOG: I am the cluster leader node Dec 2 02:41:33 postgredbprod01 pgpool[29912]: [13-2] 2020-12-02 02:41:33: pid 29912: DETAIL: our declare coordinator message is accepted by all nodes Dec 2 02:41:33 postgredbprod01 pgpool[29912]: [14-1] 2020-12-02 02:41:33: pid 29912: LOG: setting the local node "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" as watchdog cluster master Dec 2 02:41:33 postgredbprod01 pgpool[29912]: [15-1] 2020-12-02 02:41:33: pid 29912: LOG: I am the cluster leader node but we do not have enough nodes in cluster Dec 2 02:41:33 postgredbprod01 pgpool[29912]: [15-2] 2020-12-02 02:41:33: pid 29912: DETAIL: waiting for the quorum to start escalation process Dec 2 02:41:33 postgredbprod01 pgpool[29910]: [4-1] 2020-12-02 02:41:33: pid 29910: LOG: watchdog process is initialized Dec 2 02:41:33 postgredbprod01 pgpool[29910]: [4-2] 2020-12-02 02:41:33: pid 29910: DETAIL: watchdog messaging data version: 1.1 Dec 2 02:41:33 postgredbprod01 pgpool[29912]: [16-1] 2020-12-02 02:41:33: pid 29912: LOG: new IPC connection received Dec 2 02:41:33 postgredbprod01 pgpool[29912]: [17-1] 2020-12-02 02:41:33: pid 29912: LOG: new IPC connection received Dec 2 02:41:33 postgredbprod01 pgpool[29910]: [5-1] 2020-12-02 02:41:33: pid 29910: LOG: Setting up socket for 0.0.0.0:9999 Dec 2 02:41:33 postgredbprod01 pgpool[29910]: [6-1] 2020-12-02 02:41:33: pid 29910: LOG: Setting up socket for :::9999 Dec 2 02:41:33 postgredbprod01 pgpool[29925]: [5-1] 2020-12-02 02:41:33: pid 29925: LOG: 2 watchdog nodes are configured for lifecheck Dec 2 02:41:33 postgredbprod01 pgpool[29925]: [6-1] 2020-12-02 02:41:33: pid 29925: LOG: watchdog nodes ID:0 Name:"postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 02:41:33 postgredbprod01 pgpool[29925]: [6-2] 2020-12-02 02:41:33: pid 29925: DETAIL: Host:"postgredbprod01" WD Port:9000 pgpool-II port:9999 Dec 2 02:41:33 postgredbprod01 pgpool[29925]: [7-1] 2020-12-02 02:41:33: pid 29925: LOG: watchdog nodes ID:1 Name:"Not_Set" Dec 2 02:41:33 postgredbprod01 pgpool[29925]: [7-2] 2020-12-02 02:41:33: pid 29925: DETAIL: Host:"postgredbprod02" WD Port:9000 pgpool-II port:9999 Dec 2 02:41:33 postgredbprod01 pgpool[29925]: [8-1] 2020-12-02 02:41:33: pid 29925: LOG: watchdog lifecheck trusted server "postgredbprod02" added for the availability check Dec 2 02:41:33 postgredbprod01 pgpool[29910]: [7-1] 2020-12-02 02:41:33: pid 29910: LOG: find_primary_node_repeatedly: waiting for finding a primary node Dec 2 02:41:33 postgredbprod01 pgpool[29910]: [8-1] 2020-12-02 02:41:33: pid 29910: LOG: find_primary_node: primary node is 0 Dec 2 02:41:33 postgredbprod01 pgpool[529]: [9-1] 2020-12-02 02:41:33: pid 529: LOG: PCP process: 529 started Dec 2 02:41:33 postgredbprod01 pgpool[29910]: [9-1] 2020-12-02 02:41:33: pid 29910: LOG: pgpool-II successfully started. version 4.1.5 (karasukiboshi) Dec 2 02:41:33 postgredbprod01 pgpool[29910]: [10-1] 2020-12-02 02:41:33: pid 29910: LOG: node status[0]: 1 Dec 2 02:41:33 postgredbprod01 pgpool[29910]: [11-1] 2020-12-02 02:41:33: pid 29910: LOG: node status[1]: 0 Dec 2 02:41:34 postgredbprod01 pgpool[29929]: [8-1] 2020-12-02 02:41:34: pid 29929: LOG: failed to create watchdog heartbeat receive socket. Dec 2 02:41:34 postgredbprod01 pgpool[29929]: [8-2] 2020-12-02 02:41:34: pid 29929: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege Dec 2 02:41:34 postgredbprod01 pgpool[29929]: [9-1] 2020-12-02 02:41:34: pid 29929: LOG: set SO_REUSEPORT option to the socket Dec 2 02:41:34 postgredbprod01 pgpool[29929]: [10-1] 2020-12-02 02:41:34: pid 29929: LOG: creating watchdog heartbeat receive socket. Dec 2 02:41:34 postgredbprod01 pgpool[29929]: [10-2] 2020-12-02 02:41:34: pid 29929: DETAIL: set SO_REUSEPORT Dec 2 02:41:34 postgredbprod01 pgpool[29932]: [8-1] 2020-12-02 02:41:34: pid 29932: LOG: creating socket for sending heartbeat Dec 2 02:41:34 postgredbprod01 pgpool[29932]: [8-2] 2020-12-02 02:41:34: pid 29932: DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege Dec 2 02:41:34 postgredbprod01 pgpool[29932]: [9-1] 2020-12-02 02:41:34: pid 29932: LOG: set SO_REUSEPORT option to the socket Dec 2 02:41:34 postgredbprod01 pgpool[29932]: [10-1] 2020-12-02 02:41:34: pid 29932: LOG: creating socket for sending heartbeat Dec 2 02:41:34 postgredbprod01 pgpool[29932]: [10-2] 2020-12-02 02:41:34: pid 29932: DETAIL: set SO_REUSEPORT Dec 2 02:43:05 postgredbprod01 pgpool[670]: [1-1] 2020-12-02 02:43:05: pid 670: LOG: stop request sent to pgpool. waiting for termination... Dec 2 02:43:05 postgredbprod01 pgpool[29912]: [18-1] 2020-12-02 02:43:05: pid 29912: LOG: Watchdog is shutting down Dec 2 02:43:21 postgredbprod01 pgpool[714]: [1-1] 2020-12-02 02:43:21: pid 714: LOG: memory cache initialized Dec 2 02:43:21 postgredbprod01 pgpool[714]: [1-2] 2020-12-02 02:43:21: pid 714: DETAIL: memcache blocks :64 Dec 2 02:43:21 postgredbprod01 pgpool[714]: [2-1] 2020-12-02 02:43:21: pid 714: LOG: pool_discard_oid_maps: discarded memqcache oid maps Dec 2 02:43:21 postgredbprod01 pgpool[714]: [3-1] 2020-12-02 02:43:21: pid 714: LOG: waiting for watchdog to initialize Dec 2 02:43:21 postgredbprod01 pgpool[716]: [3-1] 2020-12-02 02:43:21: pid 716: LOG: setting the local watchdog node name to "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 02:43:21 postgredbprod01 pgpool[716]: [4-1] 2020-12-02 02:43:21: pid 716: LOG: watchdog cluster is configured with 1 remote nodes Dec 2 02:43:21 postgredbprod01 pgpool[716]: [5-1] 2020-12-02 02:43:21: pid 716: LOG: watchdog remote node:0 on postgredbprod02:9000 Dec 2 02:43:21 postgredbprod01 pgpool[716]: [6-1] 2020-12-02 02:43:21: pid 716: LOG: interface monitoring is disabled in watchdog Dec 2 02:43:21 postgredbprod01 pgpool[716]: [7-1] 2020-12-02 02:43:21: pid 716: LOG: watchdog node state changed from [DEAD] to [LOADING] Dec 2 02:43:26 postgredbprod01 pgpool[716]: [8-1] 2020-12-02 02:43:26: pid 716: LOG: watchdog node state changed from [LOADING] to [JOINING] Dec 2 02:43:30 postgredbprod01 pgpool[716]: [9-1] 2020-12-02 02:43:30: pid 716: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Dec 2 02:43:31 postgredbprod01 pgpool[716]: [10-1] 2020-12-02 02:43:31: pid 716: LOG: I am the only alive node in the watchdog cluster Dec 2 02:43:31 postgredbprod01 pgpool[716]: [10-2] 2020-12-02 02:43:31: pid 716: HINT: skipping stand for coordinator state Dec 2 02:43:31 postgredbprod01 pgpool[716]: [11-1] 2020-12-02 02:43:31: pid 716: LOG: watchdog node state changed from [INITIALIZING] to [MASTER] Dec 2 02:43:31 postgredbprod01 pgpool[716]: [12-1] 2020-12-02 02:43:31: pid 716: LOG: I am announcing my self as master/coordinator watchdog node Dec 2 02:43:35 postgredbprod01 pgpool[716]: [13-1] 2020-12-02 02:43:35: pid 716: LOG: I am the cluster leader node Dec 2 02:43:35 postgredbprod01 pgpool[716]: [13-2] 2020-12-02 02:43:35: pid 716: DETAIL: our declare coordinator message is accepted by all nodes Dec 2 02:43:35 postgredbprod01 pgpool[716]: [14-1] 2020-12-02 02:43:35: pid 716: LOG: setting the local node "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" as watchdog cluster master Dec 2 02:43:35 postgredbprod01 pgpool[716]: [15-1] 2020-12-02 02:43:35: pid 716: LOG: I am the cluster leader node but we do not have enough nodes in cluster Dec 2 02:43:35 postgredbprod01 pgpool[716]: [15-2] 2020-12-02 02:43:35: pid 716: DETAIL: waiting for the quorum to start escalation process Dec 2 02:43:35 postgredbprod01 pgpool[714]: [4-1] 2020-12-02 02:43:35: pid 714: LOG: watchdog process is initialized Dec 2 02:43:35 postgredbprod01 pgpool[714]: [4-2] 2020-12-02 02:43:35: pid 714: DETAIL: watchdog messaging data version: 1.1 Dec 2 02:43:35 postgredbprod01 pgpool[716]: [16-1] 2020-12-02 02:43:35: pid 716: LOG: new IPC connection received Dec 2 02:43:35 postgredbprod01 pgpool[714]: [5-1] 2020-12-02 02:43:35: pid 714: LOG: Setting up socket for 0.0.0.0:9999 Dec 2 02:43:35 postgredbprod01 pgpool[714]: [6-1] 2020-12-02 02:43:35: pid 714: LOG: Setting up socket for :::9999 Dec 2 02:43:35 postgredbprod01 pgpool[716]: [17-1] 2020-12-02 02:43:35: pid 716: LOG: new IPC connection received Dec 2 02:43:35 postgredbprod01 pgpool[728]: [5-1] 2020-12-02 02:43:35: pid 728: LOG: 2 watchdog nodes are configured for lifecheck Dec 2 02:43:35 postgredbprod01 pgpool[728]: [6-1] 2020-12-02 02:43:35: pid 728: LOG: watchdog nodes ID:0 Name:"postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 02:43:35 postgredbprod01 pgpool[728]: [6-2] 2020-12-02 02:43:35: pid 728: DETAIL: Host:"postgredbprod01" WD Port:9000 pgpool-II port:9999 Dec 2 02:43:35 postgredbprod01 pgpool[728]: [7-1] 2020-12-02 02:43:35: pid 728: LOG: watchdog nodes ID:1 Name:"Not_Set" Dec 2 02:43:35 postgredbprod01 pgpool[728]: [7-2] 2020-12-02 02:43:35: pid 728: DETAIL: Host:"postgredbprod02" WD Port:9000 pgpool-II port:9999 Dec 2 02:43:35 postgredbprod01 pgpool[728]: [8-1] 2020-12-02 02:43:35: pid 728: LOG: watchdog lifecheck trusted server "postgredbprod02" added for the availability check Dec 2 02:43:36 postgredbprod01 pgpool[714]: [7-1] 2020-12-02 02:43:36: pid 714: LOG: find_primary_node_repeatedly: waiting for finding a primary node Dec 2 02:43:36 postgredbprod01 pgpool[714]: [8-1] 2020-12-02 02:43:36: pid 714: LOG: find_primary_node: primary node is 0 Dec 2 02:43:36 postgredbprod01 pgpool[3785]: [9-1] 2020-12-02 02:43:36: pid 3785: LOG: PCP process: 3785 started Dec 2 02:43:36 postgredbprod01 pgpool[714]: [9-1] 2020-12-02 02:43:36: pid 714: LOG: pgpool-II successfully started. version 4.1.5 (karasukiboshi) Dec 2 02:43:36 postgredbprod01 pgpool[714]: [10-1] 2020-12-02 02:43:36: pid 714: LOG: node status[0]: 1 Dec 2 02:43:36 postgredbprod01 pgpool[714]: [11-1] 2020-12-02 02:43:36: pid 714: LOG: node status[1]: 0 Dec 2 02:43:36 postgredbprod01 pgpool[739]: [8-1] 2020-12-02 02:43:36: pid 739: LOG: failed to create watchdog heartbeat receive socket. Dec 2 02:43:36 postgredbprod01 pgpool[739]: [8-2] 2020-12-02 02:43:36: pid 739: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege Dec 2 02:43:36 postgredbprod01 pgpool[739]: [9-1] 2020-12-02 02:43:36: pid 739: LOG: set SO_REUSEPORT option to the socket Dec 2 02:43:36 postgredbprod01 pgpool[742]: [8-1] 2020-12-02 02:43:36: pid 742: LOG: creating socket for sending heartbeat Dec 2 02:43:36 postgredbprod01 pgpool[742]: [8-2] 2020-12-02 02:43:36: pid 742: DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege Dec 2 02:43:36 postgredbprod01 pgpool[739]: [10-1] 2020-12-02 02:43:36: pid 739: LOG: creating watchdog heartbeat receive socket. Dec 2 02:43:36 postgredbprod01 pgpool[739]: [10-2] 2020-12-02 02:43:36: pid 739: DETAIL: set SO_REUSEPORT Dec 2 02:43:36 postgredbprod01 pgpool[742]: [9-1] 2020-12-02 02:43:36: pid 742: LOG: set SO_REUSEPORT option to the socket Dec 2 02:43:36 postgredbprod01 pgpool[742]: [10-1] 2020-12-02 02:43:36: pid 742: LOG: creating socket for sending heartbeat Dec 2 02:43:36 postgredbprod01 pgpool[742]: [10-2] 2020-12-02 02:43:36: pid 742: DETAIL: set SO_REUSEPORT Dec 2 02:44:17 postgredbprod01 pgpool[3785]: [10-1] 2020-12-02 02:44:17: pid 3785: LOG: forked new pcp worker, pid=3830 socket=8 Dec 2 02:44:17 postgredbprod01 pgpool[716]: [18-1] 2020-12-02 02:44:17: pid 716: LOG: new IPC connection received Dec 2 02:44:17 postgredbprod01 pgpool[3785]: [11-1] 2020-12-02 02:44:17: pid 3785: LOG: PCP process with pid: 3830 exit with SUCCESS. Dec 2 02:44:17 postgredbprod01 pgpool[3785]: [12-1] 2020-12-02 02:44:17: pid 3785: LOG: PCP process with pid: 3830 exits with status 0 Dec 2 03:03:46 postgredbprod01 pgpool[730]: [7-1] 2020-12-02 03:03:46: pid 730: LOG: backend throws an error message Dec 2 03:03:46 postgredbprod01 pgpool[730]: [7-2] 2020-12-02 03:03:46: pid 730: DETAIL: database "pgpool" does not exist Dec 2 03:03:46 postgredbprod01 pgpool[730]: [8-1] 2020-12-02 03:03:46: pid 730: ERROR: unable to forward message to frontend Dec 2 03:03:46 postgredbprod01 pgpool[730]: [8-2] 2020-12-02 03:03:46: pid 730: DETAIL: FATAL error occured on backend Dec 2 03:03:54 postgredbprod01 pgpool[732]: [7-1] 2020-12-02 03:03:54: pid 732: LOG: backend throws an error message Dec 2 03:03:54 postgredbprod01 pgpool[732]: [7-2] 2020-12-02 03:03:54: pid 732: DETAIL: database "pgpool" does not exist Dec 2 03:03:54 postgredbprod01 pgpool[732]: [8-1] 2020-12-02 03:03:54: pid 732: ERROR: unable to forward message to frontend Dec 2 03:03:54 postgredbprod01 pgpool[732]: [8-2] 2020-12-02 03:03:54: pid 732: DETAIL: FATAL error occured on backend Dec 2 03:04:10 postgredbprod01 pgpool[734]: [7-1] 2020-12-02 03:04:10: pid 734: LOG: pool_reuse_block: blockid: 0 Dec 2 03:04:10 postgredbprod01 pgpool[734]: [7-2] 2020-12-02 03:04:10: pid 734: CONTEXT: while searching system catalog, When relcache is missed Dec 2 03:08:52 postgredbprod01 pgpool[5231]: [1-1] 2020-12-02 03:08:52: pid 5231: LOG: stop request sent to pgpool. waiting for termination... Dec 2 03:08:52 postgredbprod01 pgpool[716]: [19-1] 2020-12-02 03:08:52: pid 716: LOG: Watchdog is shutting down Dec 2 03:09:29 postgredbprod01 pgpool[5263]: [1-1] 2020-12-02 03:09:29: pid 5263: LOG: memory cache initialized Dec 2 03:09:29 postgredbprod01 pgpool[5263]: [1-2] 2020-12-02 03:09:29: pid 5263: DETAIL: memcache blocks :64 Dec 2 03:09:29 postgredbprod01 pgpool[5263]: [2-1] 2020-12-02 03:09:29: pid 5263: LOG: pool_discard_oid_maps: discarded memqcache oid maps Dec 2 03:09:29 postgredbprod01 pgpool[5263]: [3-1] 2020-12-02 03:09:29: pid 5263: LOG: waiting for watchdog to initialize Dec 2 03:09:29 postgredbprod01 pgpool[5265]: [3-1] 2020-12-02 03:09:29: pid 5265: LOG: setting the local watchdog node name to "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 03:09:29 postgredbprod01 pgpool[5265]: [4-1] 2020-12-02 03:09:29: pid 5265: LOG: watchdog cluster is configured with 1 remote nodes Dec 2 03:09:29 postgredbprod01 pgpool[5265]: [5-1] 2020-12-02 03:09:29: pid 5265: LOG: watchdog remote node:0 on postgredbprod02:9000 Dec 2 03:09:29 postgredbprod01 pgpool[5265]: [6-1] 2020-12-02 03:09:29: pid 5265: LOG: interface monitoring is disabled in watchdog Dec 2 03:09:29 postgredbprod01 pgpool[5265]: [7-1] 2020-12-02 03:09:29: pid 5265: LOG: watchdog node state changed from [DEAD] to [LOADING] Dec 2 03:09:34 postgredbprod01 pgpool[5265]: [8-1] 2020-12-02 03:09:34: pid 5265: LOG: watchdog node state changed from [LOADING] to [JOINING] Dec 2 03:09:38 postgredbprod01 pgpool[5265]: [9-1] 2020-12-02 03:09:38: pid 5265: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Dec 2 03:09:39 postgredbprod01 pgpool[5265]: [10-1] 2020-12-02 03:09:39: pid 5265: LOG: I am the only alive node in the watchdog cluster Dec 2 03:09:39 postgredbprod01 pgpool[5265]: [10-2] 2020-12-02 03:09:39: pid 5265: HINT: skipping stand for coordinator state Dec 2 03:09:39 postgredbprod01 pgpool[5265]: [11-1] 2020-12-02 03:09:39: pid 5265: LOG: watchdog node state changed from [INITIALIZING] to [MASTER] Dec 2 03:09:39 postgredbprod01 pgpool[5265]: [12-1] 2020-12-02 03:09:39: pid 5265: LOG: I am announcing my self as master/coordinator watchdog node Dec 2 03:09:43 postgredbprod01 pgpool[5265]: [13-1] 2020-12-02 03:09:43: pid 5265: LOG: I am the cluster leader node Dec 2 03:09:43 postgredbprod01 pgpool[5265]: [13-2] 2020-12-02 03:09:43: pid 5265: DETAIL: our declare coordinator message is accepted by all nodes Dec 2 03:09:43 postgredbprod01 pgpool[5265]: [14-1] 2020-12-02 03:09:43: pid 5265: LOG: setting the local node "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" as watchdog cluster master Dec 2 03:09:43 postgredbprod01 pgpool[5265]: [15-1] 2020-12-02 03:09:43: pid 5265: LOG: I am the cluster leader node but we do not have enough nodes in cluster Dec 2 03:09:43 postgredbprod01 pgpool[5265]: [15-2] 2020-12-02 03:09:43: pid 5265: DETAIL: waiting for the quorum to start escalation process Dec 2 03:09:43 postgredbprod01 pgpool[5263]: [4-1] 2020-12-02 03:09:43: pid 5263: LOG: watchdog process is initialized Dec 2 03:09:43 postgredbprod01 pgpool[5263]: [4-2] 2020-12-02 03:09:43: pid 5263: DETAIL: watchdog messaging data version: 1.1 Dec 2 03:09:43 postgredbprod01 pgpool[5265]: [16-1] 2020-12-02 03:09:43: pid 5265: LOG: new IPC connection received Dec 2 03:09:43 postgredbprod01 pgpool[5263]: [5-1] 2020-12-02 03:09:43: pid 5263: LOG: Setting up socket for 0.0.0.0:9999 Dec 2 03:09:43 postgredbprod01 pgpool[5263]: [6-1] 2020-12-02 03:09:43: pid 5263: LOG: Setting up socket for :::9999 Dec 2 03:09:43 postgredbprod01 pgpool[5265]: [17-1] 2020-12-02 03:09:43: pid 5265: LOG: new IPC connection received Dec 2 03:09:43 postgredbprod01 pgpool[5275]: [5-1] 2020-12-02 03:09:43: pid 5275: LOG: 2 watchdog nodes are configured for lifecheck Dec 2 03:09:43 postgredbprod01 pgpool[5275]: [6-1] 2020-12-02 03:09:43: pid 5275: LOG: watchdog nodes ID:0 Name:"postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 03:09:43 postgredbprod01 pgpool[5275]: [6-2] 2020-12-02 03:09:43: pid 5275: DETAIL: Host:"postgredbprod01" WD Port:9000 pgpool-II port:9999 Dec 2 03:09:43 postgredbprod01 pgpool[5275]: [7-1] 2020-12-02 03:09:43: pid 5275: LOG: watchdog nodes ID:1 Name:"Not_Set" Dec 2 03:09:43 postgredbprod01 pgpool[5275]: [7-2] 2020-12-02 03:09:43: pid 5275: DETAIL: Host:"postgredbprod02" WD Port:9000 pgpool-II port:9999 Dec 2 03:09:43 postgredbprod01 pgpool[5275]: [8-1] 2020-12-02 03:09:43: pid 5275: LOG: watchdog lifecheck trusted server "postgredbprod02" added for the availability check Dec 2 03:09:44 postgredbprod01 pgpool[5263]: [7-1] 2020-12-02 03:09:44: pid 5263: LOG: find_primary_node_repeatedly: waiting for finding a primary node Dec 2 03:09:44 postgredbprod01 pgpool[5263]: [8-1] 2020-12-02 03:09:44: pid 5263: LOG: find_primary_node: primary node is 0 Dec 2 03:09:44 postgredbprod01 pgpool[8280]: [9-1] 2020-12-02 03:09:44: pid 8280: LOG: PCP process: 8280 started Dec 2 03:09:44 postgredbprod01 pgpool[5263]: [9-1] 2020-12-02 03:09:44: pid 5263: LOG: pgpool-II successfully started. version 4.1.5 (karasukiboshi) Dec 2 03:09:44 postgredbprod01 pgpool[5263]: [10-1] 2020-12-02 03:09:44: pid 5263: LOG: node status[0]: 1 Dec 2 03:09:44 postgredbprod01 pgpool[5263]: [11-1] 2020-12-02 03:09:44: pid 5263: LOG: node status[1]: 0 Dec 2 03:09:44 postgredbprod01 pgpool[5279]: [8-1] 2020-12-02 03:09:44: pid 5279: LOG: failed to create watchdog heartbeat receive socket. Dec 2 03:09:44 postgredbprod01 pgpool[5279]: [8-2] 2020-12-02 03:09:44: pid 5279: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege Dec 2 03:09:44 postgredbprod01 pgpool[5279]: [9-1] 2020-12-02 03:09:44: pid 5279: LOG: set SO_REUSEPORT option to the socket Dec 2 03:09:44 postgredbprod01 pgpool[5279]: [10-1] 2020-12-02 03:09:44: pid 5279: LOG: creating watchdog heartbeat receive socket. Dec 2 03:09:44 postgredbprod01 pgpool[5279]: [10-2] 2020-12-02 03:09:44: pid 5279: DETAIL: set SO_REUSEPORT Dec 2 03:09:44 postgredbprod01 pgpool[5281]: [8-1] 2020-12-02 03:09:44: pid 5281: LOG: creating socket for sending heartbeat Dec 2 03:09:44 postgredbprod01 pgpool[5281]: [8-2] 2020-12-02 03:09:44: pid 5281: DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege Dec 2 03:09:44 postgredbprod01 pgpool[5281]: [9-1] 2020-12-02 03:09:44: pid 5281: LOG: set SO_REUSEPORT option to the socket Dec 2 03:09:44 postgredbprod01 pgpool[5281]: [10-1] 2020-12-02 03:09:44: pid 5281: LOG: creating socket for sending heartbeat Dec 2 03:09:44 postgredbprod01 pgpool[5281]: [10-2] 2020-12-02 03:09:44: pid 5281: DETAIL: set SO_REUSEPORT Dec 2 03:15:43 postgredbprod01 pgpool[8615]: [1-1] 2020-12-02 03:15:43: pid 8615: LOG: stop request sent to pgpool. waiting for termination... Dec 2 03:15:43 postgredbprod01 pgpool[5265]: [18-1] 2020-12-02 03:15:43: pid 5265: LOG: Watchdog is shutting down Dec 2 03:15:49 postgredbprod01 pgpool[8628]: [1-1] 2020-12-02 03:15:49: pid 8628: LOG: memory cache initialized Dec 2 03:15:49 postgredbprod01 pgpool[8628]: [1-2] 2020-12-02 03:15:49: pid 8628: DETAIL: memcache blocks :64 Dec 2 03:15:49 postgredbprod01 pgpool[8628]: [2-1] 2020-12-02 03:15:49: pid 8628: LOG: pool_discard_oid_maps: discarded memqcache oid maps Dec 2 03:15:49 postgredbprod01 pgpool[8628]: [3-1] 2020-12-02 03:15:49: pid 8628: LOG: waiting for watchdog to initialize Dec 2 03:15:49 postgredbprod01 pgpool[8630]: [3-1] 2020-12-02 03:15:49: pid 8630: LOG: setting the local watchdog node name to "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 03:15:49 postgredbprod01 pgpool[8630]: [4-1] 2020-12-02 03:15:49: pid 8630: LOG: watchdog cluster is configured with 1 remote nodes Dec 2 03:15:49 postgredbprod01 pgpool[8630]: [5-1] 2020-12-02 03:15:49: pid 8630: LOG: watchdog remote node:0 on postgredbprod02:9000 Dec 2 03:15:49 postgredbprod01 pgpool[8630]: [6-1] 2020-12-02 03:15:49: pid 8630: LOG: interface monitoring is disabled in watchdog Dec 2 03:15:49 postgredbprod01 pgpool[8630]: [7-1] 2020-12-02 03:15:49: pid 8630: LOG: watchdog node state changed from [DEAD] to [LOADING] Dec 2 03:15:54 postgredbprod01 pgpool[8630]: [8-1] 2020-12-02 03:15:54: pid 8630: LOG: watchdog node state changed from [LOADING] to [JOINING] Dec 2 03:15:58 postgredbprod01 pgpool[8630]: [9-1] 2020-12-02 03:15:58: pid 8630: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Dec 2 03:15:59 postgredbprod01 pgpool[8630]: [10-1] 2020-12-02 03:15:59: pid 8630: LOG: I am the only alive node in the watchdog cluster Dec 2 03:15:59 postgredbprod01 pgpool[8630]: [10-2] 2020-12-02 03:15:59: pid 8630: HINT: skipping stand for coordinator state Dec 2 03:15:59 postgredbprod01 pgpool[8630]: [11-1] 2020-12-02 03:15:59: pid 8630: LOG: watchdog node state changed from [INITIALIZING] to [MASTER] Dec 2 03:15:59 postgredbprod01 pgpool[8630]: [12-1] 2020-12-02 03:15:59: pid 8630: LOG: I am announcing my self as master/coordinator watchdog node Dec 2 03:16:03 postgredbprod01 pgpool[8630]: [13-1] 2020-12-02 03:16:03: pid 8630: LOG: I am the cluster leader node Dec 2 03:16:03 postgredbprod01 pgpool[8630]: [13-2] 2020-12-02 03:16:03: pid 8630: DETAIL: our declare coordinator message is accepted by all nodes Dec 2 03:16:03 postgredbprod01 pgpool[8630]: [14-1] 2020-12-02 03:16:03: pid 8630: LOG: setting the local node "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" as watchdog cluster master Dec 2 03:16:03 postgredbprod01 pgpool[8630]: [15-1] 2020-12-02 03:16:03: pid 8630: LOG: I am the cluster leader node but we do not have enough nodes in cluster Dec 2 03:16:03 postgredbprod01 pgpool[8630]: [15-2] 2020-12-02 03:16:03: pid 8630: DETAIL: waiting for the quorum to start escalation process Dec 2 03:16:03 postgredbprod01 pgpool[8628]: [4-1] 2020-12-02 03:16:03: pid 8628: LOG: watchdog process is initialized Dec 2 03:16:03 postgredbprod01 pgpool[8628]: [4-2] 2020-12-02 03:16:03: pid 8628: DETAIL: watchdog messaging data version: 1.1 Dec 2 03:16:03 postgredbprod01 pgpool[8630]: [16-1] 2020-12-02 03:16:03: pid 8630: LOG: new IPC connection received Dec 2 03:16:03 postgredbprod01 pgpool[8630]: [17-1] 2020-12-02 03:16:03: pid 8630: LOG: new IPC connection received Dec 2 03:16:03 postgredbprod01 pgpool[8640]: [5-1] 2020-12-02 03:16:03: pid 8640: LOG: 2 watchdog nodes are configured for lifecheck Dec 2 03:16:03 postgredbprod01 pgpool[8640]: [6-1] 2020-12-02 03:16:03: pid 8640: LOG: watchdog nodes ID:0 Name:"postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 03:16:03 postgredbprod01 pgpool[8640]: [6-2] 2020-12-02 03:16:03: pid 8640: DETAIL: Host:"postgredbprod01" WD Port:9000 pgpool-II port:9999 Dec 2 03:16:03 postgredbprod01 pgpool[8628]: [5-1] 2020-12-02 03:16:03: pid 8628: LOG: Setting up socket for 0.0.0.0:9999 Dec 2 03:16:03 postgredbprod01 pgpool[8640]: [7-1] 2020-12-02 03:16:03: pid 8640: LOG: watchdog nodes ID:1 Name:"Not_Set" Dec 2 03:16:03 postgredbprod01 pgpool[8640]: [7-2] 2020-12-02 03:16:03: pid 8640: DETAIL: Host:"postgredbprod02" WD Port:9000 pgpool-II port:9999 Dec 2 03:16:03 postgredbprod01 pgpool[8628]: [6-1] 2020-12-02 03:16:03: pid 8628: LOG: Setting up socket for :::9999 Dec 2 03:16:03 postgredbprod01 pgpool[8640]: [8-1] 2020-12-02 03:16:03: pid 8640: LOG: watchdog lifecheck trusted server "postgredbprod02" added for the availability check Dec 2 03:16:04 postgredbprod01 pgpool[8628]: [7-1] 2020-12-02 03:16:04: pid 8628: LOG: find_primary_node_repeatedly: waiting for finding a primary node Dec 2 03:16:04 postgredbprod01 pgpool[8628]: [8-1] 2020-12-02 03:16:04: pid 8628: LOG: find_primary_node: primary node is 0 Dec 2 03:16:04 postgredbprod01 pgpool[11644]: [9-1] 2020-12-02 03:16:04: pid 11644: LOG: PCP process: 11644 started Dec 2 03:16:04 postgredbprod01 pgpool[8628]: [9-1] 2020-12-02 03:16:04: pid 8628: LOG: pgpool-II successfully started. version 4.1.5 (karasukiboshi) Dec 2 03:16:04 postgredbprod01 pgpool[8628]: [10-1] 2020-12-02 03:16:04: pid 8628: LOG: node status[0]: 1 Dec 2 03:16:04 postgredbprod01 pgpool[8628]: [11-1] 2020-12-02 03:16:04: pid 8628: LOG: node status[1]: 0 Dec 2 03:16:04 postgredbprod01 pgpool[8641]: [8-1] 2020-12-02 03:16:04: pid 8641: LOG: failed to create watchdog heartbeat receive socket. Dec 2 03:16:04 postgredbprod01 pgpool[8641]: [8-2] 2020-12-02 03:16:04: pid 8641: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege Dec 2 03:16:04 postgredbprod01 pgpool[8641]: [9-1] 2020-12-02 03:16:04: pid 8641: LOG: set SO_REUSEPORT option to the socket Dec 2 03:16:04 postgredbprod01 pgpool[8641]: [10-1] 2020-12-02 03:16:04: pid 8641: LOG: creating watchdog heartbeat receive socket. Dec 2 03:16:04 postgredbprod01 pgpool[8641]: [10-2] 2020-12-02 03:16:04: pid 8641: DETAIL: set SO_REUSEPORT Dec 2 03:16:04 postgredbprod01 pgpool[8643]: [8-1] 2020-12-02 03:16:04: pid 8643: LOG: creating socket for sending heartbeat Dec 2 03:16:04 postgredbprod01 pgpool[8643]: [8-2] 2020-12-02 03:16:04: pid 8643: DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege Dec 2 03:16:04 postgredbprod01 pgpool[8643]: [9-1] 2020-12-02 03:16:04: pid 8643: LOG: set SO_REUSEPORT option to the socket Dec 2 03:16:04 postgredbprod01 pgpool[8643]: [10-1] 2020-12-02 03:16:04: pid 8643: LOG: creating socket for sending heartbeat Dec 2 03:16:04 postgredbprod01 pgpool[8643]: [10-2] 2020-12-02 03:16:04: pid 8643: DETAIL: set SO_REUSEPORT Dec 2 03:34:00 postgredbprod01 pgpool[12628]: [1-1] 2020-12-02 03:34:00: pid 12628: LOG: stop request sent to pgpool. waiting for termination... Dec 2 03:34:00 postgredbprod01 pgpool[8630]: [18-1] 2020-12-02 03:34:00: pid 8630: LOG: Watchdog is shutting down Dec 2 03:34:23 postgredbprod01 pgpool[12651]: [1-1] 2020-12-02 03:34:23: pid 12651: LOG: memory cache initialized Dec 2 03:34:23 postgredbprod01 pgpool[12651]: [1-2] 2020-12-02 03:34:23: pid 12651: DETAIL: memcache blocks :64 Dec 2 03:34:23 postgredbprod01 pgpool[12651]: [2-1] 2020-12-02 03:34:23: pid 12651: LOG: pool_discard_oid_maps: discarded memqcache oid maps Dec 2 03:34:23 postgredbprod01 pgpool[12651]: [3-1] 2020-12-02 03:34:23: pid 12651: LOG: waiting for watchdog to initialize Dec 2 03:34:23 postgredbprod01 pgpool[12653]: [3-1] 2020-12-02 03:34:23: pid 12653: LOG: setting the local watchdog node name to "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 03:34:23 postgredbprod01 pgpool[12653]: [4-1] 2020-12-02 03:34:23: pid 12653: LOG: watchdog cluster is configured with 1 remote nodes Dec 2 03:34:23 postgredbprod01 pgpool[12653]: [5-1] 2020-12-02 03:34:23: pid 12653: LOG: watchdog remote node:0 on postgredbprod02:9000 Dec 2 03:34:23 postgredbprod01 pgpool[12653]: [6-1] 2020-12-02 03:34:23: pid 12653: LOG: interface monitoring is disabled in watchdog Dec 2 03:34:23 postgredbprod01 pgpool[12653]: [7-1] 2020-12-02 03:34:23: pid 12653: LOG: watchdog node state changed from [DEAD] to [LOADING] Dec 2 03:34:28 postgredbprod01 pgpool[12653]: [8-1] 2020-12-02 03:34:28: pid 12653: LOG: watchdog node state changed from [LOADING] to [JOINING] Dec 2 03:34:32 postgredbprod01 pgpool[12653]: [9-1] 2020-12-02 03:34:32: pid 12653: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Dec 2 03:34:33 postgredbprod01 pgpool[12653]: [10-1] 2020-12-02 03:34:33: pid 12653: LOG: I am the only alive node in the watchdog cluster Dec 2 03:34:33 postgredbprod01 pgpool[12653]: [10-2] 2020-12-02 03:34:33: pid 12653: HINT: skipping stand for coordinator state Dec 2 03:34:33 postgredbprod01 pgpool[12653]: [11-1] 2020-12-02 03:34:33: pid 12653: LOG: watchdog node state changed from [INITIALIZING] to [MASTER] Dec 2 03:34:33 postgredbprod01 pgpool[12653]: [12-1] 2020-12-02 03:34:33: pid 12653: LOG: I am announcing my self as master/coordinator watchdog node Dec 2 03:34:37 postgredbprod01 pgpool[12653]: [13-1] 2020-12-02 03:34:37: pid 12653: LOG: I am the cluster leader node Dec 2 03:34:37 postgredbprod01 pgpool[12653]: [13-2] 2020-12-02 03:34:37: pid 12653: DETAIL: our declare coordinator message is accepted by all nodes Dec 2 03:34:37 postgredbprod01 pgpool[12653]: [14-1] 2020-12-02 03:34:37: pid 12653: LOG: setting the local node "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" as watchdog cluster master Dec 2 03:34:37 postgredbprod01 pgpool[12653]: [15-1] 2020-12-02 03:34:37: pid 12653: LOG: I am the cluster leader node but we do not have enough nodes in cluster Dec 2 03:34:37 postgredbprod01 pgpool[12653]: [15-2] 2020-12-02 03:34:37: pid 12653: DETAIL: waiting for the quorum to start escalation process Dec 2 03:34:37 postgredbprod01 pgpool[12651]: [4-1] 2020-12-02 03:34:37: pid 12651: LOG: watchdog process is initialized Dec 2 03:34:37 postgredbprod01 pgpool[12651]: [4-2] 2020-12-02 03:34:37: pid 12651: DETAIL: watchdog messaging data version: 1.1 Dec 2 03:34:37 postgredbprod01 pgpool[12653]: [16-1] 2020-12-02 03:34:37: pid 12653: LOG: new IPC connection received Dec 2 03:34:37 postgredbprod01 pgpool[12653]: [17-1] 2020-12-02 03:34:37: pid 12653: LOG: new IPC connection received Dec 2 03:34:37 postgredbprod01 pgpool[12651]: [5-1] 2020-12-02 03:34:37: pid 12651: LOG: Setting up socket for 0.0.0.0:9999 Dec 2 03:34:37 postgredbprod01 pgpool[12651]: [6-1] 2020-12-02 03:34:37: pid 12651: LOG: Setting up socket for :::9999 Dec 2 03:34:37 postgredbprod01 pgpool[12662]: [5-1] 2020-12-02 03:34:37: pid 12662: LOG: 2 watchdog nodes are configured for lifecheck Dec 2 03:34:37 postgredbprod01 pgpool[12662]: [6-1] 2020-12-02 03:34:37: pid 12662: LOG: watchdog nodes ID:0 Name:"postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 03:34:37 postgredbprod01 pgpool[12662]: [6-2] 2020-12-02 03:34:37: pid 12662: DETAIL: Host:"postgredbprod01" WD Port:9000 pgpool-II port:9999 Dec 2 03:34:37 postgredbprod01 pgpool[12662]: [7-1] 2020-12-02 03:34:37: pid 12662: LOG: watchdog nodes ID:1 Name:"Not_Set" Dec 2 03:34:37 postgredbprod01 pgpool[12662]: [7-2] 2020-12-02 03:34:37: pid 12662: DETAIL: Host:"postgredbprod02" WD Port:9000 pgpool-II port:9999 Dec 2 03:34:37 postgredbprod01 pgpool[12662]: [8-1] 2020-12-02 03:34:37: pid 12662: LOG: watchdog lifecheck trusted server "postgredbprod02" added for the availability check Dec 2 03:34:37 postgredbprod01 pgpool[12651]: [7-1] 2020-12-02 03:34:37: pid 12651: LOG: find_primary_node_repeatedly: waiting for finding a primary node Dec 2 03:34:37 postgredbprod01 pgpool[12651]: [8-1] 2020-12-02 03:34:37: pid 12651: LOG: find_primary_node: primary node is 0 Dec 2 03:34:37 postgredbprod01 pgpool[15676]: [9-1] 2020-12-02 03:34:37: pid 15676: LOG: PCP process: 15676 started Dec 2 03:34:37 postgredbprod01 pgpool[12651]: [9-1] 2020-12-02 03:34:37: pid 12651: LOG: pgpool-II successfully started. version 4.1.5 (karasukiboshi) Dec 2 03:34:37 postgredbprod01 pgpool[12651]: [10-1] 2020-12-02 03:34:37: pid 12651: LOG: node status[0]: 1 Dec 2 03:34:37 postgredbprod01 pgpool[12651]: [11-1] 2020-12-02 03:34:37: pid 12651: LOG: node status[1]: 0 Dec 2 03:34:38 postgredbprod01 pgpool[12664]: [8-1] 2020-12-02 03:34:38: pid 12664: LOG: failed to create watchdog heartbeat receive socket. Dec 2 03:34:38 postgredbprod01 pgpool[12664]: [8-2] 2020-12-02 03:34:38: pid 12664: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege Dec 2 03:34:38 postgredbprod01 pgpool[12664]: [9-1] 2020-12-02 03:34:38: pid 12664: LOG: set SO_REUSEPORT option to the socket Dec 2 03:34:38 postgredbprod01 pgpool[12664]: [10-1] 2020-12-02 03:34:38: pid 12664: LOG: creating watchdog heartbeat receive socket. Dec 2 03:34:38 postgredbprod01 pgpool[12664]: [10-2] 2020-12-02 03:34:38: pid 12664: DETAIL: set SO_REUSEPORT Dec 2 03:34:38 postgredbprod01 pgpool[12666]: [8-1] 2020-12-02 03:34:38: pid 12666: LOG: creating socket for sending heartbeat Dec 2 03:34:38 postgredbprod01 pgpool[12666]: [8-2] 2020-12-02 03:34:38: pid 12666: DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege Dec 2 03:34:38 postgredbprod01 pgpool[12666]: [9-1] 2020-12-02 03:34:38: pid 12666: LOG: set SO_REUSEPORT option to the socket Dec 2 03:34:38 postgredbprod01 pgpool[12666]: [10-1] 2020-12-02 03:34:38: pid 12666: LOG: creating socket for sending heartbeat Dec 2 03:34:38 postgredbprod01 pgpool[12666]: [10-2] 2020-12-02 03:34:38: pid 12666: DETAIL: set SO_REUSEPORT Dec 2 03:41:31 postgredbprod01 pgpool[16120]: [1-1] 2020-12-02 03:41:31: pid 16120: LOG: stop request sent to pgpool. waiting for termination... Dec 2 03:41:31 postgredbprod01 pgpool[12653]: [18-1] 2020-12-02 03:41:31: pid 12653: LOG: Watchdog is shutting down Dec 2 03:41:52 postgredbprod01 pgpool[16141]: [1-1] 2020-12-02 03:41:52: pid 16141: LOG: memory cache initialized Dec 2 03:41:52 postgredbprod01 pgpool[16141]: [1-2] 2020-12-02 03:41:52: pid 16141: DETAIL: memcache blocks :64 Dec 2 03:41:52 postgredbprod01 pgpool[16141]: [2-1] 2020-12-02 03:41:52: pid 16141: LOG: pool_discard_oid_maps: discarded memqcache oid maps Dec 2 03:41:52 postgredbprod01 pgpool[16141]: [3-1] 2020-12-02 03:41:52: pid 16141: LOG: waiting for watchdog to initialize Dec 2 03:41:52 postgredbprod01 pgpool[16143]: [3-1] 2020-12-02 03:41:52: pid 16143: LOG: setting the local watchdog node name to "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 03:41:52 postgredbprod01 pgpool[16143]: [4-1] 2020-12-02 03:41:52: pid 16143: LOG: watchdog cluster is configured with 1 remote nodes Dec 2 03:41:52 postgredbprod01 pgpool[16143]: [5-1] 2020-12-02 03:41:52: pid 16143: LOG: watchdog remote node:0 on postgredbprod02:9000 Dec 2 03:41:52 postgredbprod01 pgpool[16143]: [6-1] 2020-12-02 03:41:52: pid 16143: LOG: interface monitoring is disabled in watchdog Dec 2 03:41:52 postgredbprod01 pgpool[16143]: [7-1] 2020-12-02 03:41:52: pid 16143: LOG: watchdog node state changed from [DEAD] to [LOADING] Dec 2 03:41:57 postgredbprod01 pgpool[16143]: [8-1] 2020-12-02 03:41:57: pid 16143: LOG: watchdog node state changed from [LOADING] to [JOINING] Dec 2 03:42:01 postgredbprod01 pgpool[16143]: [9-1] 2020-12-02 03:42:01: pid 16143: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Dec 2 03:42:02 postgredbprod01 pgpool[16143]: [10-1] 2020-12-02 03:42:02: pid 16143: LOG: I am the only alive node in the watchdog cluster Dec 2 03:42:02 postgredbprod01 pgpool[16143]: [10-2] 2020-12-02 03:42:02: pid 16143: HINT: skipping stand for coordinator state Dec 2 03:42:02 postgredbprod01 pgpool[16143]: [11-1] 2020-12-02 03:42:02: pid 16143: LOG: watchdog node state changed from [INITIALIZING] to [MASTER] Dec 2 03:42:02 postgredbprod01 pgpool[16143]: [12-1] 2020-12-02 03:42:02: pid 16143: LOG: I am announcing my self as master/coordinator watchdog node Dec 2 03:42:06 postgredbprod01 pgpool[16143]: [13-1] 2020-12-02 03:42:06: pid 16143: LOG: I am the cluster leader node Dec 2 03:42:06 postgredbprod01 pgpool[16143]: [13-2] 2020-12-02 03:42:06: pid 16143: DETAIL: our declare coordinator message is accepted by all nodes Dec 2 03:42:06 postgredbprod01 pgpool[16143]: [14-1] 2020-12-02 03:42:06: pid 16143: LOG: setting the local node "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" as watchdog cluster master Dec 2 03:42:06 postgredbprod01 pgpool[16143]: [15-1] 2020-12-02 03:42:06: pid 16143: LOG: I am the cluster leader node but we do not have enough nodes in cluster Dec 2 03:42:06 postgredbprod01 pgpool[16143]: [15-2] 2020-12-02 03:42:06: pid 16143: DETAIL: waiting for the quorum to start escalation process Dec 2 03:42:06 postgredbprod01 pgpool[16141]: [4-1] 2020-12-02 03:42:06: pid 16141: LOG: watchdog process is initialized Dec 2 03:42:06 postgredbprod01 pgpool[16141]: [4-2] 2020-12-02 03:42:06: pid 16141: DETAIL: watchdog messaging data version: 1.1 Dec 2 03:42:06 postgredbprod01 pgpool[16143]: [16-1] 2020-12-02 03:42:06: pid 16143: LOG: new IPC connection received Dec 2 03:42:06 postgredbprod01 pgpool[16143]: [17-1] 2020-12-02 03:42:06: pid 16143: LOG: new IPC connection received Dec 2 03:42:06 postgredbprod01 pgpool[16141]: [5-1] 2020-12-02 03:42:06: pid 16141: LOG: Setting up socket for 0.0.0.0:9999 Dec 2 03:42:06 postgredbprod01 pgpool[16141]: [6-1] 2020-12-02 03:42:06: pid 16141: LOG: Setting up socket for :::9999 Dec 2 03:42:06 postgredbprod01 pgpool[16153]: [5-1] 2020-12-02 03:42:06: pid 16153: LOG: 2 watchdog nodes are configured for lifecheck Dec 2 03:42:06 postgredbprod01 pgpool[16153]: [6-1] 2020-12-02 03:42:06: pid 16153: LOG: watchdog nodes ID:0 Name:"postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 03:42:06 postgredbprod01 pgpool[16153]: [6-2] 2020-12-02 03:42:06: pid 16153: DETAIL: Host:"postgredbprod01" WD Port:9000 pgpool-II port:9999 Dec 2 03:42:06 postgredbprod01 pgpool[16153]: [7-1] 2020-12-02 03:42:06: pid 16153: LOG: watchdog nodes ID:1 Name:"Not_Set" Dec 2 03:42:06 postgredbprod01 pgpool[16153]: [7-2] 2020-12-02 03:42:06: pid 16153: DETAIL: Host:"postgredbprod02" WD Port:9000 pgpool-II port:9999 Dec 2 03:42:06 postgredbprod01 pgpool[16153]: [8-1] 2020-12-02 03:42:06: pid 16153: LOG: watchdog lifecheck trusted server "postgredbprod02" added for the availability check Dec 2 03:42:07 postgredbprod01 pgpool[16141]: [7-1] 2020-12-02 03:42:07: pid 16141: LOG: find_primary_node_repeatedly: waiting for finding a primary node Dec 2 03:42:07 postgredbprod01 pgpool[16141]: [8-1] 2020-12-02 03:42:07: pid 16141: LOG: find_primary_node: primary node is 0 Dec 2 03:42:07 postgredbprod01 pgpool[19160]: [9-1] 2020-12-02 03:42:07: pid 19160: LOG: PCP process: 19160 started Dec 2 03:42:07 postgredbprod01 pgpool[16141]: [9-1] 2020-12-02 03:42:07: pid 16141: LOG: pgpool-II successfully started. version 4.1.5 (karasukiboshi) Dec 2 03:42:07 postgredbprod01 pgpool[16141]: [10-1] 2020-12-02 03:42:07: pid 16141: LOG: node status[0]: 1 Dec 2 03:42:07 postgredbprod01 pgpool[16141]: [11-1] 2020-12-02 03:42:07: pid 16141: LOG: node status[1]: 0 Dec 2 03:42:07 postgredbprod01 pgpool[16155]: [8-1] 2020-12-02 03:42:07: pid 16155: LOG: failed to create watchdog heartbeat receive socket. Dec 2 03:42:07 postgredbprod01 pgpool[16155]: [8-2] 2020-12-02 03:42:07: pid 16155: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege Dec 2 03:42:07 postgredbprod01 pgpool[16155]: [9-1] 2020-12-02 03:42:07: pid 16155: LOG: set SO_REUSEPORT option to the socket Dec 2 03:42:07 postgredbprod01 pgpool[16155]: [10-1] 2020-12-02 03:42:07: pid 16155: LOG: creating watchdog heartbeat receive socket. Dec 2 03:42:07 postgredbprod01 pgpool[16155]: [10-2] 2020-12-02 03:42:07: pid 16155: DETAIL: set SO_REUSEPORT Dec 2 03:42:07 postgredbprod01 pgpool[16157]: [8-1] 2020-12-02 03:42:07: pid 16157: LOG: creating socket for sending heartbeat Dec 2 03:42:07 postgredbprod01 pgpool[16157]: [8-2] 2020-12-02 03:42:07: pid 16157: DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege Dec 2 03:42:07 postgredbprod01 pgpool[16157]: [9-1] 2020-12-02 03:42:07: pid 16157: LOG: set SO_REUSEPORT option to the socket Dec 2 03:42:07 postgredbprod01 pgpool[16157]: [10-1] 2020-12-02 03:42:07: pid 16157: LOG: creating socket for sending heartbeat Dec 2 03:42:07 postgredbprod01 pgpool[16157]: [10-2] 2020-12-02 03:42:07: pid 16157: DETAIL: set SO_REUSEPORT Dec 2 03:45:30 postgredbprod01 pgpool[16156]: [7-1] 2020-12-02 03:45:30: pid 16156: LOG: pool_reuse_block: blockid: 0 Dec 2 03:45:30 postgredbprod01 pgpool[16156]: [7-2] 2020-12-02 03:45:30: pid 16156: CONTEXT: while searching system catalog, When relcache is missed Dec 2 03:46:09 postgredbprod01 pgpool[16143]: [18-1] 2020-12-02 03:46:09: pid 16143: LOG: new watchdog node connection is received from "192.168.168.3:58054" Dec 2 05:13:16 postgredbprod01 pgpool[24177]: [1-1] 2020-12-02 05:13:16: pid 24177: LOG: stop request sent to pgpool. waiting for termination... Dec 2 05:13:16 postgredbprod01 pgpool[16143]: [19-1] 2020-12-02 05:13:16: pid 16143: LOG: Watchdog is shutting down Dec 2 05:30:01 postgredbprod01 pgpool[24801]: [1-1] 2020-12-02 05:30:01: pid 24801: LOG: memory cache initialized Dec 2 05:30:01 postgredbprod01 pgpool[24801]: [1-2] 2020-12-02 05:30:01: pid 24801: DETAIL: memcache blocks :64 Dec 2 05:30:01 postgredbprod01 pgpool[24801]: [2-1] 2020-12-02 05:30:01: pid 24801: LOG: pool_discard_oid_maps: discarded memqcache oid maps Dec 2 05:30:01 postgredbprod01 pgpool[24801]: [3-1] 2020-12-02 05:30:01: pid 24801: LOG: waiting for watchdog to initialize Dec 2 05:30:01 postgredbprod01 pgpool[24803]: [3-1] 2020-12-02 05:30:01: pid 24803: LOG: setting the local watchdog node name to "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 05:30:01 postgredbprod01 pgpool[24803]: [4-1] 2020-12-02 05:30:01: pid 24803: LOG: watchdog cluster is configured with 1 remote nodes Dec 2 05:30:01 postgredbprod01 pgpool[24803]: [5-1] 2020-12-02 05:30:01: pid 24803: LOG: watchdog remote node:0 on postgredbprod02:9000 Dec 2 05:30:01 postgredbprod01 pgpool[24803]: [6-1] 2020-12-02 05:30:01: pid 24803: LOG: interface monitoring is disabled in watchdog Dec 2 05:30:01 postgredbprod01 pgpool[24803]: [7-1] 2020-12-02 05:30:01: pid 24803: LOG: watchdog node state changed from [DEAD] to [LOADING] Dec 2 05:30:06 postgredbprod01 pgpool[24803]: [8-1] 2020-12-02 05:30:06: pid 24803: LOG: watchdog node state changed from [LOADING] to [JOINING] Dec 2 05:30:10 postgredbprod01 pgpool[24803]: [9-1] 2020-12-02 05:30:10: pid 24803: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Dec 2 05:30:11 postgredbprod01 pgpool[24803]: [10-1] 2020-12-02 05:30:11: pid 24803: LOG: I am the only alive node in the watchdog cluster Dec 2 05:30:11 postgredbprod01 pgpool[24803]: [10-2] 2020-12-02 05:30:11: pid 24803: HINT: skipping stand for coordinator state Dec 2 05:30:11 postgredbprod01 pgpool[24803]: [11-1] 2020-12-02 05:30:11: pid 24803: LOG: watchdog node state changed from [INITIALIZING] to [MASTER] Dec 2 05:30:11 postgredbprod01 pgpool[24803]: [12-1] 2020-12-02 05:30:11: pid 24803: LOG: I am announcing my self as master/coordinator watchdog node Dec 2 05:30:15 postgredbprod01 pgpool[24803]: [13-1] 2020-12-02 05:30:15: pid 24803: LOG: I am the cluster leader node Dec 2 05:30:15 postgredbprod01 pgpool[24803]: [13-2] 2020-12-02 05:30:15: pid 24803: DETAIL: our declare coordinator message is accepted by all nodes Dec 2 05:30:15 postgredbprod01 pgpool[24803]: [14-1] 2020-12-02 05:30:15: pid 24803: LOG: setting the local node "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" as watchdog cluster master Dec 2 05:30:15 postgredbprod01 pgpool[24803]: [15-1] 2020-12-02 05:30:15: pid 24803: LOG: I am the cluster leader node but we do not have enough nodes in cluster Dec 2 05:30:15 postgredbprod01 pgpool[24803]: [15-2] 2020-12-02 05:30:15: pid 24803: DETAIL: waiting for the quorum to start escalation process Dec 2 05:30:15 postgredbprod01 pgpool[24801]: [4-1] 2020-12-02 05:30:15: pid 24801: LOG: watchdog process is initialized Dec 2 05:30:15 postgredbprod01 pgpool[24801]: [4-2] 2020-12-02 05:30:15: pid 24801: DETAIL: watchdog messaging data version: 1.1 Dec 2 05:30:15 postgredbprod01 pgpool[24803]: [16-1] 2020-12-02 05:30:15: pid 24803: LOG: new IPC connection received Dec 2 05:30:15 postgredbprod01 pgpool[24803]: [17-1] 2020-12-02 05:30:15: pid 24803: LOG: new IPC connection received Dec 2 05:30:15 postgredbprod01 pgpool[24801]: [5-1] 2020-12-02 05:30:15: pid 24801: LOG: Setting up socket for 0.0.0.0:9999 Dec 2 05:30:15 postgredbprod01 pgpool[24801]: [6-1] 2020-12-02 05:30:15: pid 24801: LOG: Setting up socket for :::9999 Dec 2 05:30:15 postgredbprod01 pgpool[24814]: [5-1] 2020-12-02 05:30:15: pid 24814: LOG: 2 watchdog nodes are configured for lifecheck Dec 2 05:30:15 postgredbprod01 pgpool[24814]: [6-1] 2020-12-02 05:30:15: pid 24814: LOG: watchdog nodes ID:0 Name:"postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 05:30:15 postgredbprod01 pgpool[24814]: [6-2] 2020-12-02 05:30:15: pid 24814: DETAIL: Host:"postgredbprod01" WD Port:9000 pgpool-II port:9999 Dec 2 05:30:15 postgredbprod01 pgpool[24814]: [7-1] 2020-12-02 05:30:15: pid 24814: LOG: watchdog nodes ID:1 Name:"Not_Set" Dec 2 05:30:15 postgredbprod01 pgpool[24814]: [7-2] 2020-12-02 05:30:15: pid 24814: DETAIL: Host:"postgredbprod02" WD Port:9000 pgpool-II port:9999 Dec 2 05:30:15 postgredbprod01 pgpool[24814]: [8-1] 2020-12-02 05:30:15: pid 24814: LOG: watchdog lifecheck trusted server "postgredbprod02" added for the availability check Dec 2 05:30:16 postgredbprod01 pgpool[24801]: [7-1] 2020-12-02 05:30:16: pid 24801: LOG: find_primary_node_repeatedly: waiting for finding a primary node Dec 2 05:30:16 postgredbprod01 pgpool[24801]: [8-1] 2020-12-02 05:30:16: pid 24801: LOG: find_primary_node: primary node is 0 Dec 2 05:30:16 postgredbprod01 pgpool[27818]: [9-1] 2020-12-02 05:30:16: pid 27818: LOG: PCP process: 27818 started Dec 2 05:30:16 postgredbprod01 pgpool[24801]: [9-1] 2020-12-02 05:30:16: pid 24801: LOG: pgpool-II successfully started. version 4.1.5 (karasukiboshi) Dec 2 05:30:16 postgredbprod01 pgpool[24801]: [10-1] 2020-12-02 05:30:16: pid 24801: LOG: node status[0]: 1 Dec 2 05:30:16 postgredbprod01 pgpool[24801]: [11-1] 2020-12-02 05:30:16: pid 24801: LOG: node status[1]: 0 Dec 2 05:30:16 postgredbprod01 pgpool[24816]: [8-1] 2020-12-02 05:30:16: pid 24816: LOG: failed to create watchdog heartbeat receive socket. Dec 2 05:30:16 postgredbprod01 pgpool[24816]: [8-2] 2020-12-02 05:30:16: pid 24816: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege Dec 2 05:30:16 postgredbprod01 pgpool[24816]: [9-1] 2020-12-02 05:30:16: pid 24816: LOG: set SO_REUSEPORT option to the socket Dec 2 05:30:16 postgredbprod01 pgpool[24816]: [10-1] 2020-12-02 05:30:16: pid 24816: LOG: creating watchdog heartbeat receive socket. Dec 2 05:30:16 postgredbprod01 pgpool[24816]: [10-2] 2020-12-02 05:30:16: pid 24816: DETAIL: set SO_REUSEPORT Dec 2 05:30:16 postgredbprod01 pgpool[24818]: [8-1] 2020-12-02 05:30:16: pid 24818: LOG: creating socket for sending heartbeat Dec 2 05:30:16 postgredbprod01 pgpool[24818]: [8-2] 2020-12-02 05:30:16: pid 24818: DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege Dec 2 05:30:16 postgredbprod01 pgpool[24818]: [9-1] 2020-12-02 05:30:16: pid 24818: LOG: set SO_REUSEPORT option to the socket Dec 2 05:30:16 postgredbprod01 pgpool[24818]: [10-1] 2020-12-02 05:30:16: pid 24818: LOG: creating socket for sending heartbeat Dec 2 05:30:16 postgredbprod01 pgpool[24818]: [10-2] 2020-12-02 05:30:16: pid 24818: DETAIL: set SO_REUSEPORT Dec 2 05:47:17 postgredbprod01 pgpool[28833]: [1-1] 2020-12-02 05:47:17: pid 28833: LOG: stop request sent to pgpool. waiting for termination... Dec 2 05:47:17 postgredbprod01 pgpool[24803]: [18-1] 2020-12-02 05:47:17: pid 24803: LOG: Watchdog is shutting down Dec 2 05:47:35 postgredbprod01 pgpool[28852]: [1-1] 2020-12-02 05:47:35: pid 28852: LOG: memory cache initialized Dec 2 05:47:35 postgredbprod01 pgpool[28852]: [1-2] 2020-12-02 05:47:35: pid 28852: DETAIL: memcache blocks :64 Dec 2 05:47:35 postgredbprod01 pgpool[28852]: [2-1] 2020-12-02 05:47:35: pid 28852: LOG: pool_discard_oid_maps: discarded memqcache oid maps Dec 2 05:47:35 postgredbprod01 pgpool[28852]: [3-1] 2020-12-02 05:47:35: pid 28852: LOG: waiting for watchdog to initialize Dec 2 05:47:35 postgredbprod01 pgpool[28855]: [3-1] 2020-12-02 05:47:35: pid 28855: LOG: setting the local watchdog node name to "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 05:47:35 postgredbprod01 pgpool[28855]: [4-1] 2020-12-02 05:47:35: pid 28855: LOG: watchdog cluster is configured with 1 remote nodes Dec 2 05:47:35 postgredbprod01 pgpool[28855]: [5-1] 2020-12-02 05:47:35: pid 28855: LOG: watchdog remote node:0 on postgredbprod02:9000 Dec 2 05:47:35 postgredbprod01 pgpool[28855]: [6-1] 2020-12-02 05:47:35: pid 28855: LOG: interface monitoring is disabled in watchdog Dec 2 05:47:35 postgredbprod01 pgpool[28855]: [7-1] 2020-12-02 05:47:35: pid 28855: LOG: watchdog node state changed from [DEAD] to [LOADING] Dec 2 05:47:40 postgredbprod01 pgpool[28855]: [8-1] 2020-12-02 05:47:40: pid 28855: LOG: watchdog node state changed from [LOADING] to [JOINING] Dec 2 05:47:44 postgredbprod01 pgpool[28855]: [9-1] 2020-12-02 05:47:44: pid 28855: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Dec 2 05:47:45 postgredbprod01 pgpool[28855]: [10-1] 2020-12-02 05:47:45: pid 28855: LOG: I am the only alive node in the watchdog cluster Dec 2 05:47:45 postgredbprod01 pgpool[28855]: [10-2] 2020-12-02 05:47:45: pid 28855: HINT: skipping stand for coordinator state Dec 2 05:47:45 postgredbprod01 pgpool[28855]: [11-1] 2020-12-02 05:47:45: pid 28855: LOG: watchdog node state changed from [INITIALIZING] to [MASTER] Dec 2 05:47:45 postgredbprod01 pgpool[28855]: [12-1] 2020-12-02 05:47:45: pid 28855: LOG: I am announcing my self as master/coordinator watchdog node Dec 2 05:47:49 postgredbprod01 pgpool[28855]: [13-1] 2020-12-02 05:47:49: pid 28855: LOG: I am the cluster leader node Dec 2 05:47:49 postgredbprod01 pgpool[28855]: [13-2] 2020-12-02 05:47:49: pid 28855: DETAIL: our declare coordinator message is accepted by all nodes Dec 2 05:47:49 postgredbprod01 pgpool[28855]: [14-1] 2020-12-02 05:47:49: pid 28855: LOG: setting the local node "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" as watchdog cluster master Dec 2 05:47:49 postgredbprod01 pgpool[28855]: [15-1] 2020-12-02 05:47:49: pid 28855: LOG: I am the cluster leader node but we do not have enough nodes in cluster Dec 2 05:47:49 postgredbprod01 pgpool[28855]: [15-2] 2020-12-02 05:47:49: pid 28855: DETAIL: waiting for the quorum to start escalation process Dec 2 05:47:49 postgredbprod01 pgpool[28852]: [4-1] 2020-12-02 05:47:49: pid 28852: LOG: watchdog process is initialized Dec 2 05:47:49 postgredbprod01 pgpool[28852]: [4-2] 2020-12-02 05:47:49: pid 28852: DETAIL: watchdog messaging data version: 1.1 Dec 2 05:47:49 postgredbprod01 pgpool[28855]: [16-1] 2020-12-02 05:47:49: pid 28855: LOG: new IPC connection received Dec 2 05:47:49 postgredbprod01 pgpool[28855]: [17-1] 2020-12-02 05:47:49: pid 28855: LOG: new IPC connection received Dec 2 05:47:49 postgredbprod01 pgpool[28864]: [5-1] 2020-12-02 05:47:49: pid 28864: LOG: 2 watchdog nodes are configured for lifecheck Dec 2 05:47:49 postgredbprod01 pgpool[28864]: [6-1] 2020-12-02 05:47:49: pid 28864: LOG: watchdog nodes ID:0 Name:"postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 05:47:49 postgredbprod01 pgpool[28864]: [6-2] 2020-12-02 05:47:49: pid 28864: DETAIL: Host:"postgredbprod01" WD Port:9000 pgpool-II port:9999 Dec 2 05:47:49 postgredbprod01 pgpool[28864]: [7-1] 2020-12-02 05:47:49: pid 28864: LOG: watchdog nodes ID:1 Name:"Not_Set" Dec 2 05:47:49 postgredbprod01 pgpool[28864]: [7-2] 2020-12-02 05:47:49: pid 28864: DETAIL: Host:"postgredbprod02" WD Port:9000 pgpool-II port:9999 Dec 2 05:47:49 postgredbprod01 pgpool[28852]: [5-1] 2020-12-02 05:47:49: pid 28852: LOG: Setting up socket for 0.0.0.0:9999 Dec 2 05:47:49 postgredbprod01 pgpool[28852]: [6-1] 2020-12-02 05:47:49: pid 28852: LOG: Setting up socket for :::9999 Dec 2 05:47:49 postgredbprod01 pgpool[28864]: [8-1] 2020-12-02 05:47:49: pid 28864: LOG: watchdog lifecheck trusted server "postgredbprod02" added for the availability check Dec 2 05:47:50 postgredbprod01 pgpool[28852]: [7-1] 2020-12-02 05:47:50: pid 28852: LOG: find_primary_node_repeatedly: waiting for finding a primary node Dec 2 05:47:50 postgredbprod01 pgpool[28852]: [8-1] 2020-12-02 05:47:50: pid 28852: LOG: find_primary_node: primary node is 0 Dec 2 05:47:50 postgredbprod01 pgpool[31870]: [9-1] 2020-12-02 05:47:50: pid 31870: LOG: PCP process: 31870 started Dec 2 05:47:50 postgredbprod01 pgpool[28852]: [9-1] 2020-12-02 05:47:50: pid 28852: LOG: pgpool-II successfully started. version 4.1.5 (karasukiboshi) Dec 2 05:47:50 postgredbprod01 pgpool[28852]: [10-1] 2020-12-02 05:47:50: pid 28852: LOG: node status[0]: 1 Dec 2 05:47:50 postgredbprod01 pgpool[28852]: [11-1] 2020-12-02 05:47:50: pid 28852: LOG: node status[1]: 0 Dec 2 05:47:50 postgredbprod01 pgpool[28865]: [8-1] 2020-12-02 05:47:50: pid 28865: LOG: failed to create watchdog heartbeat receive socket. Dec 2 05:47:50 postgredbprod01 pgpool[28865]: [8-2] 2020-12-02 05:47:50: pid 28865: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege Dec 2 05:47:50 postgredbprod01 pgpool[28865]: [9-1] 2020-12-02 05:47:50: pid 28865: LOG: set SO_REUSEPORT option to the socket Dec 2 05:47:50 postgredbprod01 pgpool[28865]: [10-1] 2020-12-02 05:47:50: pid 28865: LOG: creating watchdog heartbeat receive socket. Dec 2 05:47:50 postgredbprod01 pgpool[28865]: [10-2] 2020-12-02 05:47:50: pid 28865: DETAIL: set SO_REUSEPORT Dec 2 05:47:50 postgredbprod01 pgpool[28867]: [8-1] 2020-12-02 05:47:50: pid 28867: LOG: creating socket for sending heartbeat Dec 2 05:47:50 postgredbprod01 pgpool[28867]: [8-2] 2020-12-02 05:47:50: pid 28867: DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege Dec 2 05:47:50 postgredbprod01 pgpool[28867]: [9-1] 2020-12-02 05:47:50: pid 28867: LOG: set SO_REUSEPORT option to the socket Dec 2 05:47:50 postgredbprod01 pgpool[28867]: [10-1] 2020-12-02 05:47:50: pid 28867: LOG: creating socket for sending heartbeat Dec 2 05:47:50 postgredbprod01 pgpool[28867]: [10-2] 2020-12-02 05:47:50: pid 28867: DETAIL: set SO_REUSEPORT Dec 2 05:49:57 postgredbprod01 pgpool[31996]: [1-1] 2020-12-02 05:49:57: pid 31996: LOG: stop request sent to pgpool. waiting for termination... Dec 2 05:49:57 postgredbprod01 pgpool[28855]: [18-1] 2020-12-02 05:49:57: pid 28855: LOG: Watchdog is shutting down Dec 2 06:24:15 postgredbprod01 pgpool[979]: [1-1] 2020-12-02 06:24:15: pid 979: LOG: memory cache initialized Dec 2 06:24:15 postgredbprod01 pgpool[979]: [1-2] 2020-12-02 06:24:15: pid 979: DETAIL: memcache blocks :64 Dec 2 06:24:15 postgredbprod01 pgpool[979]: [2-1] 2020-12-02 06:24:15: pid 979: LOG: pool_discard_oid_maps: discarded memqcache oid maps Dec 2 06:24:15 postgredbprod01 pgpool[979]: [3-1] 2020-12-02 06:24:15: pid 979: LOG: waiting for watchdog to initialize Dec 2 06:24:15 postgredbprod01 pgpool[981]: [3-1] 2020-12-02 06:24:15: pid 981: LOG: setting the local watchdog node name to "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 06:24:15 postgredbprod01 pgpool[981]: [4-1] 2020-12-02 06:24:15: pid 981: LOG: watchdog cluster is configured with 1 remote nodes Dec 2 06:24:15 postgredbprod01 pgpool[981]: [5-1] 2020-12-02 06:24:15: pid 981: LOG: watchdog remote node:0 on postgredbprod02:9000 Dec 2 06:24:15 postgredbprod01 pgpool[981]: [6-1] 2020-12-02 06:24:15: pid 981: LOG: interface monitoring is disabled in watchdog Dec 2 06:24:15 postgredbprod01 pgpool[981]: [7-1] 2020-12-02 06:24:15: pid 981: LOG: watchdog node state changed from [DEAD] to [LOADING] Dec 2 06:24:20 postgredbprod01 pgpool[981]: [8-1] 2020-12-02 06:24:20: pid 981: LOG: watchdog node state changed from [LOADING] to [JOINING] Dec 2 06:24:24 postgredbprod01 pgpool[981]: [9-1] 2020-12-02 06:24:24: pid 981: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Dec 2 06:24:25 postgredbprod01 pgpool[981]: [10-1] 2020-12-02 06:24:25: pid 981: LOG: I am the only alive node in the watchdog cluster Dec 2 06:24:25 postgredbprod01 pgpool[981]: [10-2] 2020-12-02 06:24:25: pid 981: HINT: skipping stand for coordinator state Dec 2 06:24:25 postgredbprod01 pgpool[981]: [11-1] 2020-12-02 06:24:25: pid 981: LOG: watchdog node state changed from [INITIALIZING] to [MASTER] Dec 2 06:24:25 postgredbprod01 pgpool[981]: [12-1] 2020-12-02 06:24:25: pid 981: LOG: I am announcing my self as master/coordinator watchdog node Dec 2 06:24:29 postgredbprod01 pgpool[981]: [13-1] 2020-12-02 06:24:29: pid 981: LOG: I am the cluster leader node Dec 2 06:24:29 postgredbprod01 pgpool[981]: [13-2] 2020-12-02 06:24:29: pid 981: DETAIL: our declare coordinator message is accepted by all nodes Dec 2 06:24:29 postgredbprod01 pgpool[981]: [14-1] 2020-12-02 06:24:29: pid 981: LOG: setting the local node "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" as watchdog cluster master Dec 2 06:24:29 postgredbprod01 pgpool[981]: [15-1] 2020-12-02 06:24:29: pid 981: LOG: I am the cluster leader node but we do not have enough nodes in cluster Dec 2 06:24:29 postgredbprod01 pgpool[981]: [15-2] 2020-12-02 06:24:29: pid 981: DETAIL: waiting for the quorum to start escalation process Dec 2 06:24:29 postgredbprod01 pgpool[979]: [4-1] 2020-12-02 06:24:29: pid 979: LOG: watchdog process is initialized Dec 2 06:24:29 postgredbprod01 pgpool[979]: [4-2] 2020-12-02 06:24:29: pid 979: DETAIL: watchdog messaging data version: 1.1 Dec 2 06:24:29 postgredbprod01 pgpool[981]: [16-1] 2020-12-02 06:24:29: pid 981: LOG: new IPC connection received Dec 2 06:24:29 postgredbprod01 pgpool[981]: [17-1] 2020-12-02 06:24:29: pid 981: LOG: new IPC connection received Dec 2 06:24:29 postgredbprod01 pgpool[979]: [5-1] 2020-12-02 06:24:29: pid 979: LOG: Setting up socket for 0.0.0.0:9999 Dec 2 06:24:29 postgredbprod01 pgpool[979]: [6-1] 2020-12-02 06:24:29: pid 979: LOG: Setting up socket for :::9999 Dec 2 06:24:29 postgredbprod01 pgpool[990]: [5-1] 2020-12-02 06:24:29: pid 990: LOG: 2 watchdog nodes are configured for lifecheck Dec 2 06:24:29 postgredbprod01 pgpool[990]: [6-1] 2020-12-02 06:24:29: pid 990: LOG: watchdog nodes ID:0 Name:"postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 06:24:29 postgredbprod01 pgpool[990]: [6-2] 2020-12-02 06:24:29: pid 990: DETAIL: Host:"postgredbprod01" WD Port:9000 pgpool-II port:9999 Dec 2 06:24:29 postgredbprod01 pgpool[990]: [7-1] 2020-12-02 06:24:29: pid 990: LOG: watchdog nodes ID:1 Name:"Not_Set" Dec 2 06:24:29 postgredbprod01 pgpool[990]: [7-2] 2020-12-02 06:24:29: pid 990: DETAIL: Host:"postgredbprod02" WD Port:9000 pgpool-II port:9999 Dec 2 06:24:29 postgredbprod01 pgpool[990]: [8-1] 2020-12-02 06:24:29: pid 990: LOG: watchdog lifecheck trusted server "postgredbprod02" added for the availability check Dec 2 06:24:30 postgredbprod01 pgpool[979]: [7-1] 2020-12-02 06:24:30: pid 979: LOG: find_primary_node_repeatedly: waiting for finding a primary node Dec 2 06:24:30 postgredbprod01 pgpool[979]: [8-1] 2020-12-02 06:24:30: pid 979: LOG: find_primary_node: primary node is 0 Dec 2 06:24:30 postgredbprod01 pgpool[4007]: [9-1] 2020-12-02 06:24:30: pid 4007: LOG: PCP process: 4007 started Dec 2 06:24:30 postgredbprod01 pgpool[979]: [9-1] 2020-12-02 06:24:30: pid 979: LOG: pgpool-II successfully started. version 4.1.5 (karasukiboshi) Dec 2 06:24:30 postgredbprod01 pgpool[979]: [10-1] 2020-12-02 06:24:30: pid 979: LOG: node status[0]: 1 Dec 2 06:24:30 postgredbprod01 pgpool[979]: [11-1] 2020-12-02 06:24:30: pid 979: LOG: node status[1]: 0 Dec 2 06:24:30 postgredbprod01 pgpool[992]: [8-1] 2020-12-02 06:24:30: pid 992: LOG: failed to create watchdog heartbeat receive socket. Dec 2 06:24:30 postgredbprod01 pgpool[992]: [8-2] 2020-12-02 06:24:30: pid 992: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege Dec 2 06:24:30 postgredbprod01 pgpool[992]: [9-1] 2020-12-02 06:24:30: pid 992: LOG: set SO_REUSEPORT option to the socket Dec 2 06:24:30 postgredbprod01 pgpool[992]: [10-1] 2020-12-02 06:24:30: pid 992: LOG: creating watchdog heartbeat receive socket. Dec 2 06:24:30 postgredbprod01 pgpool[992]: [10-2] 2020-12-02 06:24:30: pid 992: DETAIL: set SO_REUSEPORT Dec 2 06:24:30 postgredbprod01 pgpool[994]: [8-1] 2020-12-02 06:24:30: pid 994: LOG: creating socket for sending heartbeat Dec 2 06:24:30 postgredbprod01 pgpool[994]: [8-2] 2020-12-02 06:24:30: pid 994: DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege Dec 2 06:24:30 postgredbprod01 pgpool[994]: [9-1] 2020-12-02 06:24:30: pid 994: LOG: set SO_REUSEPORT option to the socket Dec 2 06:24:30 postgredbprod01 pgpool[994]: [10-1] 2020-12-02 06:24:30: pid 994: LOG: creating socket for sending heartbeat Dec 2 06:24:30 postgredbprod01 pgpool[994]: [10-2] 2020-12-02 06:24:30: pid 994: DETAIL: set SO_REUSEPORT Dec 2 10:44:44 postgredbprod01 pgpool[18339]: [1-1] 2020-12-02 10:44:44: pid 18339: LOG: stop request sent to pgpool. waiting for termination... Dec 2 10:44:44 postgredbprod01 pgpool[981]: [18-1] 2020-12-02 10:44:44: pid 981: LOG: Watchdog is shutting down Dec 2 10:44:57 postgredbprod01 pgpool[18356]: [1-1] 2020-12-02 10:44:57: pid 18356: LOG: memory cache initialized Dec 2 10:44:57 postgredbprod01 pgpool[18356]: [1-2] 2020-12-02 10:44:57: pid 18356: DETAIL: memcache blocks :64 Dec 2 10:44:57 postgredbprod01 pgpool[18356]: [2-1] 2020-12-02 10:44:57: pid 18356: LOG: pool_discard_oid_maps: discarded memqcache oid maps Dec 2 10:44:57 postgredbprod01 pgpool[18356]: [3-1] 2020-12-02 10:44:57: pid 18356: LOG: waiting for watchdog to initialize Dec 2 10:44:57 postgredbprod01 pgpool[18358]: [3-1] 2020-12-02 10:44:57: pid 18358: LOG: setting the local watchdog node name to "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 10:44:57 postgredbprod01 pgpool[18358]: [4-1] 2020-12-02 10:44:57: pid 18358: LOG: watchdog cluster is configured with 1 remote nodes Dec 2 10:44:57 postgredbprod01 pgpool[18358]: [5-1] 2020-12-02 10:44:57: pid 18358: LOG: watchdog remote node:0 on postgredbprod02:9000 Dec 2 10:44:57 postgredbprod01 pgpool[18358]: [6-1] 2020-12-02 10:44:57: pid 18358: LOG: interface monitoring is disabled in watchdog Dec 2 10:44:57 postgredbprod01 pgpool[18358]: [7-1] 2020-12-02 10:44:57: pid 18358: LOG: watchdog node state changed from [DEAD] to [LOADING] Dec 2 10:45:02 postgredbprod01 pgpool[18358]: [8-1] 2020-12-02 10:45:02: pid 18358: LOG: watchdog node state changed from [LOADING] to [JOINING] Dec 2 10:45:06 postgredbprod01 pgpool[18358]: [9-1] 2020-12-02 10:45:06: pid 18358: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Dec 2 10:45:07 postgredbprod01 pgpool[18358]: [10-1] 2020-12-02 10:45:07: pid 18358: LOG: I am the only alive node in the watchdog cluster Dec 2 10:45:07 postgredbprod01 pgpool[18358]: [10-2] 2020-12-02 10:45:07: pid 18358: HINT: skipping stand for coordinator state Dec 2 10:45:07 postgredbprod01 pgpool[18358]: [11-1] 2020-12-02 10:45:07: pid 18358: LOG: watchdog node state changed from [INITIALIZING] to [MASTER] Dec 2 10:45:07 postgredbprod01 pgpool[18358]: [12-1] 2020-12-02 10:45:07: pid 18358: LOG: I am announcing my self as master/coordinator watchdog node Dec 2 10:45:11 postgredbprod01 pgpool[18358]: [13-1] 2020-12-02 10:45:11: pid 18358: LOG: I am the cluster leader node Dec 2 10:45:11 postgredbprod01 pgpool[18358]: [13-2] 2020-12-02 10:45:11: pid 18358: DETAIL: our declare coordinator message is accepted by all nodes Dec 2 10:45:11 postgredbprod01 pgpool[18358]: [14-1] 2020-12-02 10:45:11: pid 18358: LOG: setting the local node "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" as watchdog cluster master Dec 2 10:45:11 postgredbprod01 pgpool[18358]: [15-1] 2020-12-02 10:45:11: pid 18358: LOG: I am the cluster leader node but we do not have enough nodes in cluster Dec 2 10:45:11 postgredbprod01 pgpool[18358]: [15-2] 2020-12-02 10:45:11: pid 18358: DETAIL: waiting for the quorum to start escalation process Dec 2 10:45:11 postgredbprod01 pgpool[18356]: [4-1] 2020-12-02 10:45:11: pid 18356: LOG: watchdog process is initialized Dec 2 10:45:11 postgredbprod01 pgpool[18356]: [4-2] 2020-12-02 10:45:11: pid 18356: DETAIL: watchdog messaging data version: 1.1 Dec 2 10:45:11 postgredbprod01 pgpool[18358]: [16-1] 2020-12-02 10:45:11: pid 18358: LOG: new IPC connection received Dec 2 10:45:11 postgredbprod01 pgpool[18358]: [17-1] 2020-12-02 10:45:11: pid 18358: LOG: new IPC connection received Dec 2 10:45:11 postgredbprod01 pgpool[18356]: [5-1] 2020-12-02 10:45:11: pid 18356: LOG: Setting up socket for 0.0.0.0:9999 Dec 2 10:45:11 postgredbprod01 pgpool[18356]: [6-1] 2020-12-02 10:45:11: pid 18356: LOG: Setting up socket for :::9999 Dec 2 10:45:11 postgredbprod01 pgpool[18370]: [5-1] 2020-12-02 10:45:11: pid 18370: LOG: 2 watchdog nodes are configured for lifecheck Dec 2 10:45:11 postgredbprod01 pgpool[18370]: [6-1] 2020-12-02 10:45:11: pid 18370: LOG: watchdog nodes ID:0 Name:"postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 10:45:11 postgredbprod01 pgpool[18370]: [6-2] 2020-12-02 10:45:11: pid 18370: DETAIL: Host:"postgredbprod01" WD Port:9000 pgpool-II port:9999 Dec 2 10:45:11 postgredbprod01 pgpool[18370]: [7-1] 2020-12-02 10:45:11: pid 18370: LOG: watchdog nodes ID:1 Name:"Not_Set" Dec 2 10:45:11 postgredbprod01 pgpool[18370]: [7-2] 2020-12-02 10:45:11: pid 18370: DETAIL: Host:"postgredbprod02" WD Port:9000 pgpool-II port:9999 Dec 2 10:45:11 postgredbprod01 pgpool[18370]: [8-1] 2020-12-02 10:45:11: pid 18370: LOG: watchdog lifecheck trusted server "postgredbprod02" added for the availability check Dec 2 10:45:12 postgredbprod01 pgpool[18356]: [7-1] 2020-12-02 10:45:12: pid 18356: LOG: find_primary_node_repeatedly: waiting for finding a primary node Dec 2 10:45:12 postgredbprod01 pgpool[18356]: [8-1] 2020-12-02 10:45:12: pid 18356: LOG: find_primary_node: primary node is 0 Dec 2 10:45:12 postgredbprod01 pgpool[21374]: [9-1] 2020-12-02 10:45:12: pid 21374: LOG: PCP process: 21374 started Dec 2 10:45:12 postgredbprod01 pgpool[18356]: [9-1] 2020-12-02 10:45:12: pid 18356: LOG: pgpool-II successfully started. version 4.1.5 (karasukiboshi) Dec 2 10:45:12 postgredbprod01 pgpool[18356]: [10-1] 2020-12-02 10:45:12: pid 18356: LOG: node status[0]: 1 Dec 2 10:45:12 postgredbprod01 pgpool[18356]: [11-1] 2020-12-02 10:45:12: pid 18356: LOG: node status[1]: 0 Dec 2 10:45:12 postgredbprod01 pgpool[18372]: [8-1] 2020-12-02 10:45:12: pid 18372: LOG: failed to create watchdog heartbeat receive socket. Dec 2 10:45:12 postgredbprod01 pgpool[18372]: [8-2] 2020-12-02 10:45:12: pid 18372: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege Dec 2 10:45:12 postgredbprod01 pgpool[18372]: [9-1] 2020-12-02 10:45:12: pid 18372: LOG: set SO_REUSEPORT option to the socket Dec 2 10:45:12 postgredbprod01 pgpool[18372]: [10-1] 2020-12-02 10:45:12: pid 18372: LOG: creating watchdog heartbeat receive socket. Dec 2 10:45:12 postgredbprod01 pgpool[18372]: [10-2] 2020-12-02 10:45:12: pid 18372: DETAIL: set SO_REUSEPORT Dec 2 10:45:12 postgredbprod01 pgpool[18374]: [8-1] 2020-12-02 10:45:12: pid 18374: LOG: creating socket for sending heartbeat Dec 2 10:45:12 postgredbprod01 pgpool[18374]: [8-2] 2020-12-02 10:45:12: pid 18374: DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege Dec 2 10:45:12 postgredbprod01 pgpool[18374]: [9-1] 2020-12-02 10:45:12: pid 18374: LOG: set SO_REUSEPORT option to the socket Dec 2 10:45:12 postgredbprod01 pgpool[18374]: [10-1] 2020-12-02 10:45:12: pid 18374: LOG: creating socket for sending heartbeat Dec 2 10:45:12 postgredbprod01 pgpool[18374]: [10-2] 2020-12-02 10:45:12: pid 18374: DETAIL: set SO_REUSEPORT Dec 2 14:44:18 postgredbprod01 pgpool[2156]: [1-1] 2020-12-02 14:44:18: pid 2156: LOG: stop request sent to pgpool. waiting for termination... Dec 2 14:44:18 postgredbprod01 pgpool[18358]: [18-1] 2020-12-02 14:44:18: pid 18358: LOG: Watchdog is shutting down Dec 2 14:46:06 postgredbprod01 pgpool[2276]: [1-1] 2020-12-02 14:46:06: pid 2276: LOG: memory cache initialized Dec 2 14:46:06 postgredbprod01 pgpool[2276]: [1-2] 2020-12-02 14:46:06: pid 2276: DETAIL: memcache blocks :64 Dec 2 14:46:06 postgredbprod01 pgpool[2276]: [2-1] 2020-12-02 14:46:06: pid 2276: LOG: pool_discard_oid_maps: discarded memqcache oid maps Dec 2 14:46:06 postgredbprod01 pgpool[2276]: [3-1] 2020-12-02 14:46:06: pid 2276: LOG: waiting for watchdog to initialize Dec 2 14:46:06 postgredbprod01 pgpool[2278]: [3-1] 2020-12-02 14:46:06: pid 2278: LOG: setting the local watchdog node name to "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 14:46:06 postgredbprod01 pgpool[2278]: [4-1] 2020-12-02 14:46:06: pid 2278: LOG: watchdog cluster is configured with 1 remote nodes Dec 2 14:46:06 postgredbprod01 pgpool[2278]: [5-1] 2020-12-02 14:46:06: pid 2278: LOG: watchdog remote node:0 on postgredbprod02:9000 Dec 2 14:46:06 postgredbprod01 pgpool[2278]: [6-1] 2020-12-02 14:46:06: pid 2278: LOG: interface monitoring is disabled in watchdog Dec 2 14:46:06 postgredbprod01 pgpool[2278]: [7-1] 2020-12-02 14:46:06: pid 2278: LOG: watchdog node state changed from [DEAD] to [LOADING] Dec 2 14:46:11 postgredbprod01 pgpool[2278]: [8-1] 2020-12-02 14:46:11: pid 2278: LOG: watchdog node state changed from [LOADING] to [JOINING] Dec 2 14:46:15 postgredbprod01 pgpool[2278]: [9-1] 2020-12-02 14:46:15: pid 2278: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Dec 2 14:46:16 postgredbprod01 pgpool[2278]: [10-1] 2020-12-02 14:46:16: pid 2278: LOG: I am the only alive node in the watchdog cluster Dec 2 14:46:16 postgredbprod01 pgpool[2278]: [10-2] 2020-12-02 14:46:16: pid 2278: HINT: skipping stand for coordinator state Dec 2 14:46:16 postgredbprod01 pgpool[2278]: [11-1] 2020-12-02 14:46:16: pid 2278: LOG: watchdog node state changed from [INITIALIZING] to [MASTER] Dec 2 14:46:16 postgredbprod01 pgpool[2278]: [12-1] 2020-12-02 14:46:16: pid 2278: LOG: I am announcing my self as master/coordinator watchdog node Dec 2 14:46:20 postgredbprod01 pgpool[2278]: [13-1] 2020-12-02 14:46:20: pid 2278: LOG: I am the cluster leader node Dec 2 14:46:20 postgredbprod01 pgpool[2278]: [13-2] 2020-12-02 14:46:20: pid 2278: DETAIL: our declare coordinator message is accepted by all nodes Dec 2 14:46:20 postgredbprod01 pgpool[2278]: [14-1] 2020-12-02 14:46:20: pid 2278: LOG: setting the local node "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" as watchdog cluster master Dec 2 14:46:20 postgredbprod01 pgpool[2278]: [15-1] 2020-12-02 14:46:20: pid 2278: LOG: I am the cluster leader node but we do not have enough nodes in cluster Dec 2 14:46:20 postgredbprod01 pgpool[2278]: [15-2] 2020-12-02 14:46:20: pid 2278: DETAIL: waiting for the quorum to start escalation process Dec 2 14:46:20 postgredbprod01 pgpool[2276]: [4-1] 2020-12-02 14:46:20: pid 2276: LOG: watchdog process is initialized Dec 2 14:46:20 postgredbprod01 pgpool[2276]: [4-2] 2020-12-02 14:46:20: pid 2276: DETAIL: watchdog messaging data version: 1.1 Dec 2 14:46:20 postgredbprod01 pgpool[2278]: [16-1] 2020-12-02 14:46:20: pid 2278: LOG: new IPC connection received Dec 2 14:46:20 postgredbprod01 pgpool[2278]: [17-1] 2020-12-02 14:46:20: pid 2278: LOG: new IPC connection received Dec 2 14:46:20 postgredbprod01 pgpool[2287]: [5-1] 2020-12-02 14:46:20: pid 2287: LOG: 2 watchdog nodes are configured for lifecheck Dec 2 14:46:20 postgredbprod01 pgpool[2287]: [6-1] 2020-12-02 14:46:20: pid 2287: LOG: watchdog nodes ID:0 Name:"postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 14:46:20 postgredbprod01 pgpool[2287]: [6-2] 2020-12-02 14:46:20: pid 2287: DETAIL: Host:"postgredbprod01" WD Port:9000 pgpool-II port:9999 Dec 2 14:46:20 postgredbprod01 pgpool[2287]: [7-1] 2020-12-02 14:46:20: pid 2287: LOG: watchdog nodes ID:1 Name:"Not_Set" Dec 2 14:46:20 postgredbprod01 pgpool[2287]: [7-2] 2020-12-02 14:46:20: pid 2287: DETAIL: Host:"postgredbprod02" WD Port:9000 pgpool-II port:9999 Dec 2 14:46:20 postgredbprod01 pgpool[2276]: [5-1] 2020-12-02 14:46:20: pid 2276: LOG: Setting up socket for 0.0.0.0:9999 Dec 2 14:46:20 postgredbprod01 pgpool[2276]: [6-1] 2020-12-02 14:46:20: pid 2276: LOG: Setting up socket for :::9999 Dec 2 14:46:20 postgredbprod01 pgpool[2287]: [8-1] 2020-12-02 14:46:20: pid 2287: LOG: watchdog lifecheck trusted server "postgredbprod02" added for the availability check Dec 2 14:46:21 postgredbprod01 pgpool[2276]: [7-1] 2020-12-02 14:46:21: pid 2276: LOG: find_primary_node_repeatedly: waiting for finding a primary node Dec 2 14:46:21 postgredbprod01 pgpool[2276]: [8-1] 2020-12-02 14:46:21: pid 2276: LOG: find_primary_node: primary node is 0 Dec 2 14:46:21 postgredbprod01 pgpool[5295]: [9-1] 2020-12-02 14:46:21: pid 5295: LOG: PCP process: 5295 started Dec 2 14:46:21 postgredbprod01 pgpool[2276]: [9-1] 2020-12-02 14:46:21: pid 2276: LOG: pgpool-II successfully started. version 4.1.5 (karasukiboshi) Dec 2 14:46:21 postgredbprod01 pgpool[2276]: [10-1] 2020-12-02 14:46:21: pid 2276: LOG: node status[0]: 1 Dec 2 14:46:21 postgredbprod01 pgpool[2276]: [11-1] 2020-12-02 14:46:21: pid 2276: LOG: node status[1]: 0 Dec 2 14:46:21 postgredbprod01 pgpool[2288]: [8-1] 2020-12-02 14:46:21: pid 2288: LOG: failed to create watchdog heartbeat receive socket. Dec 2 14:46:21 postgredbprod01 pgpool[2288]: [8-2] 2020-12-02 14:46:21: pid 2288: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege Dec 2 14:46:21 postgredbprod01 pgpool[2288]: [9-1] 2020-12-02 14:46:21: pid 2288: LOG: set SO_REUSEPORT option to the socket Dec 2 14:46:21 postgredbprod01 pgpool[2288]: [10-1] 2020-12-02 14:46:21: pid 2288: LOG: creating watchdog heartbeat receive socket. Dec 2 14:46:21 postgredbprod01 pgpool[2288]: [10-2] 2020-12-02 14:46:21: pid 2288: DETAIL: set SO_REUSEPORT Dec 2 14:46:21 postgredbprod01 pgpool[2289]: [8-1] 2020-12-02 14:46:21: pid 2289: LOG: creating socket for sending heartbeat Dec 2 14:46:21 postgredbprod01 pgpool[2289]: [8-2] 2020-12-02 14:46:21: pid 2289: DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege Dec 2 14:46:21 postgredbprod01 pgpool[2289]: [9-1] 2020-12-02 14:46:21: pid 2289: LOG: set SO_REUSEPORT option to the socket Dec 2 14:46:21 postgredbprod01 pgpool[2289]: [10-1] 2020-12-02 14:46:21: pid 2289: LOG: creating socket for sending heartbeat Dec 2 14:46:21 postgredbprod01 pgpool[2289]: [10-2] 2020-12-02 14:46:21: pid 2289: DETAIL: set SO_REUSEPORT Dec 2 14:47:23 postgredbprod01 pgpool[5362]: [1-1] 2020-12-02 14:47:23: pid 5362: LOG: stop request sent to pgpool. waiting for termination... Dec 2 14:47:23 postgredbprod01 pgpool[2278]: [18-1] 2020-12-02 14:47:23: pid 2278: LOG: Watchdog is shutting down Dec 2 18:44:38 postgredbprod01 pgpool[18237]: [1-1] 2020-12-02 18:44:38: pid 18237: LOG: memory cache initialized Dec 2 18:44:38 postgredbprod01 pgpool[18237]: [1-2] 2020-12-02 18:44:38: pid 18237: DETAIL: memcache blocks :64 Dec 2 18:44:38 postgredbprod01 pgpool[18237]: [2-1] 2020-12-02 18:44:38: pid 18237: LOG: pool_discard_oid_maps: discarded memqcache oid maps Dec 2 18:44:38 postgredbprod01 pgpool[18237]: [3-1] 2020-12-02 18:44:38: pid 18237: LOG: waiting for watchdog to initialize Dec 2 18:44:38 postgredbprod01 pgpool[18239]: [3-1] 2020-12-02 18:44:38: pid 18239: LOG: setting the local watchdog node name to "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 18:44:38 postgredbprod01 pgpool[18239]: [4-1] 2020-12-02 18:44:38: pid 18239: LOG: watchdog cluster is configured with 1 remote nodes Dec 2 18:44:38 postgredbprod01 pgpool[18239]: [5-1] 2020-12-02 18:44:38: pid 18239: LOG: watchdog remote node:0 on postgredbprod02:9000 Dec 2 18:44:38 postgredbprod01 pgpool[18239]: [6-1] 2020-12-02 18:44:38: pid 18239: LOG: interface monitoring is disabled in watchdog Dec 2 18:44:38 postgredbprod01 pgpool[18239]: [7-1] 2020-12-02 18:44:38: pid 18239: LOG: watchdog node state changed from [DEAD] to [LOADING] Dec 2 18:44:43 postgredbprod01 pgpool[18239]: [8-1] 2020-12-02 18:44:43: pid 18239: LOG: watchdog node state changed from [LOADING] to [JOINING] Dec 2 18:44:47 postgredbprod01 pgpool[18239]: [9-1] 2020-12-02 18:44:47: pid 18239: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Dec 2 18:44:48 postgredbprod01 pgpool[18239]: [10-1] 2020-12-02 18:44:48: pid 18239: LOG: I am the only alive node in the watchdog cluster Dec 2 18:44:48 postgredbprod01 pgpool[18239]: [10-2] 2020-12-02 18:44:48: pid 18239: HINT: skipping stand for coordinator state Dec 2 18:44:48 postgredbprod01 pgpool[18239]: [11-1] 2020-12-02 18:44:48: pid 18239: LOG: watchdog node state changed from [INITIALIZING] to [MASTER] Dec 2 18:44:48 postgredbprod01 pgpool[18239]: [12-1] 2020-12-02 18:44:48: pid 18239: LOG: I am announcing my self as master/coordinator watchdog node Dec 2 18:44:52 postgredbprod01 pgpool[18239]: [13-1] 2020-12-02 18:44:52: pid 18239: LOG: I am the cluster leader node Dec 2 18:44:52 postgredbprod01 pgpool[18239]: [13-2] 2020-12-02 18:44:52: pid 18239: DETAIL: our declare coordinator message is accepted by all nodes Dec 2 18:44:52 postgredbprod01 pgpool[18239]: [14-1] 2020-12-02 18:44:52: pid 18239: LOG: setting the local node "postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" as watchdog cluster master Dec 2 18:44:52 postgredbprod01 pgpool[18239]: [15-1] 2020-12-02 18:44:52: pid 18239: LOG: I am the cluster leader node but we do not have enough nodes in cluster Dec 2 18:44:52 postgredbprod01 pgpool[18239]: [15-2] 2020-12-02 18:44:52: pid 18239: DETAIL: waiting for the quorum to start escalation process Dec 2 18:44:52 postgredbprod01 pgpool[18237]: [4-1] 2020-12-02 18:44:52: pid 18237: LOG: watchdog process is initialized Dec 2 18:44:52 postgredbprod01 pgpool[18237]: [4-2] 2020-12-02 18:44:52: pid 18237: DETAIL: watchdog messaging data version: 1.1 Dec 2 18:44:52 postgredbprod01 pgpool[18239]: [16-1] 2020-12-02 18:44:52: pid 18239: LOG: new IPC connection received Dec 2 18:44:52 postgredbprod01 pgpool[18239]: [17-1] 2020-12-02 18:44:52: pid 18239: LOG: new IPC connection received Dec 2 18:44:52 postgredbprod01 pgpool[18237]: [5-1] 2020-12-02 18:44:52: pid 18237: LOG: Setting up socket for 0.0.0.0:9999 Dec 2 18:44:52 postgredbprod01 pgpool[18237]: [6-1] 2020-12-02 18:44:52: pid 18237: LOG: Setting up socket for :::9999 Dec 2 18:44:52 postgredbprod01 pgpool[18253]: [5-1] 2020-12-02 18:44:52: pid 18253: LOG: 2 watchdog nodes are configured for lifecheck Dec 2 18:44:52 postgredbprod01 pgpool[18253]: [6-1] 2020-12-02 18:44:52: pid 18253: LOG: watchdog nodes ID:0 Name:"postgredbprod01:9999 Linux postgredbprod01.nodc.bppt.go.id" Dec 2 18:44:52 postgredbprod01 pgpool[18253]: [6-2] 2020-12-02 18:44:52: pid 18253: DETAIL: Host:"postgredbprod01" WD Port:9000 pgpool-II port:9999 Dec 2 18:44:52 postgredbprod01 pgpool[18253]: [7-1] 2020-12-02 18:44:52: pid 18253: LOG: watchdog nodes ID:1 Name:"Not_Set" Dec 2 18:44:52 postgredbprod01 pgpool[18253]: [7-2] 2020-12-02 18:44:52: pid 18253: DETAIL: Host:"postgredbprod02" WD Port:9000 pgpool-II port:9999 Dec 2 18:44:52 postgredbprod01 pgpool[18253]: [8-1] 2020-12-02 18:44:52: pid 18253: LOG: watchdog lifecheck trusted server "postgredbprod02" added for the availability check Dec 2 18:44:53 postgredbprod01 pgpool[18237]: [7-1] 2020-12-02 18:44:53: pid 18237: LOG: find_primary_node_repeatedly: waiting for finding a primary node Dec 2 18:44:53 postgredbprod01 pgpool[18237]: [8-1] 2020-12-02 18:44:53: pid 18237: LOG: find_primary_node: primary node is 0 Dec 2 18:44:53 postgredbprod01 pgpool[21257]: [9-1] 2020-12-02 18:44:53: pid 21257: LOG: PCP process: 21257 started Dec 2 18:44:53 postgredbprod01 pgpool[18237]: [9-1] 2020-12-02 18:44:53: pid 18237: LOG: pgpool-II successfully started. version 4.1.5 (karasukiboshi) Dec 2 18:44:53 postgredbprod01 pgpool[18237]: [10-1] 2020-12-02 18:44:53: pid 18237: LOG: node status[0]: 1 Dec 2 18:44:53 postgredbprod01 pgpool[18237]: [11-1] 2020-12-02 18:44:53: pid 18237: LOG: node status[1]: 0 Dec 2 18:44:53 postgredbprod01 pgpool[18255]: [8-1] 2020-12-02 18:44:53: pid 18255: LOG: failed to create watchdog heartbeat receive socket. Dec 2 18:44:53 postgredbprod01 pgpool[18255]: [8-2] 2020-12-02 18:44:53: pid 18255: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege Dec 2 18:44:53 postgredbprod01 pgpool[18255]: [9-1] 2020-12-02 18:44:53: pid 18255: LOG: set SO_REUSEPORT option to the socket Dec 2 18:44:53 postgredbprod01 pgpool[18255]: [10-1] 2020-12-02 18:44:53: pid 18255: LOG: creating watchdog heartbeat receive socket. Dec 2 18:44:53 postgredbprod01 pgpool[18255]: [10-2] 2020-12-02 18:44:53: pid 18255: DETAIL: set SO_REUSEPORT Dec 2 18:44:53 postgredbprod01 pgpool[18257]: [8-1] 2020-12-02 18:44:53: pid 18257: LOG: creating socket for sending heartbeat Dec 2 18:44:53 postgredbprod01 pgpool[18257]: [8-2] 2020-12-02 18:44:53: pid 18257: DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege Dec 2 18:44:53 postgredbprod01 pgpool[18257]: [9-1] 2020-12-02 18:44:53: pid 18257: LOG: set SO_REUSEPORT option to the socket Dec 2 18:44:53 postgredbprod01 pgpool[18257]: [10-1] 2020-12-02 18:44:53: pid 18257: LOG: creating socket for sending heartbeat Dec 2 18:44:53 postgredbprod01 pgpool[18257]: [10-2] 2020-12-02 18:44:53: pid 18257: DETAIL: set SO_REUSEPORT