after check pgpool log on 3 servers (10.102.46.175,10.102.46.176,10.102.46.177) we found the events as below:- 10.102.46.175 is leader 10.102.46.176,10.102.46.177 are member 1. <10.102.46.177> see 10.102.46.177_pgpool-2023-09-26_000000.log.zip 2023-09-26 10:26:18: [30739]: db=[No Connection],user=[No Connection],app=health_check0 ERROR: health check timed out while waiting for reading data 2023-09-26 10:26:18: [30739]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 (round:1) 2023-09-26 10:26:25: [30739]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 succeeded 2023-09-26 10:27:16: [30739]: db=[No Connection],user=[No Connection],app=health_check0 ERROR: health check timed out while waiting for reading data 2023-09-26 10:27:33: [29515]: db=[No Connection],user=[No Connection],app=main LOG: leader watchdog has performed failover 2023-09-26 10:29:39: [30739]: db=[No Connection],user=[No Connection],app=health_check0 ERROR: failed to make persistent db connection 2023-09-26 10:29:39: [30739]: db=[No Connection],user=[No Connection],app=health_check0 DETAIL: connection to host:"customerdb-a:5432" failed 2023-09-26 10:29:39: [30739]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 (round:1) 2023-09-26 10:29:43: [30739]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 succeeded 2023-09-26 10:31:13: [30739]: db=[No Connection],user=[No Connection],app=health_check0 ERROR: failed to make persistent db connection 2023-09-26 10:31:13: [30739]: db=[No Connection],user=[No Connection],app=health_check0 DETAIL: connection to host:"customerdb-a:5432" failed 2023-09-26 10:31:13: [30739]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 (round:1) 2023-09-26 10:31:26: [29518]: db=[No Connection],user=[No Connection],app=watchdog LOG: leader/coordinator node "pgpool-0:9999 Linux T1VMPDDB89" decided to resigning from leader, probably because of split-brain 2023-09-26 10:31:27: [29518]: db=[No Connection],user=[No Connection],app=watchdog LOG: setting the remote node "pgpool-1:9999 Linux T1VMPDDB90" as watchdog cluster leader 2023-09-26 10:31:30: [30739]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 succeeded 2023-09-26 10:33:26: [30739]: db=[No Connection],user=[No Connection],app=health_check0 LOG: failed to connect to PostgreSQL server on "customerdb-a:5432" using INET socket 2023-09-26 10:33:26: [30739]: db=[No Connection],user=[No Connection],app=health_check0 DETAIL: health check timer expired 2023-09-26 10:33:26: [30739]: db=[No Connection],user=[No Connection],app=health_check0 ERROR: failed to make persistent db connection 2023-09-26 10:33:26: [30739]: db=[No Connection],user=[No Connection],app=health_check0 DETAIL: connection to host:"customerdb-a:5432" failed 2023-09-26 10:33:26: [30739]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 (round:1) 2023-09-26 10:33:39: [30739]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 succeeded 2023-09-26 10:34:26: [29518]: db=[No Connection],user=[No Connection],app=watchdog LOG: new IPC connection received everything is fine after 2023-09-26 10:34:26 2. <10.102.46.176> see 10.102.46.176_pgpool-2023-09-26_000000.log.zip 2023-09-26 10:27:16: [32514]: db=[No Connection],user=[No Connection],app=health_check0 ERROR: health check timed out while waiting for reading data 2023-09-26 10:27:16: [32514]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 (round:1) 2023-09-26 10:27:32: [32514]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 succeeded 2023-09-26 10:27:33: [31286]: db=[No Connection],user=[No Connection],app=main LOG: leader watchdog has performed failover 2023-09-26 10:29:39: [32514]: db=[No Connection],user=[No Connection],app=health_check0 LOG: failed to connect to PostgreSQL server on "customerdb-a:5432" using INET socket 2023-09-26 10:29:39: [32514]: db=[No Connection],user=[No Connection],app=health_check0 DETAIL: health check timer expired 2023-09-26 10:29:39: [32514]: db=[No Connection],user=[No Connection],app=health_check0 ERROR: failed to make persistent db connection 2023-09-26 10:29:39: [32514]: db=[No Connection],user=[No Connection],app=health_check0 DETAIL: connection to host:"customerdb-a:5432" failed 2023-09-26 10:29:39: [32514]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 (round:1) 2023-09-26 10:29:43: [32514]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 succeeded 2023-09-26 10:31:13: [32514]: db=[No Connection],user=[No Connection],app=health_check0 LOG: failed to connect to PostgreSQL server on "customerdb-a:5432" using INET socket 2023-09-26 10:31:13: [32514]: db=[No Connection],user=[No Connection],app=health_check0 DETAIL: health check timer expired 2023-09-26 10:31:13: [32514]: db=[No Connection],user=[No Connection],app=health_check0 ERROR: failed to make persistent db connection 2023-09-26 10:31:13: [32514]: db=[No Connection],user=[No Connection],app=health_check0 DETAIL: connection to host:"customerdb-a:5432" failed 2023-09-26 10:31:13: [32514]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 (round:1) 2023-09-26 10:31:26: [31289]: db=[No Connection],user=[No Connection],app=watchdog LOG: leader/coordinator node "pgpool-0:9999 Linux T1VMPDDB89" decided to resigning from leader, probably because of split-brain 2023-09-26 10:31:27: [31289]: db=[No Connection],user=[No Connection],app=watchdog LOG: setting the local node "pgpool-1:9999 Linux T1VMPDDB90" as watchdog cluster leader 2023-09-26 10:31:29: [3906]: db=[No Connection],user=[No Connection],app=child LOG: new connection received 2023-09-26 10:31:29: [3906]: db=[No Connection],user=[No Connection],app=child DETAIL: connecting host=10.102.46.107 port=55046 2023-09-26 10:31:30: [3906]: db=corebiz,user=corebiz.app,app=postgres_fdw LOG: statement: SET search_path = pg_catalog 2023-09-26 10:31:30: [32514]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 succeeded 2023-09-26 10:33:43: [32514]: db=[No Connection],user=[No Connection],app=health_check0 ERROR: failed to make persistent db connection 2023-09-26 10:33:43: [32514]: db=[No Connection],user=[No Connection],app=health_check0 DETAIL: connection to host:"customerdb-a:5432" failed 2023-09-26 10:33:43: [32514]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 (round:1) 2023-09-26 10:33:43: [2965]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:33:44: [3022]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:33:44: [3976]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:33:44: [3945]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:33:44: [3688]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:33:44: [3679]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:33:44: [2999]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:33:44: [3496]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:33:44: [3328]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:33:44: [3952]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:33:44: [3219]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:33:44: [3233]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:33:56: [3217]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:33:56: [3044]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:33:57: [3650]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:33:59: [3757]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:34:03: [3090]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket everything is fine after 2023-09-26 10:34:22 3. <10.102.46.175> see 10.102.46.175_pgpool-2023-09-26_102619.log.zip 2023-09-26 10:26:24: [22402]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 succeeded 2023-09-26 10:27:16: [22402]: db=[No Connection],user=[No Connection],app=health_check0 ERROR: health check timed out while waiting for reading data 2023-09-26 10:27:16: [22402]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 (round:1) 2023-09-26 10:27:29: [22133]: db=[No Connection],user=[No Connection],app=[unknown] LOG: failed to connect to PostgreSQL server on "customerdb-a:5432", getsockopt() failed 2023-09-26 10:27:29: [659]: db=[No Connection],user=[No Connection],app=[unknown] LOG: failed to connect to PostgreSQL server on "customerdb-a:5432", getsockopt() failed 2023-09-26 10:27:29: [655]: db=[No Connection],user=[No Connection],app=[unknown] LOG: failed to connect to PostgreSQL server on "customerdb-a:5432", getsockopt() failed 2023-09-26 10:27:29: [1891]: db=[No Connection],user=[No Connection],app=[unknown] LOG: failed to connect to PostgreSQL server on "customerdb-a:5432", getsockopt() failed 2023-09-26 10:27:29: [3156]: db=[No Connection],user=[No Connection],app=[unknown] LOG: failed to connect to PostgreSQL server on "customerdb-a:5432", getsockopt() failed 2023-09-26 10:27:29: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface 2023-09-26 10:27:29: [1891]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: executing failover on backend 2023-09-26 10:27:29: [3156]: db=[No Connection],user=[No Connection],app=[unknown] FATAL: failed to create a backend connection 2023-09-26 10:27:29: [3156]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: executing failover on backend 2023-09-26 10:27:29: [21117]: db=[No Connection],user=[No Connection],app=main LOG: failover: no valid backend node found 2023-09-26 10:27:29: [21117]: db=[No Connection],user=[No Connection],app=main LOG: Restart all children 2023-09-26 10:27:30: [8610]: db=[No Connection],user=[No Connection],app=child FATAL: pgpool is not accepting any new connections 2023-09-26 10:27:30: [8610]: db=[No Connection],user=[No Connection],app=child DETAIL: all backend nodes are down, pgpool requires at least one valid node 2023-09-26 10:27:30: [8610]: db=[No Connection],user=[No Connection],app=child HINT: repair the backend nodes and restart pgpool 2023-09-26 10:27:30: [8607]: db=[No Connection],user=[No Connection],app=child FATAL: pgpool is not accepting any new connections 2023-09-26 10:27:30: [8607]: db=[No Connection],user=[No Connection],app=child DETAIL: all backend nodes are down, pgpool requires at least one valid node 2023-09-26 10:27:30: [21117]: db=[No Connection],user=[No Connection],app=main LOG: failover: no valid backend node found 2023-09-26 10:27:30: [21117]: db=[No Connection],user=[No Connection],app=main LOG: Restart all children 2023-09-26 10:27:31: [11995]: db=[No Connection],user=[No Connection],app=child FATAL: pgpool is not accepting any new connections 2023-09-26 10:27:31: [11995]: db=[No Connection],user=[No Connection],app=child DETAIL: all backend nodes are down, pgpool requires at least one valid node 2023-09-26 10:27:31: [11995]: db=[No Connection],user=[No Connection],app=child HINT: repair the backend nodes and restart pgpool 2023-09-26 10:27:32: [20076]: db=[No Connection],user=[No Connection],app=pcp_main LOG: restart request received in pcp child process 2023-09-26 10:27:32: [11905]: db=[No Connection],user=[No Connection],app=child FATAL: pgpool is not accepting any new connections 2023-09-26 10:27:32: [11905]: db=[No Connection],user=[No Connection],app=child DETAIL: all backend nodes are down, pgpool requires at least one valid node 2023-09-26 10:27:32: [11905]: db=[No Connection],user=[No Connection],app=child HINT: repair the backend nodes and restart pgpool 2023-09-26 10:27:34: [14406]: db=[No Connection],user=[No Connection],app=child LOG: new connection received 2023-09-26 10:27:34: [14406]: db=[No Connection],user=[No Connection],app=child DETAIL: connecting host=10.102.44.103 port=44584 2023-09-26 10:27:34: [14408]: db=[No Connection],user=[No Connection],app=child LOG: new connection received 2023-09-26 10:27:34: [14408]: db=[No Connection],user=[No Connection],app=child DETAIL: connecting host=10.102.44.103 port=44586 2023-09-26 10:27:34: [14411]: db=[No Connection],user=[No Connection],app=child LOG: new connection received 2023-09-26 10:27:34: [14411]: db=[No Connection],user=[No Connection],app=child DETAIL: connecting host=10.102.44.60 port=33742 2023-09-26 10:27:34: [14377]: db=[No Connection],user=[No Connection],app=child LOG: new connection received 2023-09-26 10:27:34: [14377]: db=[No Connection],user=[No Connection],app=child DETAIL: connecting host=10.102.49.38 port=55874 2023-09-26 10:27:34: [14418]: db=[No Connection],user=[No Connection],app=child LOG: new connection received 2023-09-26 10:27:34: [14418]: db=[No Connection],user=[No Connection],app=child DETAIL: connecting host=10.102.44.103 port=44590 2023-09-26 10:27:34: [14417]: db=[No Connection],user=[No Connection],app=child LOG: new connection received 2023-09-26 10:27:34: [14417]: db=[No Connection],user=[No Connection],app=child DETAIL: connecting host=10.102.44.55 port=59680 2023-09-26 10:27:34: [14398]: db=[No Connection],user=[No Connection],app=child LOG: new connection received 2023-09-26 10:27:34: [14398]: db=[No Connection],user=[No Connection],app=child DETAIL: connecting host=10.102.44.22 port=59096 2023-09-26 10:29:12: [14154]: db=[No Connection],user=[No Connection],app=[unknown] LOG: new connection received 2023-09-26 10:29:12: [14154]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: connecting host=10.102.44.75 port=47168 2023-09-26 10:29:12: [13611]: db=[No Connection],user=[No Connection],app=[unknown] LOG: new connection received 2023-09-26 10:29:12: [13611]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: connecting host=10.102.44.68 port=52216 2023-09-26 10:29:39: [22402]: db=[No Connection],user=[No Connection],app=health_check0 LOG: failed to connect to PostgreSQL server on "customerdb-a:5432" using INET socket 2023-09-26 10:29:39: [22402]: db=[No Connection],user=[No Connection],app=health_check0 DETAIL: health check timer expired 2023-09-26 10:29:39: [22402]: db=[No Connection],user=[No Connection],app=health_check0 ERROR: failed to make persistent db connection 2023-09-26 10:29:39: [22402]: db=[No Connection],user=[No Connection],app=health_check0 DETAIL: connection to host:"customerdb-a:5432" failed 2023-09-26 10:29:39: [22402]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 (round:1) 2023-09-26 10:29:39: [13700]: db=[No Connection],user=[No Connection],app=[unknown] LOG: new connection received 2023-09-26 10:29:39: [13700]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: connecting host=10.102.44.102 port=43056 2023-09-26 10:29:39: [13491]: db=[No Connection],user=[No Connection],app=[unknown] LOG: new connection received 2023-09-26 10:29:39: [13491]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: connecting host=10.102.44.51 port=43340 2023-09-26 10:29:39: [13947]: db=[No Connection],user=[No Connection],app=[unknown] LOG: new connection received 2023-09-26 10:29:39: [13947]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: connecting host=10.102.49.38 port=56660 2023-09-26 10:29:39: [13978]: db=[No Connection],user=[No Connection],app=[unknown] LOG: new connection received 2023-09-26 10:29:39: [13978]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: connecting host=10.102.49.39 port=50192 2023-09-26 10:29:39: [14108]: db=[No Connection],user=[No Connection],app=[unknown] LOG: new connection received 2023-09-26 10:29:43: [22402]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 succeeded 2023-09-26 10:29:50: [13460]: db=[No Connection],user=[No Connection],app=[unknown] LOG: new connection received 2023-09-26 10:29:50: [13460]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: connecting host=10.102.44.103 port=35790 2023-09-26 10:29:50: [14170]: db=[No Connection],user=[No Connection],app=[unknown] LOG: new connection received 2023-09-26 10:29:50: [14170]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: connecting host=10.102.44.159 port=43832 2023-09-26 10:29:05: [14146]: db=[No Connection],user=[No Connection],app=[unknown] LOG: new connection received 2023-09-26 10:29:05: [14146]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: connecting host=10.102.44.22 port=37702 2023-09-26 10:30:50: [14146]: db=[No Connection],user=[No Connection],app=[unknown] LOG: received degenerate backend request for node_id: 0 from pid [14146] 2023-09-26 10:30:50: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: new IPC connection received 2023-09-26 10:30:50: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: watchdog received the failover command from local pgpool-II on IPC interface 2023-09-26 10:30:50: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface 2023-09-26 10:30:50: [14146]: db=[No Connection],user=[No Connection],app=[unknown] WARNING: write on backend 0 failed with error :"Broken pipe" 2023-09-26 10:30:50: [14146]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: while trying to write data from offset: 0 wlen: 5 2023-09-26 10:30:50: [14334]: db=[No Connection],user=[No Connection],app=[unknown] LOG: new connection received 2023-09-26 10:30:50: [14334]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: connecting host=10.102.44.59 port=55538 2023-09-26 10:30:50: [14334]: db=[No Connection],user=[No Connection],app=[unknown] LOG: frontend disconnection: session time: 0:00:00.003 user=corebiz_bcp.app database=corebiz host=10.102.44.59 port=55538 2023-09-26 10:30:56: [13751]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:30:56: [13751]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:30:56: [13246]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:30:56: [13246]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:30:56: [14152]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:30:56: [14152]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:30:56: [13413]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:30:56: [13413]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:30:56: [14071]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:30:56: [14071]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:30:56: [13673]: db=[No Connection],user=[No Connection],app=[unknown] LOG: new connection received 2023-09-26 10:30:56: [13673]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: connecting host=10.102.44.62 port=35218 2023-09-26 10:30:56: [13588]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:30:56: [13588]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:30:56: [14010]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:30:56: [14010]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:30:56: [14257]: db=[No Connection],user=[No Connection],app=[unknown] LOG: new connection received 2023-09-26 10:30:56: [14257]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: connecting host=10.102.49.39 port=50604 2023-09-26 10:30:56: [13462]: db=[No Connection],user=[No Connection],app=[unknown] LOG: new connection received 2023-09-26 10:30:56: [13462]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: connecting host=10.102.44.102 port=43350 2023-09-26 10:30:56: [14171]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:30:56: [14171]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:30:56: [13224]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:30:56: [13224]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:30:56: [14294]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:30:56: [14294]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:30:56: [13946]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:30:56: [13946]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:30:56: [14393]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:30:56: [14393]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:30:56: [13370]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:30:56: [13370]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:30:56: [13723]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:03: [22402]: db=[No Connection],user=[No Connection],app=health_check0 ERROR: health check timed out while waiting for reading data 2023-09-26 10:31:03: [22402]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 (round:1) 2023-09-26 10:31:03: [14155]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:09: [13513]: db=[No Connection],user=[No Connection],app=[unknown] LOG: new connection received 2023-09-26 10:31:09: [13513]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: connecting host=10.102.49.38 port=57104 2023-09-26 10:31:09: [13770]: db=[No Connection],user=[No Connection],app=[unknown] LOG: new connection received 2023-09-26 10:31:09: [13770]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: connecting host=10.102.49.62 port=42884 2023-09-26 10:31:09: [13914]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:09: [13914]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:09: [14024]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:10: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: new IPC connection received 2023-09-26 10:31:10: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: watchdog received the failover command from local pgpool-II on IPC interface 2023-09-26 10:31:10: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface 2023-09-26 10:31:10: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: failover requires the majority vote, waiting for consensus 2023-09-26 10:31:10: [21120]: db=[No Connection],user=[No Connection],app=watchdog DETAIL: failover request noted 2023-09-26 10:31:10: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: failover command [DEGENERATE_BACKEND_REQUEST] request from pgpool-II node "pgpool-0:9999 Linux T1VMPDDB89" is queued, waiting for the confirmation from other nodes 2023-09-26 10:31:10: [14121]: db=[No Connection],user=[No Connection],app=[unknown] LOG: degenerate backend request for node_id: 0 from pid [14121], will be handled by watchdog, which is building consensus for request 2023-09-26 10:31:12: [13871]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:12: [13871]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:12: [13343]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:12: [13343]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:12: [13797]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:12: [13797]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:12: [13713]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:12: [13713]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:12: [13666]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:12: [13666]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:12: [13430]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:12: [13430]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:12: [13846]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:12: [13846]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:12: [13839]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:12: [13839]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:12: [13572]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:12: [13572]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:12: [13610]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:12: [13610]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:13: [13560]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:13: [13560]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:13: [13290]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:13: [13290]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:13: [14079]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:13: [14079]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:13: [13557]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:13: [13557]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:13: [14302]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:13: [14302]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:13: [13852]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:13: [13852]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:13: [13641]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:13: [13641]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:13: [14411]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:13: [14411]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:13: [13440]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:13: [13440]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:13: [13863]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:13: [13863]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:13: [14328]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:13: [14328]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:13: [13781]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:13: [13781]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:13: [14374]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:13: [14374]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:13: [13178]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:13: [13178]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:13: [14248]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:13: [14248]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:13: [13190]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:13: [13190]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:13: [13759]: db=[No Connection],user=[No Connection],app=[unknown] LOG: trying connecting to PostgreSQL server on "customerdb-a:5432" by INET socket 2023-09-26 10:31:13: [13759]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: timed out. retrying... 2023-09-26 10:31:14: [14075]: db=[No Connection],user=[No Connection],app=[unknown] LOG: failed to connect to PostgreSQL server on "customerdb-a:5432", getsockopt() failed 2023-09-26 10:31:14: [14075]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: Operation already in progress 2023-09-26 10:31:14: [13516]: db=[No Connection],user=[No Connection],app=[unknown] LOG: failed to connect to PostgreSQL server on "customerdb-a:5432", getsockopt() failed 2023-09-26 10:31:14: [13516]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: Operation already in progress 2023-09-26 10:31:14: [13707]: db=[No Connection],user=[No Connection],app=[unknown] LOG: failed to connect to PostgreSQL server on "customerdb-a:5432", getsockopt() failed 2023-09-26 10:31:14: [13707]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: Operation already in progress 2023-09-26 10:31:14: [13306]: db=[No Connection],user=[No Connection],app=[unknown] LOG: failed to connect to PostgreSQL server on "customerdb-a:5432", getsockopt() failed 2023-09-26 10:31:14: [13306]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: Operation already in progress 2023-09-26 10:31:14: [14075]: db=[No Connection],user=[No Connection],app=[unknown] LOG: received degenerate backend request for node_id: 0 from pid [14075] 2023-09-26 10:31:14: [13516]: db=[No Connection],user=[No Connection],app=[unknown] LOG: received degenerate backend request for node_id: 0 from pid [13516] 2023-09-26 10:31:14: [13306]: db=[No Connection],user=[No Connection],app=[unknown] LOG: received degenerate backend request for node_id: 0 from pid [13306] 2023-09-26 10:31:14: [13707]: db=[No Connection],user=[No Connection],app=[unknown] LOG: received degenerate backend request for node_id: 0 from pid [13707] 2023-09-26 10:31:14: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface 2023-09-26 10:31:14: [13306]: db=[No Connection],user=[No Connection],app=[unknown] FATAL: failed to create a backend connection 2023-09-26 10:31:14: [13306]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: executing failover on backend 2023-09-26 10:31:14: [13707]: db=[No Connection],user=[No Connection],app=[unknown] LOG: degenerate backend request for 1 node(s) from pid [13707], is changed to quarantine node request by watchdog 2023-09-26 10:31:14: [13707]: db=[No Connection],user=[No Connection],app=[unknown] DETAIL: watchdog is taking time to build consensus 2023-09-26 10:31:14: [13516]: db=[No Connection],user=[No Connection],app=[unknown] LOG: degenerate backend request for 1 node(s) from pid [13516], is changed to quarantine node request by watchdog 2023-09-26 10:31:14: [21117]: db=[No Connection],user=[No Connection],app=main LOG: failover: no valid backend node found 2023-09-26 10:31:14: [21117]: db=[No Connection],user=[No Connection],app=main LOG: Restart all children 2023-09-26 10:31:14: [15466]: db=[No Connection],user=[No Connection],app=child FATAL: pgpool is not accepting any new connections 2023-09-26 10:31:14: [15466]: db=[No Connection],user=[No Connection],app=child DETAIL: all backend nodes are down, pgpool requires at least one valid node 2023-09-26 10:31:14: [15466]: db=[No Connection],user=[No Connection],app=child HINT: repair the backend nodes and restart pgpool 2023-09-26 10:31:14: [15556]: db=[No Connection],user=[No Connection],app=child HINT: repair the backend nodes and restart pgpool 2023-09-26 10:31:14: [15558]: db=[No Connection],user=[No Connection],app=child FATAL: pgpool is not accepting any new connections 2023-09-26 10:31:14: [15558]: db=[No Connection],user=[No Connection],app=child DETAIL: all backend nodes are down, pgpool requires at least one valid node 2023-09-26 10:31:16: [20274]: db=[No Connection],user=[No Connection],app=pcp_main LOG: PCP process: 20274 started 2023-09-26 10:31:16: [19551]: db=[No Connection],user=[No Connection],app=child FATAL: pgpool is not accepting any new connections 2023-09-26 10:31:16: [19551]: db=[No Connection],user=[No Connection],app=child DETAIL: all backend nodes are down, pgpool requires at least one valid node 2023-09-26 10:31:16: [19551]: db=[No Connection],user=[No Connection],app=child HINT: repair the backend nodes and restart pgpool 2023-09-26 10:31:26: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: We are not able to build consensus for our primary node failover request, got 1 votes only for failover request ID:46642 2023-09-26 10:31:26: [21120]: db=[No Connection],user=[No Connection],app=watchdog DETAIL: resigning from the coordinator 2023-09-26 10:31:26: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: watchdog node state changed from [LEADER] to [JOINING] 2023-09-26 10:31:26: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: removing the local node "pgpool-0:9999 Linux T1VMPDDB89" from watchdog cluster leader 2023-09-26 10:31:26: [21729]: db=[No Connection],user=[No Connection],app=watchdog_utility LOG: watchdog: de-escalation started 2023-09-26 10:31:26: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: watchdog node state changed from [JOINING] to [INITIALIZING] 2023-09-26 10:31:26: [21729]: db=[No Connection],user=[No Connection],app=watchdog_utility LOG: successfully released the delegate IP:"10.102.46.100" 2023-09-26 10:31:26: [21729]: db=[No Connection],user=[No Connection],app=watchdog_utility DETAIL: 'if_down_cmd' returned with success 2023-09-26 10:31:26: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: watchdog de-escalation process with pid: 21729 exit with SUCCESS. 2023-09-26 10:31:27: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR LEADER] 2023-09-26 10:31:27: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: our stand for coordinator request is rejected by node "pgpool-2:9999 Linux T1VMPDDB91" 2023-09-26 10:31:27: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: watchdog node state changed from [STANDING FOR LEADER] to [PARTICIPATING IN ELECTION] 2023-09-26 10:31:27: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [INITIALIZING] 2023-09-26 10:31:27: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: setting the remote node "pgpool-1:9999 Linux T1VMPDDB90" as watchdog cluster leader 2023-09-26 10:31:28: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2023-09-26 10:31:28: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: signal_user1_to_parent_with_reason(1) 2023-09-26 10:31:28: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: successfully joined the watchdog cluster as standby node 2023-09-26 10:31:28: [21120]: db=[No Connection],user=[No Connection],app=watchdog DETAIL: our join coordinator request is accepted by cluster leader node "pgpool-1:9999 Linux T1VMPDDB90" 2023-09-26 10:31:28: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: new IPC connection received 2023-09-26 10:31:28: [21117]: db=[No Connection],user=[No Connection],app=main LOG: we have joined the watchdog cluster as STANDBY node 2023-09-26 10:31:28: [21117]: db=[No Connection],user=[No Connection],app=main DETAIL: syncing the backend states from the LEADER watchdog node 2023-09-26 10:31:28: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: new IPC connection received 2023-09-26 10:31:28: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: received the get data request from local pgpool-II on IPC interface 2023-09-26 10:31:28: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: get data request from local pgpool-II node received on IPC interface is forwarded to leader watchdog node "pgpool-1:9999 Linux T1VMPDDB90" 2023-09-26 10:31:30: [22402]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 succeeded 2023-09-26 10:31:40: [14339]: db=[No Connection],user=[No Connection],app=sr_check_worker LOG: worker process received restart request 2023-09-26 10:31:40: [21117]: db=[No Connection],user=[No Connection],app=main LOG: worker child process with pid: 14339 exits with status 256 2023-09-26 10:31:40: [21117]: db=[No Connection],user=[No Connection],app=main LOG: fork a new worker child process with pid: 23060 2023-09-26 10:31:40: [23060]: db=[No Connection],user=[No Connection],app=sr_check_worker LOG: process started 2023-09-26 10:31:40: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: new IPC connection received 2023-09-26 10:31:50: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: new IPC connection received 2023-09-26 10:32:00: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: new IPC connection received 2023-09-26 10:32:14: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: new IPC connection received 2023-09-26 10:32:26: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: new IPC connection received 2023-09-26 10:32:40: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: new IPC connection received 2023-09-26 10:33:00: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: new IPC connection received 2023-09-26 10:33:20: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: new IPC connection received 2023-09-26 10:33:32: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: new IPC connection received 2023-09-26 10:33:43: [22402]: db=[No Connection],user=[No Connection],app=health_check0 LOG: failed to connect to PostgreSQL server on "customerdb-a:5432" using INET socket 2023-09-26 10:33:43: [22402]: db=[No Connection],user=[No Connection],app=health_check0 DETAIL: health check timer expired 2023-09-26 10:33:43: [22402]: db=[No Connection],user=[No Connection],app=health_check0 ERROR: failed to make persistent db connection 2023-09-26 10:33:43: [22402]: db=[No Connection],user=[No Connection],app=health_check0 DETAIL: connection to host:"customerdb-a:5432" failed 2023-09-26 10:33:43: [22402]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 (round:1) 2023-09-26 10:33:52: [22402]: db=[No Connection],user=[No Connection],app=health_check0 LOG: health check retrying on DB node: 0 succeeded 2023-09-26 10:34:18: [21120]: db=[No Connection],user=[No Connection],app=watchdog LOG: remote node "pgpool-1:9999 Linux T1VMPDDB90" is asking to inform about quarantined backend nodes everything is fine after 2023-09-26 10:34:26