Mar 12 13:49:13 AISGPGP01 pgpool[29590]: [11-1] 2021-03-12 13:49:13 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  informing the node status change to watchdog
Mar 12 13:49:13 AISGPGP01 pgpool[29590]: [11-2] 2021-03-12 13:49:13 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  node id :1 status = "NODE DEAD" message:"No heartbeat signal from node"
Mar 12 13:49:13 AISGPGP01 pgpool[29589]: [106-1] 2021-03-12 13:49:13 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  new IPC connection received
Mar 12 13:49:13 AISGPGP01 pgpool[29589]: [107-1] 2021-03-12 13:49:13 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  received node status change ipc message
Mar 12 13:49:13 AISGPGP01 pgpool[29589]: [107-2] 2021-03-12 13:49:13 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  No heartbeat signal from node
Mar 12 13:49:13 AISGPGP01 pgpool[29589]: [108-1] 2021-03-12 13:49:13 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  remote node "aisgpgp02.cca.cz:9999 Linux AISGPGP02.cca.cz" is lost
Mar 12 13:49:13 AISGPGP01 pgpool[29589]: [109-1] 2021-03-12 13:49:13 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  removing watchdog node "aisgpgp02.cca.cz:9999 Linux AISGPGP02.cca.cz" from the standby list
Mar 12 13:49:40 AISGPGP01 pgpool[29594]: [10-1] 2021-03-12 13:49:40 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to send watchdog heartbeat, gethostbyname() failed
Mar 12 13:49:40 AISGPGP01 pgpool[29594]: [10-2] 2021-03-12 13:49:40 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  gethostbyname on "aisgpgp02.cca.cz" failed with reason: "Unknown host"
Mar 12 13:49:41 AISGPGP01 pgpool[29918]: [32-1] 2021-03-12 13:49:41 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:49:41 AISGPGP01 pgpool[29918]: [33-1] 2021-03-12 13:49:41 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:49:41 AISGPGP01 pgpool[29918]: [33-2] 2021-03-12 13:49:41 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd01.cca.cz:5432" failed
Mar 12 13:49:41 AISGPGP01 pgpool[29918]: [34-1] 2021-03-12 13:49:41 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  health check retrying on DB node: 0 (round:1)
Mar 12 13:49:42 AISGPGP01 pgpool[30218]: [594-1] 2021-03-12 13:49:42 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:49:42 AISGPGP01 pgpool[30218]: [595-1] 2021-03-12 13:49:42 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:49:42 AISGPGP01 pgpool[30218]: [595-2] 2021-03-12 13:49:42 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd01.cca.cz:5432" failed
Mar 12 13:49:42 AISGPGP01 pgpool[29845]: [13-1] 2021-03-12 13:49:42 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:49:42 AISGPGP01 pgpool[29845]: [14-1] 2021-03-12 13:49:42 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:49:42 AISGPGP01 pgpool[29845]: [14-2] 2021-03-12 13:49:42 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd02.cca.cz:5432" failed
Mar 12 13:49:42 AISGPGP01 pgpool[29845]: [15-1] 2021-03-12 13:49:42 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  health check retrying on DB node: 1 (round:1)
Mar 12 13:50:13 AISGPGP01 pgpool[29589]: [110-1] 2021-03-12 13:50:13 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  failed to get host address for "aisgpgp02.cca.cz"
Mar 12 13:50:13 AISGPGP01 pgpool[29589]: [110-2] 2021-03-12 13:50:13 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  gethostbyaddr failed with error: "Unknown server error"
Mar 12 13:50:13 AISGPGP01 pgpool[29587]: [1129-1] 2021-03-12 13:50:13 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  Pgpool-II parent process received watchdog quorum change signal from watchdog
Mar 12 13:50:13 AISGPGP01 pgpool[29589]: [111-1] 2021-03-12 13:50:13 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  new IPC connection received
Mar 12 13:50:13 AISGPGP01 pgpool[29587]: [1130-1] 2021-03-12 13:50:13 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  watchdog cluster now holds the quorum
Mar 12 13:50:13 AISGPGP01 pgpool[29587]: [1130-2] 2021-03-12 13:50:13 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  updating the state of quarantine backend nodes
Mar 12 13:50:13 AISGPGP01 pgpool[29589]: [112-1] 2021-03-12 13:50:13 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  new IPC connection received
Mar 12 13:50:22 AISGPGP01 pgpool[30218]: [596-1] 2021-03-12 13:50:22 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:50:22 AISGPGP01 pgpool[30218]: [597-1] 2021-03-12 13:50:22 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:50:22 AISGPGP01 pgpool[30218]: [597-2] 2021-03-12 13:50:22 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd02.cca.cz:5432" failed
Mar 12 13:50:22 AISGPGP01 pgpool[29594]: [11-1] 2021-03-12 13:50:22 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to send watchdog heartbeat, gethostbyname() failed
Mar 12 13:50:22 AISGPGP01 pgpool[29594]: [11-2] 2021-03-12 13:50:22 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  gethostbyname on "aisgpgp02.cca.cz" failed with reason: "Unknown host"
Mar 12 13:50:22 AISGPGP01 pgpool[29918]: [35-1] 2021-03-12 13:50:22 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:50:22 AISGPGP01 pgpool[29918]: [36-1] 2021-03-12 13:50:22 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:50:22 AISGPGP01 pgpool[29918]: [36-2] 2021-03-12 13:50:22 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd01.cca.cz:5432" failed
Mar 12 13:50:22 AISGPGP01 pgpool[29918]: [37-1] 2021-03-12 13:50:22 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  health check retrying on DB node: 0 (round:2)
Mar 12 13:50:23 AISGPGP01 pgpool[29845]: [16-1] 2021-03-12 13:50:23 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:50:23 AISGPGP01 pgpool[29845]: [17-1] 2021-03-12 13:50:23 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:50:23 AISGPGP01 pgpool[29845]: [17-2] 2021-03-12 13:50:23 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd02.cca.cz:5432" failed
Mar 12 13:50:23 AISGPGP01 pgpool[29845]: [18-1] 2021-03-12 13:50:23 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  health check retrying on DB node: 1 (round:2)
Mar 12 13:50:27 AISGPGP01 pgpool[30218]: [598-1] 2021-03-12 13:50:27 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  Failed to check replication time lag
Mar 12 13:50:27 AISGPGP01 pgpool[30218]: [598-2] 2021-03-12 13:50:27 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  No persistent db connection for the node 0
Mar 12 13:50:27 AISGPGP01 pgpool[30218]: [598-3] 2021-03-12 13:50:27 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |HINT:  check sr_check_user and sr_check_password
Mar 12 13:50:27 AISGPGP01 pgpool[30218]: [598-4] 2021-03-12 13:50:27 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |CONTEXT:  while checking replication time lag
Mar 12 13:50:50 AISGPGP01 pgpool[29587]: [1131-1] 2021-03-12 13:50:50 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  child process with pid: 30913 exits with status 256
Mar 12 13:50:50 AISGPGP01 pgpool[29587]: [1132-1] 2021-03-12 13:50:50 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  fork a new child process with pid: 30958
Mar 12 13:50:56 AISGPGP01 pgpool[30217]: [599-1] 2021-03-12 13:50:56 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  forked new pcp worker, pid=30960 socket=8
Mar 12 13:50:56 AISGPGP01 pgpool[30217]: [600-1] 2021-03-12 13:50:56 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  PCP process with pid: 30960 exit with SUCCESS.
Mar 12 13:50:56 AISGPGP01 pgpool[30217]: [601-1] 2021-03-12 13:50:56 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  PCP process with pid: 30960 exits with status 0
Mar 12 13:51:03 AISGPGP01 pgpool[29918]: [38-1] 2021-03-12 13:51:03 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:51:03 AISGPGP01 pgpool[29918]: [39-1] 2021-03-12 13:51:03 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:51:03 AISGPGP01 pgpool[29918]: [39-2] 2021-03-12 13:51:03 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd01.cca.cz:5432" failed
Mar 12 13:51:03 AISGPGP01 pgpool[29918]: [40-1] 2021-03-12 13:51:03 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  health check retrying on DB node: 0 (round:3)
Mar 12 13:51:04 AISGPGP01 pgpool[29594]: [12-1] 2021-03-12 13:51:04 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to send watchdog heartbeat, gethostbyname() failed
Mar 12 13:51:04 AISGPGP01 pgpool[29594]: [12-2] 2021-03-12 13:51:04 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  gethostbyname on "aisgpgp02.cca.cz" failed with reason: "Unknown host"
Mar 12 13:51:04 AISGPGP01 pgpool[29845]: [19-1] 2021-03-12 13:51:04 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:51:04 AISGPGP01 pgpool[29845]: [20-1] 2021-03-12 13:51:04 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:51:04 AISGPGP01 pgpool[29845]: [20-2] 2021-03-12 13:51:04 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd02.cca.cz:5432" failed
Mar 12 13:51:04 AISGPGP01 pgpool[29845]: [21-1] 2021-03-12 13:51:04 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  health check retrying on DB node: 1 (round:3)
Mar 12 13:51:07 AISGPGP01 pgpool[30218]: [599-1] 2021-03-12 13:51:07 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:51:07 AISGPGP01 pgpool[30218]: [600-1] 2021-03-12 13:51:07 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:51:07 AISGPGP01 pgpool[30218]: [600-2] 2021-03-12 13:51:07 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd01.cca.cz:5432" failed
Mar 12 13:51:07 AISGPGP01 pgpool[30217]: [602-1] 2021-03-12 13:51:07 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  forked new pcp worker, pid=30962 socket=8
Mar 12 13:51:07 AISGPGP01 pgpool[30217]: [603-1] 2021-03-12 13:51:07 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  PCP process with pid: 30962 exit with SUCCESS.
Mar 12 13:51:07 AISGPGP01 pgpool[30217]: [604-1] 2021-03-12 13:51:07 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  PCP process with pid: 30962 exits with status 0
Mar 12 13:51:24 AISGPGP01 pgpool[29589]: [113-1] 2021-03-12 13:51:24 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  failed to get host address for "aisgpgp02.cca.cz"
Mar 12 13:51:24 AISGPGP01 pgpool[29589]: [113-2] 2021-03-12 13:51:24 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  gethostbyaddr failed with error: "Unknown server error"
Mar 12 13:51:24 AISGPGP01 pgpool[30217]: [605-1] 2021-03-12 13:51:24 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  forked new pcp worker, pid=30964 socket=8
Mar 12 13:51:24 AISGPGP01 pgpool[29589]: [114-1] 2021-03-12 13:51:24 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  new IPC connection received
Mar 12 13:51:24 AISGPGP01 pgpool[30217]: [606-1] 2021-03-12 13:51:24 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  PCP process with pid: 30964 exit with SUCCESS.
Mar 12 13:51:24 AISGPGP01 pgpool[30217]: [607-1] 2021-03-12 13:51:24 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  PCP process with pid: 30964 exits with status 0
Mar 12 13:51:44 AISGPGP01 pgpool[29918]: [41-1] 2021-03-12 13:51:44 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:51:44 AISGPGP01 pgpool[29918]: [42-1] 2021-03-12 13:51:44 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:51:44 AISGPGP01 pgpool[29918]: [42-2] 2021-03-12 13:51:44 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd01.cca.cz:5432" failed
Mar 12 13:51:44 AISGPGP01 pgpool[29918]: [43-1] 2021-03-12 13:51:44 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  health check failed on node 0 (timeout:1)
Mar 12 13:51:44 AISGPGP01 pgpool[29918]: [44-1] 2021-03-12 13:51:44 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  received degenerate backend request for node_id: 0 from pid [29918]
Mar 12 13:51:45 AISGPGP01 pgpool[29845]: [22-1] 2021-03-12 13:51:45 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:51:45 AISGPGP01 pgpool[29845]: [23-1] 2021-03-12 13:51:45 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:51:45 AISGPGP01 pgpool[29845]: [23-2] 2021-03-12 13:51:45 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd02.cca.cz:5432" failed
Mar 12 13:51:45 AISGPGP01 pgpool[29845]: [24-1] 2021-03-12 13:51:45 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  health check failed on node 1 (timeout:1)
Mar 12 13:51:45 AISGPGP01 pgpool[29845]: [25-1] 2021-03-12 13:51:45 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  received degenerate backend request for node_id: 1 from pid [29845]
Mar 12 13:51:46 AISGPGP01 pgpool[29594]: [13-1] 2021-03-12 13:51:46 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to send watchdog heartbeat, gethostbyname() failed
Mar 12 13:51:46 AISGPGP01 pgpool[29594]: [13-2] 2021-03-12 13:51:46 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  gethostbyname on "aisgpgp02.cca.cz" failed with reason: "Unknown host"
Mar 12 13:51:47 AISGPGP01 pgpool[30218]: [601-1] 2021-03-12 13:51:47 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:51:47 AISGPGP01 pgpool[30218]: [602-1] 2021-03-12 13:51:47 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:51:47 AISGPGP01 pgpool[30218]: [602-2] 2021-03-12 13:51:47 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd02.cca.cz:5432" failed
Mar 12 13:51:52 AISGPGP01 pgpool[30218]: [603-1] 2021-03-12 13:51:52 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  Failed to check replication time lag
Mar 12 13:51:52 AISGPGP01 pgpool[30218]: [603-2] 2021-03-12 13:51:52 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  No persistent db connection for the node 0
Mar 12 13:51:52 AISGPGP01 pgpool[30218]: [603-3] 2021-03-12 13:51:52 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |HINT:  check sr_check_user and sr_check_password
Mar 12 13:51:52 AISGPGP01 pgpool[30218]: [603-4] 2021-03-12 13:51:52 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |CONTEXT:  while checking replication time lag
Mar 12 13:51:52 AISGPGP01 pgpool[29918]: [45-1] 2021-03-12 13:51:52 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failover command on watchdog failed
Mar 12 13:51:52 AISGPGP01 pgpool[29918]: [45-2] 2021-03-12 13:51:52 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  ipc command timeout
Mar 12 13:51:52 AISGPGP01 pgpool[29918]: [46-1] 2021-03-12 13:51:52 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  degenerate backend request for 1 node(s) from pid [29918] is canceled by other pgpool
Mar 12 13:51:53 AISGPGP01 pgpool[29845]: [26-1] 2021-03-12 13:51:53 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failover command on watchdog failed
Mar 12 13:51:53 AISGPGP01 pgpool[29845]: [26-2] 2021-03-12 13:51:53 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  ipc command timeout
Mar 12 13:51:53 AISGPGP01 pgpool[29845]: [27-1] 2021-03-12 13:51:53 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  degenerate backend request for 1 node(s) from pid [29845] is canceled by other pgpool
Mar 12 13:52:28 AISGPGP01 pgpool[29594]: [14-1] 2021-03-12 13:52:28 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to send watchdog heartbeat, gethostbyname() failed
Mar 12 13:52:28 AISGPGP01 pgpool[29594]: [14-2] 2021-03-12 13:52:28 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  gethostbyname on "aisgpgp02.cca.cz" failed with reason: "Unknown host"
Mar 12 13:52:32 AISGPGP01 pgpool[30218]: [604-1] 2021-03-12 13:52:32 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:52:32 AISGPGP01 pgpool[30218]: [605-1] 2021-03-12 13:52:32 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:52:32 AISGPGP01 pgpool[30218]: [605-2] 2021-03-12 13:52:32 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd01.cca.cz:5432" failed
Mar 12 13:52:35 AISGPGP01 pgpool[29918]: [47-1] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:52:35 AISGPGP01 pgpool[29918]: [48-1] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:52:35 AISGPGP01 pgpool[29918]: [48-2] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd01.cca.cz:5432" failed
Mar 12 13:52:35 AISGPGP01 pgpool[29918]: [49-1] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  health check retrying on DB node: 0 (round:1)
Mar 12 13:52:35 AISGPGP01 pgpool[29589]: [115-1] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  failed to get host address for "aisgpgp02.cca.cz"
Mar 12 13:52:35 AISGPGP01 pgpool[29589]: [115-2] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  gethostbyaddr failed with error: "Unknown server error"
Mar 12 13:52:35 AISGPGP01 pgpool[29589]: [116-1] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  new IPC connection received
Mar 12 13:52:35 AISGPGP01 pgpool[29589]: [117-1] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  new IPC connection received
Mar 12 13:52:35 AISGPGP01 pgpool[29589]: [118-1] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  watchdog received the failover command from local pgpool-II on IPC interface
Mar 12 13:52:35 AISGPGP01 pgpool[29589]: [119-1] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface
Mar 12 13:52:35 AISGPGP01 pgpool[29589]: [120-1] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  we have got the consensus to perform the failover
Mar 12 13:52:35 AISGPGP01 pgpool[29589]: [120-2] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  1 node(s) voted in the favor
Mar 12 13:52:35 AISGPGP01 pgpool[29589]: [121-1] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  failed to write watchdog packet to socket
Mar 12 13:52:35 AISGPGP01 pgpool[29589]: [121-2] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  Broken pipe
Mar 12 13:52:35 AISGPGP01 pgpool[29589]: [122-1] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  watchdog received the failover command from local pgpool-II on IPC interface
Mar 12 13:52:35 AISGPGP01 pgpool[29589]: [123-1] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface
Mar 12 13:52:35 AISGPGP01 pgpool[29589]: [124-1] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  we have got the consensus to perform the failover
Mar 12 13:52:35 AISGPGP01 pgpool[29589]: [124-2] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  1 node(s) voted in the favor
Mar 12 13:52:35 AISGPGP01 pgpool[29589]: [125-1] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  failed to write watchdog packet to socket
Mar 12 13:52:35 AISGPGP01 pgpool[29589]: [125-2] 2021-03-12 13:52:35 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  Broken pipe
Mar 12 13:52:36 AISGPGP01 pgpool[29845]: [28-1] 2021-03-12 13:52:36 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:52:36 AISGPGP01 pgpool[29845]: [29-1] 2021-03-12 13:52:36 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:52:36 AISGPGP01 pgpool[29845]: [29-2] 2021-03-12 13:52:36 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd02.cca.cz:5432" failed
Mar 12 13:52:36 AISGPGP01 pgpool[29845]: [30-1] 2021-03-12 13:52:36 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  health check retrying on DB node: 1 (round:1)
Mar 12 13:53:10 AISGPGP01 pgpool[29594]: [15-1] 2021-03-12 13:53:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to send watchdog heartbeat, gethostbyname() failed
Mar 12 13:53:10 AISGPGP01 pgpool[29594]: [15-2] 2021-03-12 13:53:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  gethostbyname on "aisgpgp02.cca.cz" failed with reason: "Unknown host"
Mar 12 13:53:12 AISGPGP01 pgpool[30218]: [606-1] 2021-03-12 13:53:12 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:53:12 AISGPGP01 pgpool[30218]: [607-1] 2021-03-12 13:53:12 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:53:12 AISGPGP01 pgpool[30218]: [607-2] 2021-03-12 13:53:12 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd02.cca.cz:5432" failed
Mar 12 13:53:16 AISGPGP01 pgpool[29918]: [50-1] 2021-03-12 13:53:16 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:53:16 AISGPGP01 pgpool[29918]: [51-1] 2021-03-12 13:53:16 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:53:16 AISGPGP01 pgpool[29918]: [51-2] 2021-03-12 13:53:16 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd01.cca.cz:5432" failed
Mar 12 13:53:16 AISGPGP01 pgpool[29918]: [52-1] 2021-03-12 13:53:16 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  health check retrying on DB node: 0 (round:2)
Mar 12 13:53:17 AISGPGP01 pgpool[30218]: [608-1] 2021-03-12 13:53:17 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  Failed to check replication time lag
Mar 12 13:53:17 AISGPGP01 pgpool[30218]: [608-2] 2021-03-12 13:53:17 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  No persistent db connection for the node 0
Mar 12 13:53:17 AISGPGP01 pgpool[30218]: [608-3] 2021-03-12 13:53:17 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |HINT:  check sr_check_user and sr_check_password
Mar 12 13:53:17 AISGPGP01 pgpool[30218]: [608-4] 2021-03-12 13:53:17 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |CONTEXT:  while checking replication time lag
Mar 12 13:53:18 AISGPGP01 pgpool[29845]: [31-1] 2021-03-12 13:53:18 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:53:18 AISGPGP01 pgpool[29845]: [32-1] 2021-03-12 13:53:18 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:53:18 AISGPGP01 pgpool[29845]: [32-2] 2021-03-12 13:53:18 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd02.cca.cz:5432" failed
Mar 12 13:53:18 AISGPGP01 pgpool[29845]: [33-1] 2021-03-12 13:53:18 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  health check retrying on DB node: 1 (round:2)
Mar 12 13:53:47 AISGPGP01 pgpool[29589]: [126-1] 2021-03-12 13:53:47 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  failed to get host address for "aisgpgp02.cca.cz"
Mar 12 13:53:47 AISGPGP01 pgpool[29589]: [126-2] 2021-03-12 13:53:47 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  gethostbyaddr failed with error: "Unknown server error"
Mar 12 13:53:52 AISGPGP01 pgpool[29594]: [16-1] 2021-03-12 13:53:52 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to send watchdog heartbeat, gethostbyname() failed
Mar 12 13:53:52 AISGPGP01 pgpool[29594]: [16-2] 2021-03-12 13:53:52 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  gethostbyname on "aisgpgp02.cca.cz" failed with reason: "Unknown host"
Mar 12 13:53:57 AISGPGP01 pgpool[30218]: [609-1] 2021-03-12 13:53:57 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:53:57 AISGPGP01 pgpool[30218]: [610-1] 2021-03-12 13:53:57 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:53:57 AISGPGP01 pgpool[30218]: [610-2] 2021-03-12 13:53:57 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd01.cca.cz:5432" failed
Mar 12 13:53:58 AISGPGP01 pgpool[29918]: [53-1] 2021-03-12 13:53:58 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:53:58 AISGPGP01 pgpool[29918]: [54-1] 2021-03-12 13:53:58 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:53:58 AISGPGP01 pgpool[29918]: [54-2] 2021-03-12 13:53:58 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd01.cca.cz:5432" failed
Mar 12 13:53:58 AISGPGP01 pgpool[29918]: [55-1] 2021-03-12 13:53:58 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  health check retrying on DB node: 0 (round:3)
Mar 12 13:53:59 AISGPGP01 pgpool[29845]: [34-1] 2021-03-12 13:53:59 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
Mar 12 13:53:59 AISGPGP01 pgpool[29845]: [35-1] 2021-03-12 13:53:59 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:53:59 AISGPGP01 pgpool[29845]: [35-2] 2021-03-12 13:53:59 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd02.cca.cz:5432" failed
Mar 12 13:53:59 AISGPGP01 pgpool[29845]: [36-1] 2021-03-12 13:53:59 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  health check retrying on DB node: 1 (round:3)
Mar 12 13:54:10 AISGPGP01 pgpool[29845]: [37-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  failed to connect to PostgreSQL server on "aisgdbd02.cca.cz:5432" using INET socket
Mar 12 13:54:10 AISGPGP01 pgpool[29918]: [56-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  failed to connect to PostgreSQL server on "aisgdbd01.cca.cz:5432" using INET socket
Mar 12 13:54:10 AISGPGP01 pgpool[29845]: [37-2] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  health check timer expired
Mar 12 13:54:10 AISGPGP01 pgpool[29918]: [56-2] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  health check timer expired
Mar 12 13:54:10 AISGPGP01 pgpool[29845]: [38-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:54:10 AISGPGP01 pgpool[29918]: [57-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |ERROR:  failed to make persistent db connection
Mar 12 13:54:10 AISGPGP01 pgpool[29845]: [38-2] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd02.cca.cz:5432" failed
Mar 12 13:54:10 AISGPGP01 pgpool[29918]: [57-2] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  connection to host:"aisgdbd01.cca.cz:5432" failed
Mar 12 13:54:10 AISGPGP01 pgpool[29845]: [39-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  health check failed on node 1 (timeout:1)
Mar 12 13:54:10 AISGPGP01 pgpool[29918]: [58-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  health check failed on node 0 (timeout:1)
Mar 12 13:54:10 AISGPGP01 pgpool[29845]: [40-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  received degenerate backend request for node_id: 1 from pid [29845]
Mar 12 13:54:10 AISGPGP01 pgpool[29918]: [59-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  received degenerate backend request for node_id: 0 from pid [29918]
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [127-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  new IPC connection received
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [128-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  new outbound connection to aisgpgp02.cca.cz:9000
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [129-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  new IPC connection received
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [130-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  watchdog received the failover command from local pgpool-II on IPC interface
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [131-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [132-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  we have got the consensus to perform the failover
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [132-2] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  1 node(s) voted in the favor
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [133-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  watchdog received the failover command from local pgpool-II on IPC interface
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [134-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [135-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  we have got the consensus to perform the failover
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [135-2] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  1 node(s) voted in the favor
Mar 12 13:54:10 AISGPGP01 pgpool[29587]: [1133-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  Pgpool-II parent process has received failover request
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [136-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  new IPC connection received
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [137-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  received the failover indication from Pgpool-II on IPC interface
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [138-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  watchdog is informed of failover start by the main process
Mar 12 13:54:10 AISGPGP01 pgpool[29587]: [1134-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  starting degeneration. shutdown host aisgdbd01.cca.cz(5432)
Mar 12 13:54:10 AISGPGP01 pgpool[29587]: [1135-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  Restart all children
Mar 12 13:54:10 AISGPGP01 pgpool[29587]: [1136-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  execute command: /etc/pgpool-II/failover.sh 0 aisgdbd01.cca.cz 5432 /u01/pgsql/11/data 1 aisgdbd02.cca.cz 0 0 5432 /u01/pgsql/11/data aisgdbd01.cca.cz 5432
Mar 12 13:54:10 AISGPGP01 pgpool[29587]: [1137-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  find_primary_node_repeatedly: waiting for finding a primary node
Mar 12 13:54:10 AISGPGP01 pgpool[29587]: [1138-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  find_primary_node: primary node is 1
Mar 12 13:54:10 AISGPGP01 pgpool[29587]: [1139-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  failover: set new primary node: 1
Mar 12 13:54:10 AISGPGP01 pgpool[29587]: [1140-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  failover: set new master node: 1
Mar 12 13:54:10 AISGPGP01 pgpool[31220]: [1141-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  failover or failback event detected
Mar 12 13:54:10 AISGPGP01 pgpool[31220]: [1141-2] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  restarting myself
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [139-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  new IPC connection received
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [140-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  received the failover indication from Pgpool-II on IPC interface
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [141-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  watchdog is informed of failover end by the main process
Mar 12 13:54:10 AISGPGP01 pgpool[29587]: [1141-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  failover done. shutdown host aisgdbd01.cca.cz(5432)
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [142-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  new IPC connection received
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [143-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  received the failover indication from Pgpool-II on IPC interface
Mar 12 13:54:10 AISGPGP01 pgpool[29589]: [144-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  watchdog is informed of failover start by the main process
Mar 12 13:54:10 AISGPGP01 pgpool[29587]: [1142-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  starting degeneration. shutdown host aisgdbd02.cca.cz(5432)
Mar 12 13:54:10 AISGPGP01 pgpool[29587]: [1143-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |WARNING:  All the DB nodes are in down status and skip writing status file.
Mar 12 13:54:10 AISGPGP01 pgpool[29587]: [1144-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  failover: no valid backend node found
Mar 12 13:54:10 AISGPGP01 pgpool[29587]: [1145-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  Restart all children
Mar 12 13:54:10 AISGPGP01 pgpool[29587]: [1146-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  execute command: /etc/pgpool-II/failover.sh 0 aisgdbd01.cca.cz 5432 /u01/pgsql/11/data -1 "" 1 1 "" "" aisgdbd02.cca.cz 5432
Mar 12 13:54:10 AISGPGP01 pgpool[29587]: [1147-1] 2021-03-12 13:54:10 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  execute command: /etc/pgpool-II/failover.sh 1 aisgdbd02.cca.cz 5432 /u01/pgsql/11/data -1 "" 1 1 "" "" aisgdbd02.cca.cz 5432
Mar 12 13:54:11 AISGPGP01 pgpool[29587]: [1148-1] 2021-03-12 13:54:11 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  find_primary_node_repeatedly: all of the backends are down. Giving up finding primary node
Mar 12 13:54:11 AISGPGP01 pgpool[29587]: [1149-1] 2021-03-12 13:54:11 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  failover: set new primary node: -1
Mar 12 13:54:11 AISGPGP01 pgpool[29589]: [145-1] 2021-03-12 13:54:11 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  new IPC connection received
Mar 12 13:54:11 AISGPGP01 pgpool[29589]: [146-1] 2021-03-12 13:54:11 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  received the failover indication from Pgpool-II on IPC interface
Mar 12 13:54:11 AISGPGP01 pgpool[29589]: [147-1] 2021-03-12 13:54:11 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  watchdog is informed of failover end by the main process
Mar 12 13:54:11 AISGPGP01 pgpool[31513]: [1150-1] 2021-03-12 13:54:11 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  failover or failback event detected
Mar 12 13:54:11 AISGPGP01 pgpool[31513]: [1150-2] 2021-03-12 13:54:11 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  restarting myself
Mar 12 13:54:11 AISGPGP01 pgpool[29587]: [1150-1] 2021-03-12 13:54:11 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  failover done. shutdown host aisgdbd02.cca.cz(5432)
Mar 12 13:54:12 AISGPGP01 pgpool[30217]: [608-1] 2021-03-12 13:54:12 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  restart request received in pcp child process
Mar 12 13:54:12 AISGPGP01 pgpool[29587]: [1151-1] 2021-03-12 13:54:12 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  PCP child 30217 exits with status 0 in failover()
Mar 12 13:54:12 AISGPGP01 pgpool[29587]: [1152-1] 2021-03-12 13:54:12 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  fork a new PCP child pid 31518 in failover()
Mar 12 13:54:12 AISGPGP01 pgpool[29587]: [1153-1] 2021-03-12 13:54:12 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  child process with pid: 30156 exits with status 256
Mar 12 13:54:12 AISGPGP01 pgpool[29587]: [1154-1] 2021-03-12 13:54:12 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  child process with pid: 30156 exited with success and will not be restarted
Mar 12 13:54:12 AISGPGP01 pgpool[29587]: [1155-1] 2021-03-12 13:54:12 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  child process with pid: 30164 exits with status 256
Mar 12 13:54:12 AISGPGP01 pgpool[29587]: [1156-1] 2021-03-12 13:54:12 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  child process with pid: 30164 exited with success and will not be restarted
Mar 12 13:54:12 AISGPGP01 pgpool[29587]: [1157-1] 2021-03-12 13:54:12 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  child process with pid: 30219 exits with status 256
Mar 12 13:54:12 AISGPGP01 pgpool[29587]: [1158-1] 2021-03-12 13:54:12 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  child process with pid: 30219 exited with success and will not be restarted
Mar 12 13:54:12 AISGPGP01 pgpool[29587]: [1159-1] 2021-03-12 13:54:12 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  child process with pid: 30220 exits with status 256
Mar 12 13:54:12 AISGPGP01 pgpool[29587]: [2031-1] 2021-03-12 13:54:12 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  child process with pid: 31173 exits with status 256
Mar 12 13:54:12 AISGPGP01 pgpool[29587]: [2032-1] 2021-03-12 13:54:12 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  child process with pid: 31173 exited with success and will not be restarted

...
...
...

Mar 12 13:54:17 AISGPGP01 pgpool[30218]: [611-1] 2021-03-12 13:54:17 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  worker process received restart request
Mar 12 13:54:17 AISGPGP01 pgpool[29587]: [2155-1] 2021-03-12 13:54:17 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  worker child process with pid: 30218 exits with status 256
Mar 12 13:54:17 AISGPGP01 pgpool[29587]: [2156-1] 2021-03-12 13:54:17 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  fork a new worker child process with pid: 31528
Mar 12 13:54:17 AISGPGP01 pgpool[29589]: [162-1] 2021-03-12 13:54:17 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  new watchdog node connection is received from "172.20.15.65:18591"
Mar 12 13:54:17 AISGPGP01 pgpool[29589]: [163-1] 2021-03-12 13:54:17 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  new node joined the cluster hostname:"aisgpgp02.cca.cz" port:9000 pgpool_port:9999
Mar 12 13:54:17 AISGPGP01 pgpool[29589]: [163-2] 2021-03-12 13:54:17 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  Pgpool-II version:"4.1.5" watchdog messaging version: 1.1
Mar 12 13:54:21 AISGPGP01 pgpool[29589]: [164-1] 2021-03-12 13:54:21 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  adding watchdog node "aisgpgp02.cca.cz:9999 Linux AISGPGP02.cca.cz" to the standby list
Mar 12 13:54:21 AISGPGP01 pgpool[29587]: [2157-1] 2021-03-12 13:54:21 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  Pgpool-II parent process received watchdog quorum change signal from watchdog
Mar 12 13:54:21 AISGPGP01 pgpool[29589]: [165-1] 2021-03-12 13:54:21 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  new IPC connection received
Mar 12 13:54:21 AISGPGP01 pgpool[29587]: [2158-1] 2021-03-12 13:54:21 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  watchdog cluster now holds the quorum
Mar 12 13:54:21 AISGPGP01 pgpool[29587]: [2158-2] 2021-03-12 13:54:21 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  updating the state of quarantine backend nodes
Mar 12 13:54:21 AISGPGP01 pgpool[29589]: [166-1] 2021-03-12 13:54:21 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  new IPC connection received
Mar 12 13:54:25 AISGPGP01 pgpool[29589]: [167-1] 2021-03-12 13:54:25 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  remote node "aisgpgp02.cca.cz:9999 Linux AISGPGP02.cca.cz" is not replying to our beacons
Mar 12 13:54:25 AISGPGP01 pgpool[29589]: [167-2] 2021-03-12 13:54:25 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  missed beacon reply count:2
Mar 12 13:54:28 AISGPGP01 pgpool[31518]: [1153-1] 2021-03-12 13:54:28 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  forked new pcp worker, pid=31530 socket=8
Mar 12 13:54:28 AISGPGP01 pgpool[29589]: [168-1] 2021-03-12 13:54:28 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  new IPC connection received
Mar 12 13:54:28 AISGPGP01 pgpool[31518]: [1154-1] 2021-03-12 13:54:28 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  PCP process with pid: 31530 exit with SUCCESS.
Mar 12 13:54:28 AISGPGP01 pgpool[31518]: [1155-1] 2021-03-12 13:54:28 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  PCP process with pid: 31530 exits with status 0
Mar 12 13:54:31 AISGPGP01 pgpool[29589]: [169-1] 2021-03-12 13:54:31 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |LOG:  remote node "aisgpgp02.cca.cz:9999 Linux AISGPGP02.cca.cz" is replying again after missing 2 beacons
Mar 12 13:54:32 AISGPGP01 pgpool[31519]: [2154-1] 2021-03-12 13:54:32 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |FATAL:  pgpool is not accepting any new connections
Mar 12 13:54:32 AISGPGP01 pgpool[31519]: [2154-2] 2021-03-12 13:54:32 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |DETAIL:  all backend nodes are down, pgpool requires at least one valid node
Mar 12 13:54:32 AISGPGP01 pgpool[31519]: [2154-3] 2021-03-12 13:54:32 ::: APP - [No Connection] :A: DB - [No Connection] :D: USER - [No Connection] :U: |HINT:  repair the backend nodes and restart pgpool