0000617: Loadbalancer ambiguous nature after restoring old master

ID	Project	Category	View Status	Date Submitted	Last Update

0000617	Pgpool-II	Bug	public	2020-06-15 18:03	2020-06-24 17:46

Reporter	sajith.t.s	Assigned To	pengbo
Priority	normal	Severity	minor	Reproducibility	N/A
Status	closed	Resolution	open
Product Version	3.6.20

Summary	0000617: Loadbalancer ambiguous nature after restoring old master
Description	Hi team, I have been using PgPool-3.6.20 and EDB Failover manager together to achieve High Availbaility for my EnterpriseDB Advanced Server. I am using; -- EDB AS 10 as database -- efm3.9 for Automatic failover -- PgPool 3.6.20 for Loadbalancing Loadbalancing implemented and everything looks fine. When the Master fails, efm3.9 performs automatic failover and the same is updated on PgPool. But, when the old master restored, PgPool thinks old master is the actual master. I am attaching a screenshot . The 1st query on that screenshot is the status before restoring old master as standby. The 2nd query on that screenshot is the status after restoring old master as standby. My loadbalancer scripts are given #!/bin/bash ################################################################################### #title : efm_loadbalance_attach.sh #description : Script executes pcp_attach_node/pcp_promote_node command to # : notify pgpool for NEW Master or Attach returning nodes back to # : pgpool cluster. #date : June 15, 2020 #version : 1.0 #bash_version : GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu) #author : # #Note: Script runs pcp_* commands. Password less PCP configuration is MUST.. # ################################################################################### # quit on any error - verify any undefined shell variable set -e set -u # EFM variable #------------- EFM_HOST_FROM_HOOK=$1 # argument from EFM fencing hook # Pgpool cluster connection information #-------------------------------------- PCP_USER=postgres # PCP user name (enterprisedb or postgres) PCP_PORT=9898 # PCP port number as in pgpool.conf PCP_HOST=192.168.29.129 # hostname of Pgpool-II PGPOOL_PATH=/usr/edb/pgpool3.6/bin # Pgpool-II installation path PGPOOL_PORT=9999 PCPPASSFILE=/var/efm/pcppass # Path to PCPPASS file # EPAS/PG DB connection information #----------------------------------- PGPATH=/opt/edb/as10 # Path to EDB binaries PGUSER=postgres PGPORT=5444 PGHOST=${EFM_HOST_FROM_HOOK} # Pgpool pcp & db check commands #------------------------------- PCP_NODE_COUNT=${PGPOOL_PATH}/pcp_node_count PCP_ATTACH_NODE=${PGPOOL_PATH}/pcp_attach_node PCP_PROMOTE_NODE=${PGPOOL_PATH}/pcp_promote_node PCP_NODE_INFO=${PGPOOL_PATH}/pcp_node_info PG_ISREADY=${PGPATH}/bin/pg_isready export PCPPASSFILE PCP_USER PCP_PORT PCP_HOST PGPOOL_PATH PGPOOL_PORT PGPATH PGUSER PGPORT PG_ISREADY PCP_NODE_INFO PCP_NODE_COUNT PCP_ATTACH_NODE PCP_PROMOTE_NODE logfile=/var/log/efm-3.9/efm-scripts/efm_attach_"`date +"%Y%m%d%H%M%S"`".log # Print Node status from pool_nodes to log #----------------------------------------- print_node_status() { echo "--------------------------------------------">>$logfile echo " Node IP: $1 Node ID: $2 $3 :">>$logfile echo "--------------------------------------------">>$logfile ${PCP_NODE_INFO} -h ${PCP_HOST} -U ${PCP_USER} -p ${PCP_PORT} -w -n ${2} >>$logfile } # Get db running or not #---------------------- db_status_check() { local val=$(${PG_ISREADY} -h ${1} -U ${PGUSER} -p ${PGPORT} -t 0 -d template1 \| grep "accepting" \| wc -l) echo "$val" } #Get Node ID information from show pool_nodes #--------------------------------------------- get_node_id_using_pcp() { NO_OF_NODES=$(${PCP_NODE_COUNT} --host=${PCP_HOST} --username=${PCP_USER} --port=${PCP_PORT} --no-password ) for (( i=0 ; i < ${NO_OF_NODES} ; i++ )) do exists=$(${PCP_NODE_INFO} --host=${PCP_HOST} --username=${PCP_USER} --port=${PCP_PORT} --no-password ${i} \| grep ${1} \| wc -l) if [[ ${exists} -eq 1 ]]; then NODE_ID=${i} break fi done echo "$NODE_ID" } # Check local node is in Recovery or not #--------------------------------------- db_recovery_check() { local dbcheck=$(db_status_check ${PGHOST}) val=x if [[ ${dbcheck} -eq 1 ]]; then val=$(${PGPATH}/bin/psql -h ${PGHOST} -U ${PGUSER} -p ${PGPORT} -d template1 -Atc "select pg_is_in_recovery();") fi echo "$val" } # pcp_attach_node command # ----------------------- attach_node_to_pgpool() { echo -e "\nAttaching Node :">>$logfile echo "-----------------">>$logfile ${PCP_ATTACH_NODE} --host=${PCP_HOST} --username=${PCP_USER} --port=${PCP_PORT} --no-password --verbose ${1} >>$logfile } attaching_remaining_nodes_to_pgpool() { local tot_nodes=$(${PCP_NODE_COUNT} --host=${PCP_HOST} --username=${PCP_USER} --port=${PCP_PORT} --no-password ) for (( j=0; j < 2; j++ )) do for (( i=0 ; i < ${tot_nodes} ; i++ )) do if [[ ${i} -ne ${1} ]]; then node_ip=$(${PCP_NODE_INFO} --host=${PCP_HOST} --username=${PCP_USER} --port=${PCP_PORT} --no-password ${i} \| cut -d" " -f1) if [[ ! -z ${node_ip} ]]; then local node_state=$(db_status_check $node_ip) if [[ ${node_state} -eq 1 ]]; then echo -e "\nAttaching Node-ID: ${i} Node-IP: ${node_ip} NodeDBStatus[1-UP/0-DOWN] : ${node_state}">>$logfile attach_node_to_pgpool "${i}" print_node_status "${node_ip}" "${i}" "after attach" fi fi fi done echo -e "\n* Re-confirming DB Status to attach,due to delay when performing switchover/failover *">>$logfile done } # Get Node Id from PgPool cluster & DB status #-------------------------------------------- EFM_HOOKIP_NODE_ID=$(get_node_id_using_pcp $EFM_HOST_FROM_HOOK) DB_RECOVERY_STATUS=$(db_recovery_check) echo "-----------------------------------------------------------">>$logfile echo "EFM Attaching Nodes Rules :- ">>$logfile echo "--------------------------">>$logfile echo " Rule 1: Node should be attached if DB is in Recovery">>$logfile echo " Rule 2: Node should be Promoted if DB is NOT in Recovery">>$logfile echo "-----------------------------------------------------------">>$logfile echo "EFM Attaching Node : $EFM_HOST_FROM_HOOK">>$logfile echo "DB Recovery Status[True/False]: ${DB_RECOVERY_STATUS} ">>$logfile echo "Node ID in PgPool Cluster : ${EFM_HOOKIP_NODE_ID}">>$logfile echo "-----------------------------------------------------------">>$logfile # Attach Node if DB node is back as Standby # ----------------------------------------- if [[ ${DB_RECOVERY_STATUS} == "t" ]]; then print_node_status $EFM_HOST_FROM_HOOK ${EFM_HOOKIP_NODE_ID} "before attach" attach_node_to_pgpool ${EFM_HOOKIP_NODE_ID} print_node_status $EFM_HOST_FROM_HOOK ${EFM_HOOKIP_NODE_ID} "after attach" fi # Promote Only if Node is back as Master # -------------------------------------- if [[ ${DB_RECOVERY_STATUS} == "f" ]]; then ${PCP_PROMOTE_NODE} --host=${PCP_HOST} --username=${PCP_USER} --port=${PCP_PORT} --no-password --verbose ${EFM_HOOKIP_NODE_ID} >>$logfile print_node_status $EFM_HOST_FROM_HOOK ${EFM_HOOKIP_NODE_ID} "after promote" attaching_remaining_nodes_to_pgpool $EFM_HOOKIP_NODE_ID fi echo -e "\nNote: Check 'show pool_nodes;' if logs show node status as 'down/waiting'">>$logfile exit 0 DETATCH SCRIPT `````````````` #!/bin/bash ################################################################################### #title : efm_loadbalance_detach.sh #description : Script executes pcp_detach_node command to notify pgpool on node # : removal from pgpool cluster. #date : June 26, 2020 #version : 1.0 #bash_version : GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu) #author : # #Mandatory : SCRIPT REQUIRES PASSWORD LESS EXECUTION. PCPPASS SHOULD BE # : CONFIGURED BEFORE CALLING IT. REFER TO THE BLOG ON THE USAGE ################################################################################### # quit on any error set -e set -u # Set environment variables, so EFM can connect to pgpool cluster and database # to perform node status checks # # EFM variable #------------- EFM_HOST_FROM_HOOK=$1 # argument from EFM fencing hook # Pgpool cluster connection information PCP_USER=postgres # PCP user name (enterprisedb or postgres) PCP_PORT=9898 # PCP port number as in pgpool.conf PCP_HOST=192.168.29.129 # hostname of Pgpool-II PGPOOL_PATH=/usr/edb/pgpool3.6/bin # Pgpool-II installation path PCPPASSFILE=/var/efm/pcppass # Path to PCPPASS file PGPOOL_PORT=9999 # EPAS/PG DB connection information #----------------------------------- PGPATH=/opt/edb/as10 # Path to EDB binaries PGUSER=postgres PGPORT=5444 PGHOST=${EFM_HOST_FROM_HOOK} # Pgpool pcp & db check commands #------------------------------- PCP_NODE_COUNT=${PGPOOL_PATH}/pcp_node_count PCP_NODE_INFO=${PGPOOL_PATH}/pcp_node_info PCP_DETACH_NODE=${PGPOOL_PATH}/pcp_detach_node PG_ISREADY=${PGPATH}/bin/pg_isready export PCPPASSFILE PCP_USER PCP_PORT PCP_HOST PGPOOL_PATH PGPOOL_PORT PGPATH PGUSER PGHOST PGPORT PG_ISREADY PCP_DETACH_NODE PCP_NODE_COUNT logfile=/var/log/efm-3.9/efm-scripts/efm_detach_script_"`date +"%Y%m%d%H%M%S"`".log # Print Node status from pool_nodes to log #----------------------------------------- print_node_status() { echo -e "\nNode $EFM_HOST_FROM_HOOK Status $2 $3 :">>$logfile echo "--------------------------------------------">>$logfile ${PGPATH}/bin/psql -h ${PCP_HOST} -U ${PCP_USER} -p ${PGPOOL_PORT} -d template1 -c "show pool_nodes;" \| awk -v ip="$1" 'NR==1 \|\| NR==2\|\| $0 ~ ip' >>$logfile } #Get Node ID information from show pool_nodes #--------------------------------------------- get_info_from_pool_nodes() { NO_OF_NODES=$(${PCP_NODE_COUNT} --host=${PCP_HOST} --username=${PCP_USER} --port=${PCP_PORT} --no-password ) for (( i=0 ; i < ${NO_OF_NODES} ; i++ )) do exists=$(${PCP_NODE_INFO} --host=${PCP_HOST} --username=${PCP_USER} --port=${PCP_PORT} --no-password ${i} \|grep ${1} \| wc -l) if [[ ${exists} -eq 1 ]]; then NODE_ID=${i} break fi done echo "$NODE_ID" } # Get db running or not #---------------------- db_status_check() { local val=$(${PG_ISREADY} -h ${PGHOST} -U ${PGUSER} -p ${PGPORT} -t 0 -d template1 \| grep "accepting" \| wc -l) echo "$val" } DB_STATUS=$(db_status_check) echo "Dettach Node = ( DBNode=Down )">>$logfile echo "DB Status on $EFM_HOST_FROM_HOOK: `[[ $DB_STATUS -eq 1 ]] && { echo UP; } \|\| { echo DOWN; }`">>$logfile if [[ ${DB_STATUS} -eq 0 ]]; then echo "Performing Detach operation">>$logfile print_node_status $EFM_HOST_FROM_HOOK before detach EFM_HOOKIP_NODE_ID=$(get_info_from_pool_nodes $EFM_HOST_FROM_HOOK 1) echo -e "\nDetaching Node :">>$logfile echo "-----------------">>$logfile ${PCP_DETACH_NODE} --host=${PCP_HOST} --username=${PCP_USER} --port=${PCP_PORT} --no-password --verbose ${EFM_HOOKIP_NODE_ID} >>$logfile print_node_status $EFM_HOST_FROM_HOOK after detach fi exit 0
Additional Information	PgPool log Info 2020-06-15 04:52:53: pid 61605: LOG: Backend status file /var/log/edb/pgpool3.6/pgpool_status does not exist 2020-06-15 04:52:53: pid 61605: LOG: Setting up socket for 0.0.0.0:9999 2020-06-15 04:52:53: pid 61605: LOG: Setting up socket for :::9999 2020-06-15 04:52:53: pid 61605: LOG: pgpool-II successfully started. version 3.6.20 (subaruboshi) 2020-06-15 04:53:31: pid 61639: LOG: forked new pcp worker, pid=61667 socket=7 2020-06-15 04:53:31: pid 61639: LOG: PCP process with pid: 61667 exit with SUCCESS. 2020-06-15 04:53:31: pid 61639: LOG: PCP process with pid: 61667 exits with status 0 2020-06-15 04:54:11: pid 61639: LOG: forked new pcp worker, pid=61735 socket=7 2020-06-15 04:54:11: pid 61735: FATAL: invalid PCP packet 2020-06-15 04:54:11: pid 61735: DETAIL: incorrect packet length (2052) 2020-06-15 04:54:11: pid 61639: LOG: forked new pcp worker, pid=61736 socket=7 2020-06-15 04:54:11: pid 61736: FATAL: invalid PCP packet 2020-06-15 04:54:11: pid 61736: DETAIL: incorrect packet length (22528) 2020-06-15 04:54:11: pid 61639: LOG: PCP process with pid: 61735 exit with SUCCESS. 2020-06-15 04:54:11: pid 61639: LOG: PCP process with pid: 61735 exits with status 256 2020-06-15 04:54:11: pid 61639: LOG: PCP process with pid: 61736 exit with SUCCESS. 2020-06-15 04:54:11: pid 61639: LOG: PCP process with pid: 61736 exits with status 256 2020-06-15 04:59:19: pid 61605: LOG: child process with pid: 61624 exits with status 256 2020-06-15 04:59:19: pid 61605: LOG: fork a new child process with pid: 61884 2020-06-15 05:06:32: pid 61884: LOG: reading and processing packets 2020-06-15 05:06:32: pid 61884: DETAIL: postmaster on DB node 0 was shutdown by administrative command 2020-06-15 05:06:32: pid 61884: LOG: RAW_MODE:0 REAL_MASTER_NODE_ID:0 pool_is_node_to_be_sent_in_current_query:1 my_backend_status:3 2020-06-15 05:06:32: pid 61884: ERROR: unable to read message kind from backend 2020-06-15 05:06:32: pid 61884: DETAIL: 0 th backend is not valid 2020-06-15 05:06:58: pid 61605: WARNING: child process with pid: 61884 was terminated by segmentation fault 2020-06-15 05:06:58: pid 61605: LOG: fork a new child process with pid: 62687 2020-06-15 05:07:06: pid 62687: LOG: failed to connect to PostgreSQL server on "192.168.29.130:5444", getsockopt() detected error "Connection refused" 2020-06-15 05:07:06: pid 62687: LOG: received degenerate backend request for node_id: 0 from pid [62687] 2020-06-15 05:07:06: pid 62687: FATAL: failed to create a backend connection 2020-06-15 05:07:06: pid 62687: DETAIL: executing failover on backend 2020-06-15 05:07:06: pid 61605: LOG: Pgpool-II parent process has received failover request 2020-06-15 05:07:06: pid 61605: LOG: starting degeneration. shutdown host 192.168.29.130(5444) 2020-06-15 05:07:06: pid 61605: LOG: Restart all children 2020-06-15 05:07:06: pid 61605: LOG: failover: set new primary node: -1 2020-06-15 05:07:06: pid 61605: LOG: failover: set new master node: 1 failover done. shutdown host 192.168.29.130(5444)2020-06-15 05:07:06: pid 61605: LOG: failover done. shutdown host 192.168.29.130(5444) 2020-06-15 05:07:06: pid 61640: LOG: worker process received restart request 2020-06-15 05:07:07: pid 61639: LOG: restart request received in pcp child process 2020-06-15 05:07:07: pid 61605: LOG: PCP child 61639 exits with status 0 in failover() 2020-06-15 05:07:07: pid 61605: LOG: fork a new PCP child pid 62728 in failover() 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61607 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61607 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61608 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61608 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61609 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61609 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61610 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61610 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61611 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61611 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61612 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61612 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61613 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61613 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61614 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61614 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61615 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61615 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61616 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61616 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61617 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61617 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61618 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61618 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61619 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61619 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61620 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61620 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61621 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61621 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61622 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61622 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61623 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61623 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61625 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61625 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61626 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61626 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61627 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61627 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61628 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61628 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61629 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61629 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61630 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61630 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61631 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61631 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61632 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61632 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61633 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61633 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61634 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61634 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61635 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61635 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61636 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61636 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61637 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61637 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61638 exits with status 0 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 61638 exited with success and will not be restarted 2020-06-15 05:07:07: pid 61605: LOG: worker child process with pid: 61640 exits with status 256 2020-06-15 05:07:07: pid 61605: LOG: fork a new worker child process with pid: 62729 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 62687 exits with status 256 2020-06-15 05:07:07: pid 61605: LOG: child process with pid: 62687 exited with success and will not be restarted 2020-06-15 05:14:42: pid 61605: LOG: child process with pid: 62723 exits with status 256 2020-06-15 05:14:42: pid 61605: LOG: fork a new child process with pid: 63142 2020-06-15 05:16:57: pid 62728: LOG: forked new pcp worker, pid=63218 socket=7 2020-06-15 05:16:57: pid 62728: LOG: PCP process with pid: 63218 exit with SUCCESS. 2020-06-15 05:16:57: pid 62728: LOG: PCP process with pid: 63218 exits with status 0 2020-06-15 05:16:57: pid 62728: LOG: forked new pcp worker, pid=63219 socket=7 2020-06-15 05:16:57: pid 62728: LOG: PCP process with pid: 63219 exit with SUCCESS. 2020-06-15 05:16:57: pid 62728: LOG: PCP process with pid: 63219 exits with status 0 2020-06-15 05:16:57: pid 62728: LOG: forked new pcp worker, pid=63220 socket=7 2020-06-15 05:16:57: pid 62728: LOG: PCP process with pid: 63220 exit with SUCCESS. 2020-06-15 05:16:57: pid 62728: LOG: PCP process with pid: 63220 exits with status 0 2020-06-15 05:16:57: pid 62728: LOG: forked new pcp worker, pid=63221 socket=7 2020-06-15 05:16:57: pid 63221: LOG: received failback request for node_id: 0 from pid [63221] wd_failover_id [0] 2020-06-15 05:16:57: pid 61605: LOG: Pgpool-II parent process has received failover request 2020-06-15 05:16:57: pid 61605: LOG: starting fail back. reconnect host 192.168.29.130(5444) 2020-06-15 05:16:57: pid 61605: LOG: Node 1 is not down (status: 2) 2020-06-15 05:16:57: pid 62728: LOG: PCP process with pid: 63221 exit with SUCCESS. 2020-06-15 05:16:57: pid 62728: LOG: PCP process with pid: 63221 exits with status 0 2020-06-15 05:16:57: pid 61605: LOG: Restart all children 2020-06-15 05:16:57: pid 61605: LOG: failover: set new primary node: -1 2020-06-15 05:16:57: pid 61605: LOG: failover: set new master node: 0 2020-06-15 05:16:57: pid 62728: LOG: forked new pcp worker, pid=63250 socket=7 2020-06-15 05:16:57: pid 61605: LOG: failback done. reconnect host 192.168.29.130(5444) 2020-06-15 05:16:57: pid 62729: LOG: worker process received restart request 2020-06-15 05:16:57: pid 62728: LOG: PCP process with pid: 63250 exit with SUCCESS. 2020-06-15 05:16:57: pid 62728: LOG: PCP process with pid: 63250 exits with status 0 2020-06-15 05:16:58: pid 62728: LOG: restart request received in pcp child process 2020-06-15 05:16:58: pid 61605: LOG: PCP child 62728 exits with status 0 in failover() 2020-06-15 05:16:58: pid 61605: LOG: fork a new PCP child pid 63256 in failover() 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62695 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62695 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62696 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62696 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62697 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62697 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62698 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62698 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62699 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62699 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62700 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62700 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62701 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62701 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62702 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62702 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62703 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62703 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62704 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62704 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62705 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62705 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62706 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62706 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62707 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62707 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62708 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62708 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62709 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62709 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62710 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62710 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62711 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62711 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62712 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62712 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62713 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62713 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62714 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62714 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62715 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62715 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62716 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62716 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62717 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62717 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62718 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62718 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62719 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62719 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62720 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62720 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62721 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62721 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62722 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62722 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62724 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62724 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62725 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62725 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62726 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 62726 exited with success and will not be restarted 2020-06-15 05:16:58: pid 61605: LOG: worker child process with pid: 62729 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: fork a new worker child process with pid: 63257 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 63142 exits with status 256 2020-06-15 05:16:58: pid 61605: LOG: child process with pid: 63142 exited with success and will not be restarted 2020-06-15 05:18:48: pid 63254: LOG: pool_send_and_wait: Error or notice message from backend: : DB node id: 0 backend pid: 62604 statement: "insert into tab values(10);" message: "cannot execute INSERT in a read-only transaction" 2020-06-15 05:33:13: pid 63934: LOG: stop request sent to pgpool. waiting for termination... ...NOTICE: pid file found but it seems bogus. Trying to start pgpool anyway... 2020-06-15 05:35:15: pid 63997: LOG: Backend status file /var/log/edb/pgpool3.6/pgpool_status discarded 2020-06-15 05:35:15: pid 63997: LOG: Setting up socket for 0.0.0.0:9999 2020-06-15 05:35:15: pid 63997: LOG: Setting up socket for :::9999 2020-06-15 05:35:15: pid 63997: LOG: pgpool-II successfully started. version 3.6.20 (subaruboshi) 2020-06-15 05:35:58: pid 64028: LOG: Unable to parse the query: "insert ito tab values(8);" from client 192.168.29.131(34776) 2020-06-15 05:35:58: pid 64028: LOG: pool_send_and_wait: Error or notice message from backend: : DB node id: 0 backend pid: 64062 statement: "insert ito tab values(8);" message: "syntax error at or near "ito"" 2020-06-15 05:37:18: pid 64028: LOG: reading and processing packets 2020-06-15 05:37:18: pid 64028: DETAIL: postmaster on DB node 0 was shutdown by administrative command 2020-06-15 05:37:18: pid 64028: LOG: RAW_MODE:0 REAL_MASTER_NODE_ID:0 pool_is_node_to_be_sent_in_current_query:1 my_backend_status:3 2020-06-15 05:37:18: pid 64028: ERROR: unable to read message kind from backend 2020-06-15 05:37:18: pid 64028: DETAIL: 0 th backend is not valid 2020-06-15 05:37:28: pid 63997: WARNING: child process with pid: 64028 was terminated by segmentation fault 2020-06-15 05:37:28: pid 63997: LOG: fork a new child process with pid: 64356 2020-06-15 05:37:28: pid 64031: LOG: forked new pcp worker, pid=64362 socket=7 2020-06-15 05:37:28: pid 64031: LOG: PCP process with pid: 64362 exit with SUCCESS. 2020-06-15 05:37:28: pid 64031: LOG: PCP process with pid: 64362 exits with status 0 2020-06-15 05:37:28: pid 64031: LOG: forked new pcp worker, pid=64368 socket=7 2020-06-15 05:37:28: pid 64031: LOG: PCP process with pid: 64368 exit with SUCCESS. 2020-06-15 05:37:28: pid 64031: LOG: PCP process with pid: 64368 exits with status 0 2020-06-15 05:37:28: pid 64031: LOG: forked new pcp worker, pid=64370 socket=7 2020-06-15 05:37:28: pid 64370: LOG: received degenerate backend request for node_id: 0 from pid [64370] 2020-06-15 05:37:28: pid 63997: LOG: Pgpool-II parent process has received failover request 2020-06-15 05:37:28: pid 63997: LOG: starting degeneration. shutdown host 192.168.29.129(5444) 2020-06-15 05:37:28: pid 63997: LOG: Restart all children 2020-06-15 05:37:28: pid 63997: LOG: failover: set new primary node: -1 2020-06-15 05:37:28: pid 63997: LOG: failover: set new master node: 1 failover done. shutdown host 192.168.29.129(5444)2020-06-15 05:37:28: pid 63997: LOG: failover done. shutdown host 192.168.29.129(5444) 2020-06-15 05:37:28: pid 64032: LOG: worker process received restart request 2020-06-15 05:37:28: pid 64031: LOG: PCP process with pid: 64370 exit with SUCCESS. 2020-06-15 05:37:28: pid 64031: LOG: PCP process with pid: 64370 exits with status 0 2020-06-15 05:37:29: pid 64031: LOG: restart request received in pcp child process 2020-06-15 05:37:29: pid 63997: LOG: PCP child 64031 exits with status 0 in failover() 2020-06-15 05:37:29: pid 63997: LOG: fork a new PCP child pid 64407 in failover() 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 63999 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 63999 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64000 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64000 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64001 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64001 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64002 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64002 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64003 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64003 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64004 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64004 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64005 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64005 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64006 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64006 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64007 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64007 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64008 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64008 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64009 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64009 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64010 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64010 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64011 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64011 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64012 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64012 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64013 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64013 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64014 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64014 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64015 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64015 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64016 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64016 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64017 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64017 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64018 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64018 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64019 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64019 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64020 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64020 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64021 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64021 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64022 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64022 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64023 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64023 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64024 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64024 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64025 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64025 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64026 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64026 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64027 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64027 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64029 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64029 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64030 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64030 exited with success and will not be restarted 2020-06-15 05:37:29: pid 63997: LOG: worker child process with pid: 64032 exits with status 256 2020-06-15 05:37:29: pid 63997: LOG: fork a new worker child process with pid: 64408 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64356 exits with status 0 2020-06-15 05:37:29: pid 63997: LOG: child process with pid: 64356 exited with success and will not be restarted 2020-06-15 05:37:34: pid 64407: LOG: forked new pcp worker, pid=64548 socket=7 2020-06-15 05:37:34: pid 64407: LOG: PCP process with pid: 64548 exit with SUCCESS. 2020-06-15 05:37:34: pid 64407: LOG: PCP process with pid: 64548 exits with status 0 2020-06-15 05:37:34: pid 64407: LOG: forked new pcp worker, pid=64549 socket=7 2020-06-15 05:37:34: pid 64407: LOG: PCP process with pid: 64549 exit with SUCCESS. 2020-06-15 05:37:34: pid 64407: LOG: PCP process with pid: 64549 exits with status 0 2020-06-15 05:37:34: pid 64407: LOG: forked new pcp worker, pid=64550 socket=7 2020-06-15 05:37:34: pid 64407: LOG: PCP process with pid: 64550 exit with SUCCESS. 2020-06-15 05:37:34: pid 64407: LOG: PCP process with pid: 64550 exits with status 0 2020-06-15 05:37:34: pid 64407: LOG: forked new pcp worker, pid=64551 socket=7 2020-06-15 05:37:34: pid 64551: FATAL: invalid pgpool mode for process recovery request 2020-06-15 05:37:34: pid 64551: DETAIL: not in streaming replication mode, can't promote node id 1 2020-06-15 05:37:34: pid 64407: LOG: PCP process with pid: 64551 exit with SUCCESS. 2020-06-15 05:37:34: pid 64407: LOG: PCP process with pid: 64551 exits with status 256 2020-06-15 05:37:34: pid 64407: LOG: forked new pcp worker, pid=64552 socket=7 2020-06-15 05:37:34: pid 64407: LOG: PCP process with pid: 64552 exit with SUCCESS. 2020-06-15 05:37:34: pid 64407: LOG: PCP process with pid: 64552 exits with status 0 2020-06-15 05:37:34: pid 64407: LOG: forked new pcp worker, pid=64553 socket=7 2020-06-15 05:37:34: pid 64407: LOG: PCP process with pid: 64553 exit with SUCCESS. 2020-06-15 05:37:34: pid 64407: LOG: PCP process with pid: 64553 exits with status 0 2020-06-15 05:37:34: pid 64407: LOG: forked new pcp worker, pid=64554 socket=7 2020-06-15 05:37:34: pid 64407: LOG: PCP process with pid: 64554 exit with SUCCESS. 2020-06-15 05:37:34: pid 64407: LOG: PCP process with pid: 64554 exits with status 0 2020-06-15 05:37:34: pid 64407: LOG: forked new pcp worker, pid=64555 socket=7 2020-06-15 05:37:34: pid 64407: LOG: PCP process with pid: 64555 exit with SUCCESS. 2020-06-15 05:37:34: pid 64407: LOG: PCP process with pid: 64555 exits with status 0 2020-06-15 05:38:49: pid 64407: LOG: forked new pcp worker, pid=64794 socket=7 2020-06-15 05:38:49: pid 64407: LOG: PCP process with pid: 64794 exit with SUCCESS. 2020-06-15 05:38:49: pid 64407: LOG: PCP process with pid: 64794 exits with status 0 2020-06-15 05:38:49: pid 64407: LOG: forked new pcp worker, pid=64799 socket=7 2020-06-15 05:38:49: pid 64407: LOG: PCP process with pid: 64799 exit with SUCCESS. 2020-06-15 05:38:49: pid 64407: LOG: PCP process with pid: 64799 exits with status 0 2020-06-15 05:38:49: pid 64407: LOG: forked new pcp worker, pid=64815 socket=7 2020-06-15 05:38:49: pid 64407: LOG: PCP process with pid: 64815 exit with SUCCESS. 2020-06-15 05:38:49: pid 64407: LOG: PCP process with pid: 64815 exits with status 0 2020-06-15 05:38:49: pid 64407: LOG: forked new pcp worker, pid=64817 socket=7 2020-06-15 05:38:49: pid 64817: LOG: received failback request for node_id: 0 from pid [64817] wd_failover_id [0] 2020-06-15 05:38:49: pid 63997: LOG: Pgpool-II parent process has received failover request 2020-06-15 05:38:49: pid 63997: LOG: starting fail back. reconnect host 192.168.29.129(5444) 2020-06-15 05:38:49: pid 63997: LOG: Node 1 is not down (status: 2) 2020-06-15 05:38:49: pid 64407: LOG: PCP process with pid: 64817 exit with SUCCESS. 2020-06-15 05:38:49: pid 64407: LOG: PCP process with pid: 64817 exits with status 0 2020-06-15 05:38:49: pid 64407: LOG: forked new pcp worker, pid=64819 socket=7 2020-06-15 05:38:49: pid 64407: LOG: PCP process with pid: 64819 exit with SUCCESS. 2020-06-15 05:38:49: pid 64407: LOG: PCP process with pid: 64819 exits with status 0 2020-06-15 05:38:49: pid 63997: LOG: Restart all children 2020-06-15 05:38:49: pid 63997: LOG: failover: set new primary node: -1 2020-06-15 05:38:49: pid 63997: LOG: failover: set new master node: 0 2020-06-15 05:38:49: pid 63997: LOG: failback done. reconnect host 192.168.29.129(5444) 2020-06-15 05:38:49: pid 64408: LOG: worker process received restart request 2020-06-15 05:38:50: pid 64407: LOG: restart request received in pcp child process 2020-06-15 05:38:50: pid 63997: LOG: PCP child 64407 exits with status 0 in failover() 2020-06-15 05:38:50: pid 63997: LOG: fork a new PCP child pid 64876 in failover() 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64373 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64373 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64374 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64374 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64375 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64375 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64376 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64376 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64377 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64377 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64378 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64378 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64379 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64379 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64380 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64380 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64381 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64381 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64382 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64382 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64383 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64383 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64384 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64384 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64385 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64385 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64386 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64386 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64387 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64387 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64388 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64388 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64389 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64389 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64390 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64390 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64391 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64391 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64392 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64392 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64393 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64393 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64394 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64394 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64395 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64395 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64396 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64396 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64397 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64397 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64398 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64398 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64399 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64399 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64400 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64400 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64401 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64401 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64402 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64402 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64403 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64403 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64404 exits with status 0 2020-06-15 05:38:50: pid 63997: LOG: child process with pid: 64404 exited with success and will not be restarted 2020-06-15 05:38:50: pid 63997: LOG: worker child process with pid: 64408 exits with status 256 2020-06-15 05:38:50: pid 63997: LOG: fork a new worker child process with pid: 64877 2020-06-15 05:39:50: pid 64876: LOG: forked new pcp worker, pid=64999 socket=7 2020-06-15 05:39:50: pid 64999: FATAL: invalid PCP packet 2020-06-15 05:39:50: pid 64999: DETAIL: incorrect packet length (2052) 2020-06-15 05:39:50: pid 64876: LOG: forked new pcp worker, pid=65000 socket=7 2020-06-15 05:39:50: pid 64876: LOG: PCP process with pid: 64999 exit with SUCCESS. 2020-06-15 05:39:50: pid 64876: LOG: PCP process with pid: 64999 exits with status 256 2020-06-15 05:39:50: pid 65000: FATAL: invalid PCP packet 2020-06-15 05:39:50: pid 65000: DETAIL: incorrect packet length (22528) 2020-06-15 05:39:50: pid 64876: LOG: PCP process with pid: 65000 exit with SUCCESS. 2020-06-15 05:39:50: pid 64876: LOG: PCP process with pid: 65000 exits with status 256 2020-06-15 05:47:55: pid 63997: LOG: child process with pid: 64851 exits with status 256 2020-06-15 05:47:55: pid 63997: LOG: fork a new child process with pid: 65461 2020-06-15 05:49:38: pid 63997: LOG: child process with pid: 64831 exits with status 256 2020-06-15 05:49:38: pid 63997: LOG: fork a new child process with pid: 65504 2020-06-15 06:30:15: pid 66906: LOG: stop request sent to pgpool. waiting for termination... .done. 2020-06-15 06:30:17: pid 66912: LOG: Backend status file /var/log/edb/pgpool3.6/pgpool_status discarded 2020-06-15 06:30:17: pid 66912: LOG: Setting up socket for 0.0.0.0:9999 2020-06-15 06:30:17: pid 66912: LOG: Setting up socket for :::9999 2020-06-15 06:30:17: pid 66912: LOG: pgpool-II successfully started. version 3.6.20 (subaruboshi) 2020-06-15 06:30:53: pid 66934: LOG: failed to connect to PostgreSQL server on "192.168.29.130:5444", getsockopt() detected error "Connection refused" 2020-06-15 06:30:53: pid 66934: LOG: received degenerate backend request for node_id: 0 from pid [66934] 2020-06-15 06:30:53: pid 66934: FATAL: failed to create a backend connection 2020-06-15 06:30:53: pid 66934: DETAIL: executing failover on backend 2020-06-15 06:30:53: pid 66912: LOG: Pgpool-II parent process has received failover request 2020-06-15 06:30:53: pid 66912: LOG: starting degeneration. shutdown host 192.168.29.130(5444) 2020-06-15 06:30:53: pid 66912: LOG: Restart all children 2020-06-15 06:30:53: pid 66912: LOG: failover: set new primary node: -1 2020-06-15 06:30:53: pid 66912: LOG: failover: set new master node: 1 failover done. shutdown host 192.168.29.130(5444)2020-06-15 06:30:53: pid 66912: LOG: failover done. shutdown host 192.168.29.130(5444) 2020-06-15 06:30:53: pid 66947: LOG: worker process received restart request 2020-06-15 06:30:54: pid 66946: LOG: restart request received in pcp child process 2020-06-15 06:30:54: pid 66912: LOG: PCP child 66946 exits with status 0 in failover() 2020-06-15 06:30:54: pid 66912: LOG: fork a new PCP child pid 67022 in failover() 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66914 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66914 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66915 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66915 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66916 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66916 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66917 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66917 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66918 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66918 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66919 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66919 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66920 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66920 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66921 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66921 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66922 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66922 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66923 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66923 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66924 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66924 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66925 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66925 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66926 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66926 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66927 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66927 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66928 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66928 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66929 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66929 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66930 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66930 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66931 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66931 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66932 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66932 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66933 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66933 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66934 exits with status 0 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66934 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66935 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66935 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66936 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66936 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66937 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66937 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66938 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66938 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66939 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66939 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66940 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66940 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66941 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66941 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66942 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66942 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66943 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66943 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66944 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66944 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66945 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: child process with pid: 66945 exited with success and will not be restarted 2020-06-15 06:30:54: pid 66912: LOG: worker child process with pid: 66947 exits with status 256 2020-06-15 06:30:54: pid 66912: LOG: fork a new worker child process with pid: 67023 2020-06-15 06:31:06: pid 67022: LOG: forked new pcp worker, pid=67100 socket=7 2020-06-15 06:31:06: pid 67022: LOG: PCP process with pid: 67100 exit with SUCCESS. 2020-06-15 06:31:06: pid 67022: LOG: PCP process with pid: 67100 exits with status 0 2020-06-15 06:31:06: pid 67022: LOG: forked new pcp worker, pid=67105 socket=7 2020-06-15 06:31:06: pid 67022: LOG: PCP process with pid: 67105 exit with SUCCESS. 2020-06-15 06:31:06: pid 67022: LOG: PCP process with pid: 67105 exits with status 0 2020-06-15 06:31:06: pid 67022: LOG: forked new pcp worker, pid=67110 socket=7 2020-06-15 06:31:06: pid 67022: LOG: PCP process with pid: 67110 exit with SUCCESS. 2020-06-15 06:31:06: pid 67022: LOG: PCP process with pid: 67110 exits with status 0 2020-06-15 06:31:06: pid 67022: LOG: forked new pcp worker, pid=67124 socket=7 2020-06-15 06:31:06: pid 67124: FATAL: invalid pgpool mode for process recovery request 2020-06-15 06:31:06: pid 67124: DETAIL: not in streaming replication mode, can't promote node id 1 2020-06-15 06:31:06: pid 67022: LOG: PCP process with pid: 67124 exit with SUCCESS. 2020-06-15 06:31:06: pid 67022: LOG: PCP process with pid: 67124 exits with status 256 2020-06-15 06:31:06: pid 67022: LOG: forked new pcp worker, pid=67126 socket=7 2020-06-15 06:31:06: pid 67022: LOG: PCP process with pid: 67126 exit with SUCCESS. 2020-06-15 06:31:06: pid 67022: LOG: PCP process with pid: 67126 exits with status 0 2020-06-15 06:31:06: pid 67022: LOG: forked new pcp worker, pid=67128 socket=7 2020-06-15 06:31:06: pid 67022: LOG: PCP process with pid: 67128 exit with SUCCESS. 2020-06-15 06:31:06: pid 67022: LOG: PCP process with pid: 67128 exits with status 0 2020-06-15 06:31:06: pid 67022: LOG: forked new pcp worker, pid=67132 socket=7 2020-06-15 06:31:06: pid 67022: LOG: PCP process with pid: 67132 exit with SUCCESS. 2020-06-15 06:31:06: pid 67022: LOG: PCP process with pid: 67132 exits with status 0 2020-06-15 06:31:06: pid 67022: LOG: forked new pcp worker, pid=67141 socket=7 2020-06-15 06:31:06: pid 67022: LOG: PCP process with pid: 67141 exit with SUCCESS. 2020-06-15 06:31:06: pid 67022: LOG: PCP process with pid: 67141 exits with status 0 2020-06-15 06:31:31: pid 67022: LOG: forked new pcp worker, pid=67233 socket=7 2020-06-15 06:31:31: pid 67022: LOG: forked new pcp worker, pid=67234 socket=7 2020-06-15 06:31:31: pid 67022: LOG: PCP process with pid: 67233 exit with SUCCESS. 2020-06-15 06:31:31: pid 67022: LOG: PCP process with pid: 67233 exits with status 0 2020-06-15 06:31:31: pid 67022: LOG: PCP process with pid: 67234 exit with SUCCESS. 2020-06-15 06:31:31: pid 67022: LOG: PCP process with pid: 67234 exits with status 0 2020-06-15 06:31:31: pid 67022: LOG: forked new pcp worker, pid=67235 socket=7 2020-06-15 06:31:31: pid 67022: LOG: PCP process with pid: 67235 exit with SUCCESS. 2020-06-15 06:31:31: pid 67022: LOG: PCP process with pid: 67235 exits with status 0 2020-06-15 06:31:31: pid 67022: LOG: forked new pcp worker, pid=67236 socket=7 2020-06-15 06:31:31: pid 67236: LOG: received failback request for node_id: 0 from pid [67236] wd_failover_id [0] 2020-06-15 06:31:31: pid 66912: LOG: Pgpool-II parent process has received failover request 2020-06-15 06:31:31: pid 66912: LOG: starting fail back. reconnect host 192.168.29.130(5444) 2020-06-15 06:31:31: pid 66912: LOG: Node 1 is not down (status: 2) 2020-06-15 06:31:31: pid 67022: LOG: PCP process with pid: 67236 exit with SUCCESS. 2020-06-15 06:31:31: pid 67022: LOG: PCP process with pid: 67236 exits with status 0 2020-06-15 06:31:31: pid 66912: LOG: Restart all children 2020-06-15 06:31:31: pid 66912: LOG: failover: set new primary node: -1 2020-06-15 06:31:31: pid 66912: LOG: failover: set new master node: 0 2020-06-15 06:31:31: pid 66912: LOG: failback done. reconnect host 192.168.29.130(5444) 2020-06-15 06:31:31: pid 67023: LOG: worker process received restart request 2020-06-15 06:31:31: pid 67022: LOG: forked new pcp worker, pid=67269 socket=7 2020-06-15 06:31:31: pid 67022: LOG: PCP process with pid: 67269 exit with SUCCESS. 2020-06-15 06:31:31: pid 67022: LOG: PCP process with pid: 67269 exits with status 0 2020-06-15 06:31:32: pid 67022: LOG: restart request received in pcp child process 2020-06-15 06:31:32: pid 66912: LOG: PCP child 67022 exits with status 0 in failover() 2020-06-15 06:31:32: pid 66912: LOG: fork a new PCP child pid 67271 in failover() 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66984 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66984 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66985 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66985 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66986 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66986 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66987 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66987 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66988 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66988 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66989 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66989 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66990 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66990 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66991 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66991 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66992 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66992 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66993 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66993 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66994 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66994 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66995 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66995 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66996 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66996 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66997 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66997 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66998 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66998 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66999 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 66999 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67000 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67000 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67001 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67001 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67002 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67002 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67003 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67003 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67004 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67004 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67005 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67005 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67006 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67006 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67007 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67007 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67008 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67008 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67009 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67009 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67010 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67010 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67011 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67011 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67012 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67012 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67013 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67013 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67014 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67014 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67015 exits with status 0 2020-06-15 06:31:32: pid 66912: LOG: child process with pid: 67015 exited with success and will not be restarted 2020-06-15 06:31:32: pid 66912: LOG: worker child process with pid: 67023 exits with status 256 2020-06-15 06:31:32: pid 66912: LOG: fork a new worker child process with pid: 67272 2020-06-15 06:36:39: pid 66912: LOG: child process with pid: 67250 exits with status 256 2020-06-15 06:36:39: pid 66912: LOG: fork a new child process with pid: 67634 2020-06-15 06:57:45: pid 81424: LOG: stop request sent to pgpool. waiting for termination... .done. 2020-06-15 06:57:47: pid 81429: LOG: Backend status file /var/log/edb/pgpool3.6/pgpool_status discarded 2020-06-15 06:57:47: pid 81429: LOG: Setting up socket for 0.0.0.0:9999 2020-06-15 06:57:47: pid 81429: LOG: Setting up socket for :::9999 2020-06-15 06:57:47: pid 81429: LOG: pgpool-II successfully started. version 3.6.20 (subaruboshi) 2020-06-15 06:58:55: pid 81534: LOG: stop request sent to pgpool. waiting for termination... .done. 2020-06-15 06:58:57: pid 81539: LOG: Backend status file /var/log/edb/pgpool3.6/pgpool_status discarded 2020-06-15 06:58:57: pid 81539: LOG: Setting up socket for 0.0.0.0:9999 2020-06-15 06:58:57: pid 81539: LOG: Setting up socket for :::9999 2020-06-15 06:58:57: pid 81539: LOG: pgpool-II successfully started. version 3.6.20 (subaruboshi) 2020-06-15 06:59:22: pid 81571: LOG: failed to connect to PostgreSQL server on "192.168.29.129:5444", getsockopt() detected error "Connection refused" 2020-06-15 06:59:22: pid 81571: LOG: received degenerate backend request for node_id: 0 from pid [81571] 2020-06-15 06:59:22: pid 81571: FATAL: failed to create a backend connection 2020-06-15 06:59:22: pid 81571: DETAIL: executing failover on backend 2020-06-15 06:59:22: pid 81539: LOG: Pgpool-II parent process has received failover request 2020-06-15 06:59:22: pid 81539: LOG: starting degeneration. shutdown host 192.168.29.129(5444) 2020-06-15 06:59:22: pid 81539: LOG: Restart all children 2020-06-15 06:59:22: pid 81539: LOG: failover: set new primary node: -1 2020-06-15 06:59:22: pid 81539: LOG: failover: set new master node: 1 failover done. shutdown host 192.168.29.129(5444)2020-06-15 06:59:22: pid 81539: LOG: failover done. shutdown host 192.168.29.129(5444) 2020-06-15 06:59:22: pid 81574: LOG: worker process received restart request 2020-06-15 06:59:23: pid 81573: LOG: restart request received in pcp child process 2020-06-15 06:59:23: pid 81539: LOG: PCP child 81573 exits with status 0 in failover() 2020-06-15 06:59:23: pid 81539: LOG: fork a new PCP child pid 81661 in failover() 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81541 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81541 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81542 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81542 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81543 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81543 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81544 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81544 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81545 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81545 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81546 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81546 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81547 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81547 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81548 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81548 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81549 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81549 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81550 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81550 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81551 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81551 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81552 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81552 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81553 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81553 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81554 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81554 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81555 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81555 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81556 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81556 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81557 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81557 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81558 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81558 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81559 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81559 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81560 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81560 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81561 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81561 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81562 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81562 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81563 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81563 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81564 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81564 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81565 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81565 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81566 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81566 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81567 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81567 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81568 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81568 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81569 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81569 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81570 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81570 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81571 exits with status 256 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81571 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81572 exits with status 0 2020-06-15 06:59:23: pid 81539: LOG: child process with pid: 81572 exited with success and will not be restarted 2020-06-15 06:59:23: pid 81539: LOG: worker child process with pid: 81574 exits with status 256 2020-06-15 06:59:23: pid 81539: LOG: fork a new worker child process with pid: 81662 2020-06-15 06:59:25: pid 81661: LOG: forked new pcp worker, pid=81732 socket=7 2020-06-15 06:59:25: pid 81661: LOG: PCP process with pid: 81732 exit with SUCCESS. 2020-06-15 06:59:25: pid 81661: LOG: PCP process with pid: 81732 exits with status 0 2020-06-15 06:59:25: pid 81661: LOG: forked new pcp worker, pid=81738 socket=7 2020-06-15 06:59:25: pid 81661: LOG: PCP process with pid: 81738 exit with SUCCESS. 2020-06-15 06:59:25: pid 81661: LOG: PCP process with pid: 81738 exits with status 0 2020-06-15 06:59:25: pid 81661: LOG: forked new pcp worker, pid=81740 socket=7 2020-06-15 06:59:25: pid 81740: ERROR: invalid degenerate backend request, node id : 0 status: [3] is not valid for failover 2020-06-15 06:59:25: pid 81661: LOG: PCP process with pid: 81740 exit with SUCCESS. 2020-06-15 06:59:25: pid 81661: LOG: PCP process with pid: 81740 exits with status 0 2020-06-15 06:59:29: pid 81661: LOG: forked new pcp worker, pid=81746 socket=7 2020-06-15 06:59:29: pid 81661: LOG: PCP process with pid: 81746 exit with SUCCESS. 2020-06-15 06:59:29: pid 81661: LOG: PCP process with pid: 81746 exits with status 0 2020-06-15 06:59:29: pid 81661: LOG: forked new pcp worker, pid=81747 socket=7 2020-06-15 06:59:29: pid 81661: LOG: PCP process with pid: 81747 exit with SUCCESS. 2020-06-15 06:59:29: pid 81661: LOG: PCP process with pid: 81747 exits with status 0 2020-06-15 06:59:29: pid 81661: LOG: forked new pcp worker, pid=81748 socket=7 2020-06-15 06:59:29: pid 81661: LOG: PCP process with pid: 81748 exit with SUCCESS. 2020-06-15 06:59:29: pid 81661: LOG: PCP process with pid: 81748 exits with status 0 2020-06-15 06:59:29: pid 81661: LOG: forked new pcp worker, pid=81749 socket=7 2020-06-15 06:59:29: pid 81749: FATAL: invalid pgpool mode for process recovery request 2020-06-15 06:59:29: pid 81749: DETAIL: not in streaming replication mode, can't promote node id 1 2020-06-15 06:59:29: pid 81661: LOG: PCP process with pid: 81749 exit with SUCCESS. 2020-06-15 06:59:29: pid 81661: LOG: PCP process with pid: 81749 exits with status 256 2020-06-15 06:59:29: pid 81661: LOG: forked new pcp worker, pid=81750 socket=7 2020-06-15 06:59:29: pid 81661: LOG: PCP process with pid: 81750 exit with SUCCESS. 2020-06-15 06:59:29: pid 81661: LOG: PCP process with pid: 81750 exits with status 0 2020-06-15 06:59:29: pid 81661: LOG: forked new pcp worker, pid=81751 socket=7 2020-06-15 06:59:29: pid 81661: LOG: PCP process with pid: 81751 exit with SUCCESS. 2020-06-15 06:59:29: pid 81661: LOG: PCP process with pid: 81751 exits with status 0 2020-06-15 06:59:29: pid 81661: LOG: forked new pcp worker, pid=81752 socket=7 2020-06-15 06:59:29: pid 81661: LOG: PCP process with pid: 81752 exit with SUCCESS. 2020-06-15 06:59:29: pid 81661: LOG: PCP process with pid: 81752 exits with status 0 2020-06-15 06:59:29: pid 81661: LOG: forked new pcp worker, pid=81753 socket=7 2020-06-15 06:59:29: pid 81661: LOG: PCP process with pid: 81753 exit with SUCCESS. 2020-06-15 06:59:29: pid 81661: LOG: PCP process with pid: 81753 exits with status 0 2020-06-15 07:01:41: pid 81661: LOG: forked new pcp worker, pid=82181 socket=7 2020-06-15 07:01:41: pid 81661: LOG: PCP process with pid: 82181 exit with SUCCESS. 2020-06-15 07:01:41: pid 81661: LOG: PCP process with pid: 82181 exits with status 0 2020-06-15 07:01:41: pid 81661: LOG: forked new pcp worker, pid=82186 socket=7 2020-06-15 07:01:41: pid 81661: LOG: PCP process with pid: 82186 exit with SUCCESS. 2020-06-15 07:01:41: pid 81661: LOG: PCP process with pid: 82186 exits with status 0 2020-06-15 07:01:41: pid 81661: LOG: forked new pcp worker, pid=82200 socket=7 2020-06-15 07:01:41: pid 81661: LOG: PCP process with pid: 82200 exit with SUCCESS. 2020-06-15 07:01:41: pid 81661: LOG: PCP process with pid: 82200 exits with status 0 2020-06-15 07:01:41: pid 81661: LOG: forked new pcp worker, pid=82203 socket=7 2020-06-15 07:01:41: pid 82203: LOG: received failback request for node_id: 0 from pid [82203] wd_failover_id [0] 2020-06-15 07:01:41: pid 81539: LOG: Pgpool-II parent process has received failover request 2020-06-15 07:01:41: pid 81539: LOG: starting fail back. reconnect host 192.168.29.129(5444) 2020-06-15 07:01:41: pid 81539: LOG: Node 1 is not down (status: 2) 2020-06-15 07:01:41: pid 81539: LOG: Restart all children 2020-06-15 07:01:41: pid 81539: LOG: failover: set new primary node: -1 2020-06-15 07:01:41: pid 81539: LOG: failover: set new master node: 0 2020-06-15 07:01:41: pid 81661: LOG: PCP process with pid: 82203 exit with SUCCESS. 2020-06-15 07:01:41: pid 81661: LOG: PCP process with pid: 82203 exits with status 0 2020-06-15 07:01:41: pid 81661: LOG: forked new pcp worker, pid=82232 socket=7 2020-06-15 07:01:41: pid 81539: LOG: failback done. reconnect host 192.168.29.129(5444) 2020-06-15 07:01:41: pid 81662: LOG: worker process received restart request 2020-06-15 07:01:41: pid 81661: LOG: PCP process with pid: 82232 exit with SUCCESS. 2020-06-15 07:01:41: pid 81661: LOG: PCP process with pid: 82232 exits with status 0 2020-06-15 07:01:42: pid 81661: LOG: restart request received in pcp child process 2020-06-15 07:01:42: pid 81539: LOG: PCP child 81661 exits with status 0 in failover() 2020-06-15 07:01:42: pid 81539: LOG: fork a new PCP child pid 82268 in failover() 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81626 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81626 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81627 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81627 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81628 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81628 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81629 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81629 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81630 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81630 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81631 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81631 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81632 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81632 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81633 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81633 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81634 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81634 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81635 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81635 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81636 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81636 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81637 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81637 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81638 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81638 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81639 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81639 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81640 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81640 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81641 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81641 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81642 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81642 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81643 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81643 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81644 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81644 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81645 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81645 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81646 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81646 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81647 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81647 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81648 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81648 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81649 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81649 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81650 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81650 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81651 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81651 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81652 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81652 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81653 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81653 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81654 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81654 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81655 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81655 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81656 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81656 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81657 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: child process with pid: 81657 exited with success and will not be restarted 2020-06-15 07:01:42: pid 81539: LOG: worker child process with pid: 81662 exits with status 256 2020-06-15 07:01:42: pid 81539: LOG: fork a new worker child process with pid: 82269 2020-06-15 07:06:47: pid 81539: LOG: child process with pid: 82236 exits with status 256 2020-06-15 07:06:47: pid 81539: LOG: fork a new child process with pid: 82479
Tags	failover, load balancing, pgpool in load balancing mode, pgpool master failover identification slave changeover, pgpool-II

sajith.t.s 2020-06-15 18:03 reporter	pool_error.JPG (61,703 bytes) pool_error.JPG (61,703 bytes)

pengbo 2020-06-16 14:38 developer ~0003392	Could you share your pgpool.conf?

sajith.t.s 2020-06-16 14:51 reporter ~0003393	@pengbo I'm attaching pgpool.conf from my node. pgpool.conf (36,292 bytes)

pengbo 2020-06-21 17:49 developer ~0003405	The role "master/slave" in the result of "show pool_nodes" does not represent "Primary/Standby". In streaming replication mode, the role should be displayed as "Primary/Standby". I found you didn't configure streaming replication mode correctly. -------------- master_slave_sub_mode = 'slony' -------------- If you are using streaming replication, you need to set: -------------- master_slave_sub_mode = 'stream' -------------- I am not familiar with efm, but I think if you don't need Pgpool-II's automatic failover capability, Pgpool-II can't detect new primary node even if the standby node is promoted to new primary by efm. Only when a client connects to pgpool and an error occurs, then pgpool will start to look for the new primary. And when the old master is restored, pgpool can't detect the node automatically, you need to run "pcp_attact_node" manually.

sajith.t.s 2020-06-23 16:14 reporter ~0003419	@pengbo Issue solved. Configuration was incorrect as you say. master_slave_sub_mode = 'slony' instead of master_slave_sub_mode = 'stream'. Kindly close the issue. Thank you.

pengbo 2020-06-24 17:46 developer ~0003423	I am going to close this issue.

Date Modified	Username	Field	Change
2020-06-15 18:03	sajith.t.s	New Issue
2020-06-15 18:03	sajith.t.s	File Added: pool_error.JPG
2020-06-15 18:03	sajith.t.s	Tag Attached: failover
2020-06-15 18:03	sajith.t.s	Tag Attached: load balancing
2020-06-15 18:03	sajith.t.s	Tag Attached: pgpool in load balancing mode
2020-06-15 18:03	sajith.t.s	Tag Attached: pgpool master failover identification slave changeover
2020-06-15 18:03	sajith.t.s	Tag Attached: pgpool-II
2020-06-16 14:07	pengbo	Assigned To	=> pengbo
2020-06-16 14:07	pengbo	Status	new => assigned
2020-06-16 14:07	pengbo	Description Updated
2020-06-16 14:07	pengbo	Additional Information Updated
2020-06-16 14:38	pengbo	Note Added: 0003392
2020-06-16 14:51	sajith.t.s	File Added: pgpool.conf
2020-06-16 14:51	sajith.t.s	Note Added: 0003393
2020-06-21 17:49	pengbo	Note Added: 0003405
2020-06-21 17:50	pengbo	Status	assigned => feedback
2020-06-23 16:14	sajith.t.s	Note Added: 0003419
2020-06-23 16:14	sajith.t.s	Status	feedback => assigned
2020-06-24 17:46	pengbo	Note Added: 0003423
2020-06-24 17:46	pengbo	Status	assigned => closed

View Issue Details

Activities

Issue History