View Issue Details

IDProjectCategoryView StatusLast Update
0000342Pgpool-IIBugpublic2017-11-09 16:56
ReporterreanAssigned ToMuhammad Usama 
PriorityimmediateSeveritymajorReproducibilityhave not tried
Status feedbackResolutionopen 
Product Version3.6.6 
Target VersionFixed in Version 
Summary0000342: On master fail slave not able to convert itself to master
DescriptionUsing debian based OS

Master 10.100.108.10 (Installed postgresql 9.6 and pgpool II 3.6) in read write mode
Slave 10.100.108.9 (Installed postgresql 9.6 and pgpool II 3.6) in read only mode
Virtual IP 10.100.108.150

Postgresql own replication procedure is used

On shutting down of master machine/ stopping the postgresql in master, the slave should convert itself to master, that is from read mode to readwrite mode. In my case it is not happening. What may be the error ?
Steps To Reproduce1. Shutting down of master machine/stopping the postgresql in master.
2. Virtual ip should shift to slave.
3. Slave should convert to master from read to read-write mode.
Additional InformationSep 25 17:02:45 Machine1 crmd[2127]: notice: State transition S_IDLE -> S_POLICY_ENGINE
Sep 25 17:02:45 Machine1 pengine[2126]: warning: Fencing and resource management disabled due to lack of quorum
Sep 25 17:02:45 Machine1 pengine[2126]: error: Resource start-up disabled since no STONITH resources have been defined
Sep 25 17:02:45 Machine1 pengine[2126]: error: Either configure some or disable STONITH with the stonith-enabled option
Sep 25 17:02:45 Machine1 pengine[2126]: error: NOTE: Clusters with shared data need STONITH to ensure data integrity
Sep 25 17:02:45 Machine1 pengine[2126]: notice: Delaying fencing operations until there are resources to manage
Sep 25 17:02:45 Machine1 pengine[2126]: notice: Calculated transition 391, saving inputs in /var/lib/pacemaker/pengine/pe-input-2.bz2
Sep 25 17:02:45 Machine1 pengine[2126]: notice: Configuration ERRORs found during PE processing. Please run "crm_verify -L" to identify issues.
Sep 25 17:02:45 Machine1 crmd[2127]: notice: Transition 391 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-2.bz2): Complete
Sep 25 17:02:45 Machine1 crmd[2127]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE
Sep 25 17:09:01 Machine1 CRON[17142]: (root) CMD ( [ -x /usr/lib/php5/sessionclean ] && /usr/lib/php5/sessionclean)
Sep 25 17:12:19 Machine1 pgpool[17184]: [1-1] 2017-09-25 17:12:19: pid 17184: LOG: stop request sent to pgpool. waiting for termination...
Sep 25 17:12:25 Machine1 pgpool[17222]: [1-1] 2017-09-25 17:12:25: pid 17222: WARNING: checking setuid bit of if_up_cmd
Sep 25 17:12:25 Machine1 pgpool[17222]: [1-2] 2017-09-25 17:12:25: pid 17222: DETAIL: ifup[/bin/ip_w] doesn't have setuid bit
Sep 25 17:12:25 Machine1 pgpool[17222]: [2-1] 2017-09-25 17:12:25: pid 17222: WARNING: checking setuid bit of if_down_cmd
Sep 25 17:12:25 Machine1 pgpool[17222]: [2-2] 2017-09-25 17:12:25: pid 17222: DETAIL: ifdown[/bin/ip_w] doesn't have setuid bit
Sep 25 17:12:25 Machine1 pgpool[17222]: [3-1] 2017-09-25 17:12:25: pid 17222: LOG: reading status file: 0 th backend is set to down status
Sep 25 17:12:25 Machine1 pgpool[17222]: [4-1] 2017-09-25 17:12:25: pid 17222: LOG: waiting for watchdog to initialize
Sep 25 17:12:25 Machine1 pgpool[17232]: [4-1] 2017-09-25 17:12:25: pid 17232: LOG: setting the local watchdog node name to "Machine1:5432 Linux Machine1"
Sep 25 17:12:25 Machine1 pgpool[17232]: [5-1] 2017-09-25 17:12:25: pid 17232: LOG: watchdog cluster is configured with 1 remote nodes
Sep 25 17:12:25 Machine1 pgpool[17232]: [6-1] 2017-09-25 17:12:25: pid 17232: LOG: watchdog remote node:0 on Machine3:9000
Sep 25 17:12:25 Machine1 pgpool[17232]: [7-1] 2017-09-25 17:12:25: pid 17232: LOG: interface monitoring is disabled in watchdog
Sep 25 17:12:25 Machine1 pgpool[17232]: [8-1] 2017-09-25 17:12:25: pid 17232: LOG: connect on socket failed
Sep 25 17:12:25 Machine1 pgpool[17232]: [8-2] 2017-09-25 17:12:25: pid 17232: DETAIL: connect failed with error: "Network is unreachable"
Sep 25 17:12:25 Machine1 pgpool[17232]: [9-1] 2017-09-25 17:12:25: pid 17232: LOG: watchdog node state changed from [DEAD] to [LOADING]
Sep 25 17:12:30 Machine1 pgpool[17232]: [10-1] 2017-09-25 17:12:30: pid 17232: LOG: watchdog node state changed from [LOADING] to [JOINING]
Sep 25 17:12:30 Machine1 pgpool[17232]: [11-1] 2017-09-25 17:12:30: pid 17232: LOG: connect on socket failed
Sep 25 17:12:30 Machine1 pgpool[17232]: [11-2] 2017-09-25 17:12:30: pid 17232: DETAIL: connect failed with error: "Network is unreachable"
Sep 25 17:12:34 Machine1 pgpool[17232]: [12-1] 2017-09-25 17:12:34: pid 17232: LOG: watchdog node state changed from [JOINING] to [INITIALIZING]
Sep 25 17:12:35 Machine1 pgpool[17232]: [13-1] 2017-09-25 17:12:35: pid 17232: LOG: I am the only alive node in the watchdog cluster
Sep 25 17:12:35 Machine1 pgpool[17232]: [13-2] 2017-09-25 17:12:35: pid 17232: HINT: skiping stand for coordinator state
Sep 25 17:12:35 Machine1 pgpool[17232]: [14-1] 2017-09-25 17:12:35: pid 17232: LOG: watchdog node state changed from [INITIALIZING] to [MASTER]
Sep 25 17:12:35 Machine1 pgpool[17232]: [15-1] 2017-09-25 17:12:35: pid 17232: LOG: I am announcing my self as master/coordinator watchdog node
Sep 25 17:12:39 Machine1 pgpool[17232]: [16-1] 2017-09-25 17:12:39: pid 17232: LOG: I am the cluster leader node
Sep 25 17:12:39 Machine1 pgpool[17232]: [16-2] 2017-09-25 17:12:39: pid 17232: DETAIL: our declare coordinator message is accepted by all nodes
Sep 25 17:12:39 Machine1 pgpool[17232]: [17-1] 2017-09-25 17:12:39: pid 17232: LOG: I am the cluster leader node. Starting escalation process
Sep 25 17:12:39 Machine1 pgpool[17222]: [5-1] 2017-09-25 17:12:39: pid 17222: LOG: watchdog process is initialized
Sep 25 17:12:39 Machine1 pgpool[17232]: [18-1] 2017-09-25 17:12:39: pid 17232: LOG: escalation process started with PID:17265
Sep 25 17:12:39 Machine1 pgpool[17232]: [19-1] 2017-09-25 17:12:39: pid 17232: LOG: new IPC connection received
Sep 25 17:12:39 Machine1 pgpool[17232]: [20-1] 2017-09-25 17:12:39: pid 17232: LOG: received the get data request from local pgpool-II on IPC interface
Sep 25 17:12:39 Machine1 pgpool[17222]: [6-1] 2017-09-25 17:12:39: pid 17222: LOG: I am the master watchdog node
Sep 25 17:12:39 Machine1 pgpool[17222]: [6-2] 2017-09-25 17:12:39: pid 17222: DETAIL: using the local backend node status
Sep 25 17:12:39 Machine1 pgpool[17222]: [7-1] 2017-09-25 17:12:39: pid 17222: LOG: Setting up socket for 0.0.0.0:5432
Sep 25 17:12:39 Machine1 pgpool[17222]: [8-1] 2017-09-25 17:12:39: pid 17222: LOG: Setting up socket for :::5432
Sep 25 17:12:39 Machine1 pgpool[17232]: [21-1] 2017-09-25 17:12:39: pid 17232: LOG: new IPC connection received
Sep 25 17:12:39 Machine1 pgpool[17264]: [6-1] 2017-09-25 17:12:39: pid 17264: LOG: 2 watchdog nodes are configured for lifecheck
Sep 25 17:12:39 Machine1 pgpool[17265]: [18-1] 2017-09-25 17:12:39: pid 17265: LOG: watchdog: escalation started
Sep 25 17:12:39 Machine1 pgpool[17264]: [7-1] 2017-09-25 17:12:39: pid 17264: LOG: watchdog nodes ID:0 Name:"Machine1:5432 Linux Machine1"
Sep 25 17:12:39 Machine1 pgpool[17264]: [7-2] 2017-09-25 17:12:39: pid 17264: DETAIL: Host:"Machine1" WD Port:9000 pgpool-II port:5432
Sep 25 17:12:39 Machine1 pgpool[17264]: [8-1] 2017-09-25 17:12:39: pid 17264: LOG: watchdog nodes ID:1 Name:"Not_Set"
Sep 25 17:12:39 Machine1 pgpool[17264]: [8-2] 2017-09-25 17:12:39: pid 17264: DETAIL: Host:"Machine3" WD Port:9000 pgpool-II port:5432
Sep 25 17:12:39 Machine1 pgpool[17222]: [9-1] 2017-09-25 17:12:39: pid 17222: LOG: pgpool-II successfully started. version 3.6.1 (subaruboshi)
Sep 25 17:12:39 Machine1 pgpool[17222]: [10-1] 2017-09-25 17:12:39: pid 17222: LOG: find_primary_node: checking backend no 0
Sep 25 17:12:39 Machine1 pgpool[17222]: [11-1] 2017-09-25 17:12:39: pid 17222: LOG: find_primary_node: checking backend no 1
Sep 25 17:12:40 Machine1 pgpool[17271]: [9-1] 2017-09-25 17:12:40: pid 17271: LOG: set SO_REUSEPORT option to the socket
Sep 25 17:12:40 Machine1 pgpool[17272]: [9-1] 2017-09-25 17:12:40: pid 17272: LOG: set SO_REUSEPORT option to the socket
Sep 25 17:12:40 Machine1 pgpool[17272]: [10-1] 2017-09-25 17:12:40: pid 17272: LOG: creating socket for sending heartbeat
Sep 25 17:12:40 Machine1 pgpool[17272]: [10-2] 2017-09-25 17:12:40: pid 17272: DETAIL: set SO_REUSEPORT
Sep 25 17:12:40 Machine1 pgpool[17271]: [10-1] 2017-09-25 17:12:40: pid 17271: LOG: creating watchdog heartbeat receive socket.
Sep 25 17:12:40 Machine1 pgpool[17271]: [10-2] 2017-09-25 17:12:40: pid 17271: DETAIL: set SO_REUSEPORT
Sep 25 17:12:42 Machine1 pgpool[17304]: [9-1] 2017-09-25 17:12:42: pid 17304: LOG: failed to connect to PostgreSQL server on "Machine1:5433", getsockopt() detected error "No route to host"
Sep 25 17:12:42 Machine1 pgpool[17304]: [10-1] 2017-09-25 17:12:42: pid 17304: ERROR: failed to make persistent db connection
Sep 25 17:12:42 Machine1 pgpool[17304]: [10-2] 2017-09-25 17:12:42: pid 17304: DETAIL: connection to host:"Machine1:5433" failed
Sep 25 17:12:42 Machine1 pgpool[17222]: [12-1] 2017-09-25 17:12:42: pid 17222: LOG: failed to connect to PostgreSQL server on "Machine1:5433", getsockopt() detected error "No route to host"
Sep 25 17:12:42 Machine1 pgpool[17222]: [13-1] 2017-09-25 17:12:42: pid 17222: ERROR: failed to make persistent db connection
Sep 25 17:12:42 Machine1 pgpool[17222]: [13-2] 2017-09-25 17:12:42: pid 17222: DETAIL: connection to host:"Machine1:5433" failed
Sep 25 17:12:43 Machine1 pgpool[17265]: [19-1] 2017-09-25 17:12:43: pid 17265: LOG: watchdog bringing up delegate IP, 'if_up_cmd' succeeded
Sep 25 17:12:43 Machine1 pgpool[17232]: [22-1] 2017-09-25 17:12:43: pid 17232: LOG: watchdog escalation process with pid: 17265 exit with SUCCESS.
Sep 25 17:12:45 Machine1 pgpool[17222]: [14-1] 2017-09-25 17:12:45: pid 17222: LOG: failed to connect to PostgreSQL server on "Machine1:5433", getsockopt() detected error "No route to host"
Sep 25 17:12:45 Machine1 pgpool[17222]: [15-1] 2017-09-25 17:12:45: pid 17222: ERROR: failed to make persistent db connection
Sep 25 17:12:45 Machine1 pgpool[17222]: [15-2] 2017-09-25 17:12:45: pid 17222: DETAIL: connection to host:"Machine1:5433" failed
Sep 25 17:12:48 Machine1 pgpool[17304]: [11-1] 2017-09-25 17:12:48: pid 17304: LOG: failed to connect to PostgreSQL server on "Machine1:5433", getsockopt() detected error "No route to host"
Sep 25 17:12:48 Machine1 pgpool[17304]: [12-1] 2017-09-25 17:12:48: pid 17304: ERROR: failed to make persistent db connection
Sep 25 17:12:48 Machine1 pgpool[17304]: [12-2] 2017-09-25 17:12:48: pid 17304: DETAIL: connection to host:"Machine1:5433" failed
Sep 25 17:12:48 Machine1 pgpool[17222]: [16-1] 2017-09-25 17:12:48: pid 17222: LOG: failed to connect to PostgreSQL server on "Machine1:5433", getsockopt() detected error "No route to host"
Sep 25 17:12:48 Machine1 pgpool[17222]: [17-1] 2017-09-25 17:12:48: pid 17222: ERROR: failed to make persistent db connection
Sep 25 17:12:48 Machine1 pgpool[17222]: [17-2] 2017-09-25 17:12:48: pid 17222: DETAIL: connection to host:"Machine1:5433" failed
Sep 25 17:12:48 Machine1 pgpool[17222]: [18-1] 2017-09-25 17:12:48: pid 17222: LOG: setting backend node 1 status to NODE DOWN
Sep 25 17:12:48 Machine1 pgpool[17222]: [19-1] 2017-09-25 17:12:48: pid 17222: LOG: received degenerate backend request for node_id: 1 from pid [17222]
Sep 25 17:12:48 Machine1 pgpool[17232]: [23-1] 2017-09-25 17:12:48: pid 17232: LOG: new IPC connection received
Sep 25 17:12:48 Machine1 pgpool[17232]: [24-1] 2017-09-25 17:12:48: pid 17232: LOG: watchdog received the failover command from local pgpool-II on IPC interface
Sep 25 17:12:48 Machine1 pgpool[17232]: [25-1] 2017-09-25 17:12:48: pid 17232: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface
Sep 25 17:12:48 Machine1 pgpool[17232]: [26-1] 2017-09-25 17:12:48: pid 17232: LOG: I am the only pgpool-II node in the watchdog cluster
Sep 25 17:12:48 Machine1 pgpool[17232]: [26-2] 2017-09-25 17:12:48: pid 17232: DETAIL: no need to propagate the failover command [DEGENERATE_BACKEND_REQUEST]
Sep 25 17:12:48 Machine1 pgpool[17232]: [27-1] 2017-09-25 17:12:48: pid 17232: LOG: new IPC connection received
Sep 25 17:12:48 Machine1 pgpool[17232]: [28-1] 2017-09-25 17:12:48: pid 17232: LOG: received the failover command lock request from local pgpool-II on IPC interface
Sep 25 17:12:48 Machine1 pgpool[17232]: [29-1] 2017-09-25 17:12:48: pid 17232: LOG: local pgpool-II node "Machine1:5432 Linux Machine1" is requesting to become a lock holder for failover ID: 0
Sep 25 17:12:48 Machine1 pgpool[17232]: [30-1] 2017-09-25 17:12:48: pid 17232: LOG: local pgpool-II node "Machine1:5432 Linux Machine1" is the lock holder
Sep 25 17:12:48 Machine1 pgpool[17222]: [20-1] 2017-09-25 17:12:48: pid 17222: LOG: starting degeneration. shutdown host Machine1(5433)
Sep 25 17:12:48 Machine1 pgpool[17222]: [21-1] 2017-09-25 17:12:48: pid 17222: WARNING: All the DB nodes are in down status and skip writing status file.
Sep 25 17:12:48 Machine1 pgpool[17222]: [22-1] 2017-09-25 17:12:48: pid 17222: LOG: failover: no valid backends node found
Sep 25 17:12:48 Machine1 pgpool[17222]: [23-1] 2017-09-25 17:12:48: pid 17222: LOG: Restart all children
Sep 25 17:12:48 Machine1 pgpool[17266]: [9-1] 2017-09-25 17:12:48: pid 17266: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17267]: [9-1] 2017-09-25 17:12:48: pid 17267: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17268]: [9-1] 2017-09-25 17:12:48: pid 17268: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17269]: [9-1] 2017-09-25 17:12:48: pid 17269: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17273]: [9-1] 2017-09-25 17:12:48: pid 17273: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17274]: [9-1] 2017-09-25 17:12:48: pid 17274: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17275]: [9-1] 2017-09-25 17:12:48: pid 17275: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17276]: [9-1] 2017-09-25 17:12:48: pid 17276: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17277]: [9-1] 2017-09-25 17:12:48: pid 17277: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17279]: [9-1] 2017-09-25 17:12:48: pid 17279: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17280]: [9-1] 2017-09-25 17:12:48: pid 17280: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17281]: [9-1] 2017-09-25 17:12:48: pid 17281: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17282]: [9-1] 2017-09-25 17:12:48: pid 17282: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17283]: [9-1] 2017-09-25 17:12:48: pid 17283: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17284]: [9-1] 2017-09-25 17:12:48: pid 17284: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17285]: [9-1] 2017-09-25 17:12:48: pid 17285: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17286]: [9-1] 2017-09-25 17:12:48: pid 17286: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17287]: [9-1] 2017-09-25 17:12:48: pid 17287: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17288]: [9-1] 2017-09-25 17:12:48: pid 17288: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17289]: [9-1] 2017-09-25 17:12:48: pid 17289: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17290]: [9-1] 2017-09-25 17:12:48: pid 17290: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17291]: [9-1] 2017-09-25 17:12:48: pid 17291: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17292]: [9-1] 2017-09-25 17:12:48: pid 17292: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17293]: [9-1] 2017-09-25 17:12:48: pid 17293: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17294]: [9-1] 2017-09-25 17:12:48: pid 17294: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17295]: [9-1] 2017-09-25 17:12:48: pid 17295: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17296]: [9-1] 2017-09-25 17:12:48: pid 17296: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17299]: [9-1] 2017-09-25 17:12:48: pid 17299: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17300]: [9-1] 2017-09-25 17:12:48: pid 17300: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17301]: [9-1] 2017-09-25 17:12:48: pid 17301: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17302]: [9-1] 2017-09-25 17:12:48: pid 17302: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17222]: [24-1] 2017-09-25 17:12:48: pid 17222: LOG: execute command: /etc/pgpool2/3.6.0/failover.sh 1 1 "" reppassword /etc/postgresql/9.6/main/im_the_master
Sep 25 17:12:48 Machine1 pgpool[17298]: [9-1] 2017-09-25 17:12:48: pid 17298: LOG: child process received shutdown request signal 3
Sep 25 17:12:48 Machine1 pgpool[17232]: [31-1] 2017-09-25 17:12:48: pid 17232: LOG: new IPC connection received
Sep 25 17:12:48 Machine1 pgpool[17232]: [32-1] 2017-09-25 17:12:48: pid 17232: LOG: received the failover command lock request from local pgpool-II on IPC interface
Sep 25 17:12:48 Machine1 pgpool[17232]: [33-1] 2017-09-25 17:12:48: pid 17232: LOG: local pgpool-II node "Machine1:5432 Linux Machine1" is requesting to release [FAILOVER] lock for failover ID 0
Sep 25 17:12:48 Machine1 pgpool[17232]: [34-1] 2017-09-25 17:12:48: pid 17232: LOG: local pgpool-II node "Machine1:5432 Linux Machine1" has released the [FAILOVER] lock for failover ID 0
Sep 25 17:12:48 Machine1 pgpool[17232]: [35-1] 2017-09-25 17:12:48: pid 17232: LOG: new IPC connection received
Sep 25 17:12:48 Machine1 pgpool[17232]: [36-1] 2017-09-25 17:12:48: pid 17232: LOG: received the failover command lock request from local pgpool-II on IPC interface
Sep 25 17:12:48 Machine1 pgpool[17232]: [37-1] 2017-09-25 17:12:48: pid 17232: LOG: local pgpool-II node "Machine1:5432 Linux Machine1" is requesting to release [FOLLOW MASTER] lock for failover ID 0
Sep 25 17:12:48 Machine1 pgpool[17232]: [38-1] 2017-09-25 17:12:48: pid 17232: LOG: local pgpool-II node "Machine1:5432 Linux Machine1" has released the [FOLLOW MASTER] lock for failover ID 0
Sep 25 17:12:48 Machine1 pgpool[17222]: [25-1] 2017-09-25 17:12:48: pid 17222: LOG: failover: set new primary node: -1
Sep 25 17:12:48 Machine1 pgpool[17304]: [13-1] 2017-09-25 17:12:48: pid 17304: LOG: worker process received restart request
Sep 25 17:12:48 Machine1 pgpool[17232]: [39-1] 2017-09-25 17:12:48: pid 17232: LOG: new IPC connection received
Sep 25 17:12:48 Machine1 pgpool[17232]: [40-1] 2017-09-25 17:12:48: pid 17232: LOG: received the failover command lock request from local pgpool-II on IPC interface
Sep 25 17:12:48 Machine1 pgpool[17232]: [41-1] 2017-09-25 17:12:48: pid 17232: LOG: local pgpool-II node "Machine1:5432 Linux Machine1" is requesting to resign from a lock holder for failover ID 0
Sep 25 17:12:48 Machine1 pgpool[17232]: [42-1] 2017-09-25 17:12:48: pid 17232: LOG: local pgpool-II node "Machine1:5432 Linux Machine1" has resigned from the lock holder
Sep 25 17:12:48 Machine1 pgpool[17222]: [26-1] 2017-09-25 17:12:48: pid 17222: LOG: failover done. shutdown host Machine1(5433)
Sep 25 17:12:49 Machine1 pgpool[17303]: [9-1] 2017-09-25 17:12:49: pid 17303: LOG: restart request received in pcp child process
Sep 25 17:12:49 Machine1 pgpool[17222]: [27-1] 2017-09-25 17:12:49: pid 17222: LOG: PCP child 17303 exits with status 0 in failover()
Sep 25 17:12:49 Machine1 pgpool[17222]: [28-1] 2017-09-25 17:12:49: pid 17222: LOG: fork a new PCP child pid 17346 in failover()
Sep 25 17:12:49 Machine1 pgpool[17222]: [29-1] 2017-09-25 17:12:49: pid 17222: WARNING: All the DB nodes are in down status and skip writing status file.
Sep 25 17:12:49 Machine1 pgpool[17222]: [30-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17266 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [31-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17266 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [32-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17267 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [33-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17267 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [34-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17268 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [35-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17268 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [36-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17269 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [37-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17269 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [38-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17273 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [39-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17273 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [40-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17274 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [41-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17274 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [42-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17275 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [43-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17275 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [44-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17276 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [45-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17276 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [46-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17277 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [47-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17277 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [48-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17279 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [49-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17279 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [50-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17280 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [51-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17280 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [52-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17281 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [53-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17281 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [54-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17282 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [55-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17282 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [56-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17283 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [57-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17283 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [58-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17284 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [59-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17284 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [60-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17285 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [61-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17285 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [62-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17286 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [63-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17286 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [64-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17287 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [65-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17287 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [66-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17288 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [67-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17288 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [68-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17289 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [69-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17289 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [70-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17290 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [71-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17290 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [72-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17291 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [73-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17291 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [74-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17292 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [75-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17292 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [76-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17293 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [77-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17293 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [78-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17294 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [79-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17294 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [80-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17295 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [81-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17295 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [82-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17296 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [83-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17296 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [84-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17298 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [85-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17298 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [86-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17299 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [87-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17299 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [88-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17300 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [89-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17300 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [90-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17301 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [91-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17301 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [92-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17302 exits with status 0
Sep 25 17:12:49 Machine1 pgpool[17222]: [93-1] 2017-09-25 17:12:49: pid 17222: LOG: child process with pid: 17302 exited with success and will not be restarted
Sep 25 17:12:49 Machine1 pgpool[17222]: [94-1] 2017-09-25 17:12:49: pid 17222: LOG: worker child process with pid: 17304 exits with status 256
Sep 25 17:12:49 Machine1 pgpool[17222]: [95-1] 2017-09-25 17:12:49: pid 17222: LOG: fork a new worker child process with pid: 17347
Sep 25 17:17:01 Machine1 CRON[17353]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Sep 25 17:17:45 Machine1 crmd[2127]: notice: State transition S_IDLE -> S_POLICY_ENGINE
Sep 25 17:17:46 Machine1 pengine[2126]: warning: Fencing and resource management disabled due to lack of quorum
Sep 25 17:17:46 Machine1 pengine[2126]: error: Resource start-up disabled since no STONITH resources have been defined
Sep 25 17:17:46 Machine1 pengine[2126]: error: Either configure some or disable STONITH with the stonith-enabled option
Sep 25 17:17:46 Machine1 pengine[2126]: error: NOTE: Clusters with shared data need STONITH to ensure data integrity
Sep 25 17:17:46 Machine1 pengine[2126]: notice: Delaying fencing operations until there are resources to manage
Sep 25 17:17:46 Machine1 crmd[2127]: notice: Transition 392 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-2.bz2): Complete
Sep 25 17:17:46 Machine1 crmd[2127]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE
Sep 25 17:17:46 Machine1 pengine[2126]: notice: Calculated transition 392, saving inputs in /var/lib/pacemaker/pengine/pe-input-2.bz2
Sep 25 17:17:46 Machine1 pengine[2126]: notice: Configuration ERRORs found during PE processing. Please run "crm_verify -L" to identify issues.
Tagspgpool master failover identification slave changeover

Activities

rean

2017-09-25 22:12

reporter  

pgpoolpostConfFiles.zip (4,462 bytes)

eldad

2017-10-16 23:57

reporter   ~0001757

What does your failover script do?
check the parameters it get from pgpool
from the log looks like pgpool think all the nodes are down

Sep 25 17:12:48 Machine1 pgpool[17222]: [21-1] 2017-09-25 17:12:48: pid 17222: WARNING: All the DB nodes are in down status and skip writing status file.
Sep 25 17:12:48 Machine1 pgpool[17222]: [22-1] 2017-09-25 17:12:48: pid 17222: LOG: failover: no valid backends node found

t-ishii

2017-11-09 16:56

developer   ~0001818

This maybe already fixed in 3.6.7.

Issue History

Date Modified Username Field Change
2017-09-25 22:12 rean New Issue
2017-09-25 22:12 rean File Added: pgpoolpostConfFiles.zip
2017-09-25 22:12 rean Tag Attached: pgpool master failover identification slave changeover
2017-10-16 23:57 eldad Note Added: 0001757
2017-11-09 16:56 t-ishii Note Added: 0001818
2017-11-09 16:56 t-ishii Assigned To => Muhammad Usama
2017-11-09 16:56 t-ishii Status new => feedback