View Issue Details

IDProjectCategoryView StatusLast Update
0000227Pgpool-IIBugpublic2017-08-29 09:34
Reportersupp_kAssigned ToMuhammad Usama 
PriorityimmediateSeveritymajorReproducibilityrandom
Status closedResolutionopen 
PlatformOSCentOSOS Version6 & 7
Product Version3.5.3 
Target VersionFixed in Version 
Summary0000227: failover not performed by stanby node
DescriptionIn my production environment the following case not always leads to failover procedure.

I have 2 pgpool nodes and watchdog cluster. Every pgpool node server also has PostgreSQL instance running. Sometimes we are facing network troubles and the server where the master pgpool and master postgresql are hosted becomes unavailable. In this case standby pgpool node switching into Master and it is ok. But the problem here is that it doesnt trigger failover command. I see the pgpool standby is promoted into master but it doesnt trigger his failover command.

Why? The case is reproduced randomly.
Steps To ReproduceSetup environment (2 server):
1) pgpool + postgres
2) pgpool + postgres

Make the pgpool in the server 1 master and the postgres master as well. Postgres should perform sync replication onto 2nd server;

Unplug network cable at 1st server.

Result: 2nd pgpool switched into Master but it doesn't trigger failover procedure which in my scenario should promote 2nd postgres into master.

Please consider the issue, to me it very critical.
Additional InformationConfiguration of the 1st pgpool server is attached. 2nd's server configuration is identical.
Tagsmaster slave, streaming replication, watchdog

Activities

supp_k

2016-07-30 09:19

reporter  

pgpool.conf (31,012 bytes)

guptesh.cg4@gmail.com

2016-08-01 23:42

reporter   ~0000952

I am facing this issue very often. Doing a hard power-off(master pgpool+postgres) node to simulate a crash situation instead of making the interface down. The STANDBY pgpool is escalated to master but fail over does not happen and postgres remains in replication slave mode.

My environment is
CentOS 7.0
Pgpool 3.5.3
Postgresql 9.4.7

supp_k

2016-08-02 00:09

reporter   ~0000954

I see not only me facing the issue (((. Moreover it can be considered like a but provement. To my production environment it is very critical because pgpool here becomes a significant bottleneck.

Muhammad Usama

2016-08-03 05:23

developer   ~0000959

I am looking into the issue. Meanwhile I had recently committed a fix related to the watchdog heartbeat problem, So could you also try building from the latest source code to verify if you still face the issue. And if the issue still persists, please do share the complete pgpool-II log file with debug enabled. That would help to solve the problem more quickly.

Kind regards

supp_k

2016-08-03 20:57

reporter   ~0000962

Hi,

I have verified the patch. Now it works and I think I can apply elsewhere.

One more question: when will it be possible to provide RPM package (where the issue is solved) available in pgpool's download link?

Thank you!

z0rb1n0

2016-08-05 19:32

reporter  

pgpool_no_failover_fabio.tar.bz2 (99,548 bytes)

z0rb1n0

2016-08-05 19:32

reporter   ~0000968

Sorry but this is not solved and I can reliably reproduce the issue (built from 3.5.3 tree at 2016-08-04 16:00:00 UTC).

Our setup is a 3-way watchdog handling 3 PG backends.

wd_escalation works flawlessly, however after any watchdog failover/escalation has occurred, "degenerate backend request"s have no effect whatsoever (the backend slot stays in state "2" and the failover_command is not called).

This detection issue ONLY occurs for backend TCP timeouts that happen AFTER a watchdog standby was promoted: backend failover happens regularly if the original watchdog master is still in charge or the "failed" backend can respond instantly with TCP resets/ICMP "unreachable" messages. My gut feeling tells me this could be some race condition.

Same results for both fail_over_on_backend_error and health_check_period based fail-over settings.

I attached a tarball with my configuration files and logs (pgpool_47.conf is the original master watchdog that got taken down at the beginning of the test, so there is no corresponding pgpool_47.log).

The configuration files are generated by a script and should be absolutely identical, obviously except for the list of watchdog siblings and the wd_priority.


Sequence of events in the log:

- the current watchdog master (47) is unplugged
- one of the remaining pgpools wins the election and wd_escalation happens regularly
- I kill one of the backends at the network level ("unplug" the cord)
- the message "setting backend node 1 status to NODE DOWN" appears in the logs of both remaining, however all of the configured nodes stay "up" forever and no scripts are called
- I get fired due to repeated downtime-inducing failures



TIA and best regards


F

z0rb1n0

2016-08-05 21:44

reporter   ~0000969

I forgot to mention: not sure if it's clear from the logs, but I'm terminating SSL at pgpool and re-initiating it towards the nodes (sorry, silly security requirements).

guptesh.cg4@gmail.com

2016-08-10 16:09

reporter   ~0000977

I built from the latest code. I am not facing the issue when I disable the health check. Fail over of the DB(promoting the slave postgres to master) is taking around 3 minutes. Is it possible to reduce this time further. What parameters are considered to decide the fail over trigger time when health check is disabled?

I will check the behavior with health check enabled and share the logs.

cohavisi

2016-08-19 04:03

reporter   ~0001015

Hi,
I facing this issue also (pgpool 3.5.3 with postgres 9.3.5). cluster with 2 nodes (pgpool and postgres on each node).
pgpool 3.5.3 - trunk (pgpool2-f2b5d17.tar.gz)
postgresql 9.3.5


when restarting:
1. primary postgres (with the secondary pgpool) - no postgres failover!
2. primary postgres (with primary pgpool) - pgpool failover + postgresql failover to the 2nd node. but when the faulty node (1st) boots up its pgpool became primary again (demoted the 2nd pgpool) and show the 1st postgres as primary and the 2nd postgres as secondary (even when its status was primary) - should be detached? split brain!

checked its scenario again - pgpool failover + NO postgresql failover. when the faulty node boots up its pgpool became primary again (demoted the 2nd pgpool) and show the postgres nodes as both “up” and current (old) state!


3. secondary postgres (with primary pgpool) - pgool failover! but when the faulty node boots up it pgpool become primary and demote the 2nd pgpool (after 2 min - duplicate ip for 2 min)! but ! show the secondary postgres as "up" -why?
 node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay
---------+----------+------+--------+-----------+---------+------------+-------------------+-------------------
 0 | 1.1.1.84 | 5432 | up | 0.500000 | primary | 801 | true | 0
 1 | 1.1.1.85 | 5432 | up | 0.500000 | standby | 911 | false | 0


4. secondary pgpool & secondary postgres - no failover - as expected. but faulty postgres is shown as “up” - should be detached?


Please check all the scenarios - its is a blocker regarding the pgpool HA!

z0rb1n0

2016-08-23 20:21

reporter   ~0001020

@Muhammad

Not sure if this info can help you at all in reproducing the issue but here it goes:

- I tried disabling TLS between pools and backends, to no effect (still no failover_command is run after NODE DOWN event)
- Keep in mind that our monitoring scripts/API/SNMP presentation layer issues quite a few pcp_* commands on a regular basis against the surviving pgpools, especially during a failover.


Thank you

tscheuren

2016-08-24 23:20

reporter   ~0001021

I can confirm this behavior on
Debian jessie (8.5)
Pgpool 3.5.3
Postgresql 9.5.4

Muhammad Usama

2016-08-25 23:40

developer   ~0001024

Hi Thanks for the information.

Apparently the problem is when the backend node and the pgpool-II becomes unreachable at the same instance then the failover function becomes confused.

I am looking into at priority and will update on this as soon as some I have an update.

Kind regards
Muhammad Usama

gabrimonfa

2016-09-21 00:53

reporter   ~0001073

I also can confirm this behavior on

Centos 7
Pgpool 3.5.3
Postgresql 9.5.4

This is a quite important issue, any news?

supp_k

2016-09-27 21:39

reporter   ~0001088

I have also create similar issue: http://www.pgpool.net/mantisbt/view.php?id=251

In my case failover not performed by master pgpool.

supp_k

2016-09-27 22:09

reporter   ~0001089

Guys, please update the status of the issue! When will you be able to solve it?

It is very critical for our environment!!

Muhammad Usama

2016-09-27 23:23

developer  

failover_standby_fix.diff (45,682 bytes)
diff --git a/src/include/pool.h b/src/include/pool.h
index 65b3c92..2a0cf56 100644
--- a/src/include/pool.h
+++ b/src/include/pool.h
@@ -373,7 +373,7 @@ extern int my_master_node_id;
 #define ACCEPT_FD_SEM			5
 #define MAX_REQUEST_QUEUE_SIZE	10
 
-#define MAX_SEC_WAIT_FOR_CLUSTER_TRANSATION 6 /* time in seconds to keep retrying for a
+#define MAX_SEC_WAIT_FOR_CLUSTER_TRANSATION 10 /* time in seconds to keep retrying for a
 											   * watchdog command if the cluster is not
 											   * in stable state */
 
diff --git a/src/include/watchdog/watchdog.h b/src/include/watchdog/watchdog.h
index e7c9770..fd986a4 100644
--- a/src/include/watchdog/watchdog.h
+++ b/src/include/watchdog/watchdog.h
@@ -94,6 +94,7 @@ typedef struct SocketConnection
 	struct			timeval tv;		/* connect time of socket */
 	char			addr[48];		/* ip address of socket connection*/
 	WD_SOCK_STATE	sock_state;		/* current state of socket */
+	bool			eof;
 }SocketConnection;
 
 typedef struct WatchdogNode
@@ -103,6 +104,7 @@ typedef struct WatchdogNode
 	struct timeval last_rcv_time;			/* timestamp when last packet
 											 * was received from the node
 											 */
+	struct timeval last_sent_time;
 	char nodeName[WD_MAX_HOST_NAMELEN];		/* name of this node */
 	char hostname[WD_MAX_HOST_NAMELEN];		/* host name */
 	int wd_port;							/* watchdog port */
diff --git a/src/include/watchdog/wd_ipc_commands.h b/src/include/watchdog/wd_ipc_commands.h
index d5dd148..17eadab 100644
--- a/src/include/watchdog/wd_ipc_commands.h
+++ b/src/include/watchdog/wd_ipc_commands.h
@@ -31,7 +31,7 @@
 
 typedef enum WdCommandResult
 {
-	CLUSTER_IN_TRANSATIONING,
+	CLUSTER_IN_TRANSATIONING = 0,
 	COMMAND_OK,
 	COMMAND_FAILED,
 	COMMAND_TIMEOUT
diff --git a/src/main/pgpool_main.c b/src/main/pgpool_main.c
index cd490be..39bc6dd 100644
--- a/src/main/pgpool_main.c
+++ b/src/main/pgpool_main.c
@@ -1195,7 +1195,7 @@ bool degenerate_backend_set_ex(int *node_id_set, int count, bool error, bool tes
 			 * Is proceeding to failover is the right choice ???
 			 */
 			ereport(NOTICE,
-					(errmsg("rdegenerate backend request for %d node(s) from pid [%d], But cluster is not in stable state"
+					(errmsg("degenerate backend request for %d node(s) from pid [%d], But cluster is not in stable state"
 							, node_count, getpid())));
 		}
 
diff --git a/src/utils/pool_stream.c b/src/utils/pool_stream.c
index e1336d3..9fa9db1 100644
--- a/src/utils/pool_stream.c
+++ b/src/utils/pool_stream.c
@@ -1173,9 +1173,32 @@ int socket_read(int fd, void* buf, size_t len, int timeout)
 {
 	int ret, read_len;
 	read_len = 0;
+	struct timeval timeoutval;
+	fd_set readmask;
+	int fds;
 
 	while (read_len < len)
 	{
+		FD_ZERO(&readmask);
+		FD_SET(fd, &readmask);
+
+		timeoutval.tv_sec = timeout;
+		timeoutval.tv_usec = 0;
+
+		fds = select(fd+1, &readmask, NULL, NULL, timeout?&timeoutval:NULL);
+		if (fds == -1)
+		{
+			if (errno == EAGAIN || errno == EINTR)
+				continue;
+
+			ereport(WARNING,
+					(errmsg("select failed with error: \"%s\"", strerror(errno))));
+			return -1;
+		}
+		else if (fds == 0)
+		{
+			return -2;
+		}
 		ret = read(fd, buf + read_len, (len - read_len));
 		if(ret < 0)
 		{
diff --git a/src/watchdog/watchdog.c b/src/watchdog/watchdog.c
index 791efa6..d7ccb7b 100644
--- a/src/watchdog/watchdog.c
+++ b/src/watchdog/watchdog.c
@@ -77,6 +77,9 @@ typedef enum IPC_CMD_PREOCESS_RES
 
 #define BEACON_MESSAGE_INTERVAL_SECONDS		10 /* interval between beacon messages */
 
+#define MAX_SECS_WAIT_FOR_REPLY_FROM_NODE	5
+
+
 
 
 #define WD_NO_MESSAGE						0
@@ -210,6 +213,7 @@ typedef struct WDIPCCommandData
 	char				*data_buf;
 	
 	unsigned int	sendTo_count;
+	unsigned int	sendTo_error_count;
 	unsigned int	reply_from_count;
 	unsigned int	timeout_secs;
 
@@ -260,7 +264,6 @@ typedef struct WDCommandData
 	struct timeval			commandTime;
 	unsigned int			commandSendToCount;
 	unsigned int			commandReplyFromCount;
-	int						commandFinished;
 	int						partial_sent;
 }WDCommandData;
 
@@ -316,6 +319,9 @@ static void try_connecting_with_all_unreachable_nodes(void);
 static bool connect_to_node(WatchdogNode* wdNode);
 static bool is_socket_connection_connected(SocketConnection* conn);
 
+static bool is_node_reachable(WatchdogNode* wdNode);
+static void service_unreachable_nodes(void);
+
 static int update_successful_outgoing_cons(fd_set* wmask, int pending_fds_count);
 static int prepare_fds(fd_set* rmask, fd_set* wmask, fd_set* emask);
 
@@ -335,11 +341,13 @@ static WDPacketData* read_packet(SocketConnection* conn);
 static WDPacketData* get_message_of_type(char type);
 
 static int issue_watchdog_internal_command(WatchdogNode* wdNode, WDPacketData *pkt, int timeout_sec);
-static char get_current_command_resultant_message_type(void);
 static void check_for_current_command_timeout(void);
+static bool is_cluster_command_in_progress(void);
 static bool watchdog_internal_command_packet_processor(WatchdogNode* wdNode, WDPacketData* pkt);
 static bool service_lost_connections(void);
 static void service_internal_command(void);
+static void service_ipc_commands(void);
+static void node_lost_while_ipc_command(WatchdogNode* wdNode);
 
 static unsigned int get_next_commandID(void);
 static WatchdogNode* parse_node_info_message(WDPacketData* pkt, char **authkey);
@@ -396,13 +404,14 @@ static WDIPCCommandData* get_wd_IPC_command_from_reply(WDPacketData* pkt);
 static WDIPCCommandData* get_wd_IPC_command_from_socket(int sock);
 
 static IPC_CMD_PREOCESS_RES process_IPC_command(WDIPCCommandData* ipcCommand);
-static IPC_CMD_PREOCESS_RES process_IPC_nodeStatusChange_command(WDIPCCommandData* IPCCommand);
-static IPC_CMD_PREOCESS_RES process_IPC_nodeList_command(WDIPCCommandData* IPCCommand);
-static IPC_CMD_PREOCESS_RES process_IPC_replicate_variable(WDIPCCommandData* IPCCommand);
-static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandData *IPCCommand);
-static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandData *IPCCommand);
+static IPC_CMD_PREOCESS_RES process_IPC_nodeStatusChange_command(WDIPCCommandData* ipcCommand);
+static IPC_CMD_PREOCESS_RES process_IPC_nodeList_command(WDIPCCommandData* ipcCommand);
+static IPC_CMD_PREOCESS_RES process_IPC_replicate_variable(WDIPCCommandData* ipcCommand);
+static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandData *ipcCommand);
+static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandData *ipcCommand);
 static IPC_CMD_PREOCESS_RES execute_replicate_command(WDIPCCommandData* ipcCommand);
-static bool write_ipc_command_with_result_data(WDIPCCommandData* IPCCommand, char type, char* data, int len);
+static IPC_CMD_PREOCESS_RES node_lost_during_replicate_command_processing(WatchdogNode* wdNode, WDIPCCommandData* ipcCommand);
+static bool write_ipc_command_with_result_data(WDIPCCommandData* ipcCommand, char type, char* data, int len);
 
 static int node_has_requested_for_interlocking(WatchdogNode* wdNode, WDPacketData* pkt);
 static bool node_has_resigned_from_interlocking(WatchdogNode* wdNode, WDPacketData* pkt);
@@ -437,6 +446,7 @@ static void print_received_packet_info(WDPacketData* pkt,WatchdogNode* wdNode);
 static void update_interface_status(void);
 static bool any_interface_available(void);
 static WDPacketData* process_data_request(WatchdogNode* wdNode, WDPacketData* pkt);
+static void allocate_resultNodes_in_IPCCommand(WDIPCCommandData* ipcCommand);
 
 /* global variables */
 wd_cluster g_cluster;
@@ -713,7 +723,6 @@ wd_create_recv_socket(int port)
 			(errmsg("failed to create watchdog receive socket"),
 				 errdetail("listen failed with reason: \"%s\"", strerror(saved_errno))));
 	}
-
 	return sock;
 }
 
@@ -834,8 +843,15 @@ try_connecting_with_all_unreachable_nodes(void)
 		if (wdNode->client_socket.sock_state != WD_SOCK_WAITING_FOR_CONNECT && wdNode->client_socket.sock_state != WD_SOCK_CONNECTED &&
 			wdNode->server_socket.sock_state != WD_SOCK_WAITING_FOR_CONNECT && wdNode->server_socket.sock_state != WD_SOCK_CONNECTED)
 		{
-			wdNode->state = WD_DEAD;
+			if (wdNode->state == WD_SHUTDOWN)
+				continue;
 			connect_to_node(wdNode);
+			if (wdNode->client_socket.sock_state == WD_SOCK_CONNECTED)
+			{
+				ereport(LOG,
+						(errmsg("connection to the remote node \"%s\" is restored",wdNode->nodeName)));
+				watchdog_state_machine(WD_EVENT_NEW_OUTBOUND_CONNECTION, wdNode, NULL);
+			}
 		}
 	}
 }
@@ -1026,7 +1042,6 @@ watchdog_main(void)
 			if (WD_TIME_DIFF_SEC(ref_time,g_tm_set_time) >=  g_timeout_sec)
 			{
 				timeout_event = true;
-				g_timeout_sec = 0;
 			}
 		}
 		if (select_ret > 0)
@@ -1037,21 +1052,71 @@ watchdog_main(void)
 			processed_fds += read_sockets(&rmask,(select_ret - processed_fds));
 		}
 
-		if (timeout_event)
-			watchdog_state_machine(WD_EVENT_TIMEOUT, NULL, NULL);
 		if (WD_TIME_DIFF_SEC(ref_time,g_tm_set_time) >=  1)
+		{
 			process_wd_func_commands_for_timer_events();
+		}
+
+		if (timeout_event)
+		{
+			g_timeout_sec = 0;
+			watchdog_state_machine(WD_EVENT_TIMEOUT, NULL, NULL);
+		}
 
 		check_for_current_command_timeout();
 
 		if (service_lost_connections() == true)
+		{
 			service_internal_command();
+			service_ipc_commands();
+		}
+
+		service_unreachable_nodes();
 
 		update_connected_node_count();
 	}
 	return 0;
 }
 
+/* remove the unreachable nodes from cluster */
+static void service_unreachable_nodes(void)
+{
+	int i;
+	struct timeval currTime;
+	gettimeofday(&currTime,NULL);
+
+	for (i = 0; i< g_cluster.remoteNodeCount; i++)
+	{
+		WatchdogNode* wdNode = &(g_cluster.remoteNodes[i]);
+		if (wdNode->state == WD_DEAD || wdNode->state == WD_LOST || wdNode->state == WD_SHUTDOWN)
+			continue;
+
+		if (is_node_reachable(wdNode) || wdNode->client_socket.sock_state == WD_SOCK_WAITING_FOR_CONNECT)
+		{
+			/* check if we are waiting for reply from this node */
+			if (wdNode->last_sent_time.tv_sec > 0)
+			{
+				if (WD_TIME_DIFF_SEC(currTime,wdNode->last_sent_time) >=  MAX_SECS_WAIT_FOR_REPLY_FROM_NODE)
+				{
+					ereport(LOG,
+						(errmsg("remote node \"%s\" is not replying..",wdNode->nodeName),
+							 errdetail("marking the node as lost")));
+					/* mark the node as lost */
+					watchdog_state_machine(WD_EVENT_REMOTE_NODE_LOST, wdNode, NULL);
+				}
+			}
+		}
+		else
+		{
+			ereport(LOG,
+				(errmsg("remote node \"%s\" is not reachable",wdNode->nodeName),
+					 errdetail("marking the node as lost")));
+			watchdog_state_machine(WD_EVENT_REMOTE_NODE_LOST, wdNode, NULL);
+		}
+	}
+}
+
+
 static int
 wd_create_command_server_socket(void)
 {
@@ -1233,8 +1298,17 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 				if (pkt)
 				{
 					watchdog_state_machine(WD_EVENT_PACKET_RCV, wdNode, pkt);
+					/* since a packet is received reset last sent time */
+					wdNode->last_sent_time.tv_sec = 0;
+					wdNode->last_sent_time.tv_usec = 0;
 					free_packet(pkt);
 				}
+				else
+				{
+					ereport(LOG,
+							(errmsg("client socket of %s is closed", wdNode->nodeName)));
+				}
+
 				count++;
 				if (count >= pending_fds_count)
 					return count;
@@ -1250,9 +1324,16 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 				if (pkt)
 				{
 					watchdog_state_machine(WD_EVENT_PACKET_RCV, wdNode, pkt);
+					/* since a packet is received reset last sent time */
+					wdNode->last_sent_time.tv_sec = 0;
+					wdNode->last_sent_time.tv_usec = 0;
 					free_packet(pkt);
 				}
-				
+				else
+				{
+					ereport(LOG,
+							(errmsg("server socket of %s is closed", wdNode->nodeName)));
+				}
 				count++;
 				if (count >= pending_fds_count)
 					return count;
@@ -1283,7 +1364,7 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 					print_watchdog_node_info(tempNode);
 					authenticated = verify_authhash_for_node(tempNode, authkey);
 					ereport(DEBUG1,
-							(errmsg("ADD NODE MESSAGE from Hostname:\"%s\" PORT:%d pgpool_port:%d",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port)));
+							(errmsg("ADD NODE MESSAGE from hostname:\"%s\" PORT:%d pgpool_port:%d",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port)));
 					/* verify this node */
 					if (authenticated)
 					{
@@ -1311,7 +1392,7 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 						{
 							/* reply with node info message */
 							ereport(NOTICE,
-									(errmsg("New node joined the cluster Hostname:\"%s\" PORT:%d pgpool_port:%d",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port)));
+									(errmsg("New node joined the cluster hostname:\"%s\" port:%d pgpool_port:%d",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port)));
 
 							watchdog_state_machine(WD_EVENT_PACKET_RCV, wdNode, pkt);
 						}
@@ -1324,7 +1405,7 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 					else
 					{
 						ereport(NOTICE,
-								(errmsg("authentication failed for add node from Hostname:\"%s\" PORT:%d pgpool_port:%d",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port),
+								(errmsg("authentication failed for add node from hostname:\"%s\" port:%d pgpool_port:%d",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port),
 								 errdetail("make sure wd_authkey configuration is same on all nodes")));
 					}
 
@@ -1398,10 +1479,10 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 	{
 		g_cluster.ipc_command_socks = list_delete_int(g_cluster.ipc_command_socks,lfirst_int(lc));
 	}
-	
+
 	list_free(socks_to_del);
 	socks_to_del = NULL;
-	
+
 	if (count >= pending_fds_count)
 		return count;
 	
@@ -1431,7 +1512,6 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 	list_free(socks_to_del);
 	socks_to_del = NULL;
 
-
 	/* Finally check if something waits us on interface monitoring socket */
 	if (g_cluster.network_monitor_sock > 0 &&  FD_ISSET(g_cluster.network_monitor_sock, rmask))
 	{
@@ -1440,7 +1520,7 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 		if (read_interface_change_event(g_cluster.network_monitor_sock, &link_event, &deleted))
 		{
 			ereport(DEBUG1,
-					(errmsg("Network event received"),
+					(errmsg("network event received"),
 						errdetail("deleted = %s Link change event = %s",
 							deleted?"YES":"NO",
 							link_event?"YES":"NO")));
@@ -1464,22 +1544,25 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 	return count;
 }
 
-static bool write_ipc_command_with_result_data(WDIPCCommandData* IPCCommand, char type, char* data, int len)
+static bool write_ipc_command_with_result_data(WDIPCCommandData* ipcCommand, char type, char* data, int len)
 {
 	int send_len = 0;
-	if (socket_write(IPCCommand->issueing_sock, &type, 1) < 0)
+	if (socket_write(ipcCommand->issueing_sock, &type, 1) < 0)
+	{
 		return false;
-
+	}
 	if (len > 0)
 	{
 		send_len = htonl(len);
-		if (socket_write(IPCCommand->issueing_sock, &send_len, sizeof(int)) < 0)
+		if (socket_write(ipcCommand->issueing_sock, &send_len, sizeof(int)) < 0)
 			return false;
-		if (socket_write(IPCCommand->issueing_sock, data, len) < 0)
+		if (socket_write(ipcCommand->issueing_sock, data, len) < 0)
 			return false;
 	}
-	else if (socket_write(IPCCommand->issueing_sock, &send_len, sizeof(int)) < 0)
+	else if (socket_write(ipcCommand->issueing_sock, &send_len, sizeof(int)) < 0)
+	{
 		return false;
+	}
 	return true;
 }
 
@@ -1633,17 +1716,17 @@ static IPC_CMD_PREOCESS_RES process_IPC_command(WDIPCCommandData* ipcCommand)
 }
 
 
-static IPC_CMD_PREOCESS_RES process_IPC_nodeList_command(WDIPCCommandData* IPCCommand)
+static IPC_CMD_PREOCESS_RES process_IPC_nodeList_command(WDIPCCommandData* ipcCommand)
 {
 	/* get the json for node list */
 	JsonNode* jNode = NULL;
 	int NodeID = -1;
 	bool ret;
 
-	if (IPCCommand->data_len <= 0 || IPCCommand->data_buf == NULL)
+	if (ipcCommand->data_len <= 0 || ipcCommand->data_buf == NULL)
 		return IPC_CMD_ERROR;
 
-	json_value *root = json_parse(IPCCommand->data_buf,IPCCommand->data_len);
+	json_value *root = json_parse(ipcCommand->data_buf,ipcCommand->data_len);
 	/* The root node must be object */
 	if (root == NULL || root->type != json_object)
 	{
@@ -1661,7 +1744,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_nodeList_command(WDIPCCommandData* IPCCo
 
 	json_value_free(root);
 	jNode = get_node_list_json(NodeID);
-	ret = write_ipc_command_with_result_data(IPCCommand, WD_IPC_CMD_RESULT_OK,
+	ret = write_ipc_command_with_result_data(ipcCommand, WD_IPC_CMD_RESULT_OK,
 											 jw_get_json_string(jNode), jw_get_json_length(jNode) +1);
 	jw_destroy(jNode);
 	if (ret == false)
@@ -1669,17 +1752,17 @@ static IPC_CMD_PREOCESS_RES process_IPC_nodeList_command(WDIPCCommandData* IPCCo
 	return IPC_CMD_COMPLETE;
 }
 
-static IPC_CMD_PREOCESS_RES process_IPC_nodeStatusChange_command(WDIPCCommandData* IPCCommand)
+static IPC_CMD_PREOCESS_RES process_IPC_nodeStatusChange_command(WDIPCCommandData* ipcCommand)
 {
 	int nodeStatus;
 	int nodeID;
 	char *message;
 	bool ret;
 
-	if (IPCCommand->data_len <= 0 || IPCCommand->data_buf == NULL)
+	if (ipcCommand->data_len <= 0 || ipcCommand->data_buf == NULL)
 		return IPC_CMD_ERROR;
 
-	ret = parse_node_status_json(IPCCommand->data_buf, IPCCommand->data_len, &nodeID, &nodeStatus, &message);
+	ret = parse_node_status_json(ipcCommand->data_buf, ipcCommand->data_len, &nodeID, &nodeStatus, &message);
 	
 	if (ret == false)
 	{
@@ -1754,14 +1837,14 @@ static bool fire_node_status_event(int nodeID, int nodeStatus)
 
 
 
-static IPC_CMD_PREOCESS_RES process_IPC_replicate_variable(WDIPCCommandData* IPCCommand)
+static IPC_CMD_PREOCESS_RES process_IPC_replicate_variable(WDIPCCommandData* ipcCommand)
 {
 	char res_type = WD_IPC_CMD_RESULT_BAD;
 
 	if (get_local_node_state() == WD_STANDBY ||
 		get_local_node_state() == WD_COORDINATOR)
 	{
-		IPC_CMD_PREOCESS_RES execute_res = execute_replicate_command(IPCCommand);
+		IPC_CMD_PREOCESS_RES execute_res = execute_replicate_command(ipcCommand);
 
 		if (execute_res == IPC_CMD_COMPLETE)
 		{
@@ -1785,7 +1868,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_replicate_variable(WDIPCCommandData* IPC
 		res_type = WD_IPC_CMD_CLUSTER_IN_TRAN;
 	}
 
-	if (write_ipc_command_with_result_data(IPCCommand, res_type, NULL, 0))
+	if (write_ipc_command_with_result_data(ipcCommand, res_type, NULL, 0))
 	{
 		/*
 		 * This is the complete lifecycle of command.
@@ -1797,7 +1880,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_replicate_variable(WDIPCCommandData* IPC
 	return IPC_CMD_ERROR;
 }
 
-static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandData *IPCCommand)
+static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandData *ipcCommand)
 {
 	char res_type = WD_IPC_CMD_RESULT_BAD;
 	/*
@@ -1807,7 +1890,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandDat
 	ereport(LOG,
 			(errmsg("processing master node data request from IPC socket")));
 
-	IPCCommand->type = WD_GET_MASTER_DATA_REQUEST;
+	ipcCommand->type = WD_GET_MASTER_DATA_REQUEST;
 	if (get_local_node_state() == WD_STANDBY)
 	{
 		/* I am a standby node, Just forward the request to coordinator */
@@ -1816,9 +1899,10 @@ static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandDat
 		init_wd_packet(&wdPacket);
 		set_message_type(&wdPacket, WD_GET_MASTER_DATA_REQUEST);
 		set_next_commandID_in_message(&wdPacket);
-		set_message_data(&wdPacket, IPCCommand->data_buf , IPCCommand->data_len);
+		set_message_data(&wdPacket, ipcCommand->data_buf , ipcCommand->data_len);
 		/* save the command ID */
-		IPCCommand->internal_command_id = wdPacket.command_id;
+		ipcCommand->internal_command_id = wdPacket.command_id;
+		allocate_resultNodes_in_IPCCommand(ipcCommand);
 		if (send_message(g_cluster.masterNode, &wdPacket) <= 0)
 		{
 			ereport(LOG,
@@ -1829,6 +1913,20 @@ static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandDat
 		}
 		else
 		{
+			int i;
+			for (i=0; i< g_cluster.remoteNodeCount; i++)
+			{
+				WDCommandNodeResult* nodeResult = &ipcCommand->nodeResults[i];
+				if (nodeResult->wdNode == g_cluster.masterNode)
+				{
+					nodeResult->cmdState = COMMAND_STATE_SENT;
+				}
+				else
+				{
+					nodeResult->cmdState = COMMAND_STATE_DO_NOT_SEND;
+				}
+			}
+
 			/*
 			 * we need to wait for the result
 			 */
@@ -1849,7 +1947,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandDat
 		res_type = WD_IPC_CMD_CLUSTER_IN_TRAN;
 	}
 
-	if (write_ipc_command_with_result_data(IPCCommand, res_type, NULL, 0))
+	if (write_ipc_command_with_result_data(ipcCommand, res_type, NULL, 0))
 	{
 		/*
 		 * This is the complete lifecycle of command.
@@ -1861,7 +1959,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandDat
 
 }
 
-static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandData *IPCCommand)
+static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandData *ipcCommand)
 {
 	char res_type = WD_IPC_CMD_RESULT_BAD;
 	/*
@@ -1871,7 +1969,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandDat
 	ereport(LOG,
 		(errmsg("processing sync request from IPC socket")));
 
-	IPCCommand->type = WD_FAILOVER_CMD_SYNC_REQUEST;
+	ipcCommand->type = WD_FAILOVER_CMD_SYNC_REQUEST;
 	if (get_local_node_state() == WD_STANDBY)
 	{
 		/* I am a standby node, Just forward the request to coordinator */
@@ -1880,9 +1978,10 @@ static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandDat
 		init_wd_packet(&wdPacket);
 		set_message_type(&wdPacket, WD_FAILOVER_CMD_SYNC_REQUEST);
 		set_next_commandID_in_message(&wdPacket);
-		set_message_data(&wdPacket, IPCCommand->data_buf , IPCCommand->data_len);
+		set_message_data(&wdPacket, ipcCommand->data_buf , ipcCommand->data_len);
 		/* save the command ID */
-		IPCCommand->internal_command_id = wdPacket.command_id;
+		ipcCommand->internal_command_id = wdPacket.command_id;
+		allocate_resultNodes_in_IPCCommand(ipcCommand);
 		if (send_message(g_cluster.masterNode, &wdPacket) <= 0)
 		{
 			ereport(LOG,
@@ -1893,6 +1992,19 @@ static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandDat
 		}
 		else
 		{
+			int i;
+			for (i=0; i< g_cluster.remoteNodeCount; i++)
+			{
+				WDCommandNodeResult* nodeResult = &ipcCommand->nodeResults[i];
+				if (nodeResult->wdNode == g_cluster.masterNode)
+				{
+					nodeResult->cmdState = COMMAND_STATE_SENT;
+				}
+				else
+				{
+					nodeResult->cmdState = COMMAND_STATE_DO_NOT_SEND;
+				}
+			}
 			/*
 			 * we need to wait for the result
 			 */
@@ -1908,7 +2020,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandDat
 		/*
 		 * If I am coordinator, Just process the request locally
 		 */
-		process_failover_command_sync_requests(g_cluster.localNode, NULL, IPCCommand);
+		process_failover_command_sync_requests(g_cluster.localNode, NULL, ipcCommand);
 		return IPC_CMD_COMPLETE;
 	}
 	else /* we are not in any stable state at the moment */
@@ -1916,7 +2028,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandDat
 		res_type = WD_IPC_CMD_CLUSTER_IN_TRAN;
 	}
 
-	if (write_ipc_command_with_result_data(IPCCommand, res_type, NULL, 0))
+	if (write_ipc_command_with_result_data(ipcCommand, res_type, NULL, 0))
 	{
 		/*
 		 * This is the complete lifecycle of command.
@@ -3184,7 +3296,7 @@ static int standard_packet_processor(WatchdogNode* wdNode, WDPacketData* pkt)
 		case WD_IAM_COORDINATOR_MESSAGE:
 		{
 			/*
-			 * if the message is received from coordinator reply with infor,
+			 * if the message is received from coordinator reply with info,
 			 * otherwise reject
 			 */
 			if (g_cluster.masterNode != NULL && wdNode != g_cluster.masterNode)
@@ -3230,13 +3342,36 @@ static bool send_message_to_connection(SocketConnection* conn, WDPacketData *pkt
 
 static bool send_message_to_node(WatchdogNode* wdNode, WDPacketData *pkt)
 {
-	if (send_message_to_connection(&wdNode->client_socket,pkt) == true)
-		return true;
-	if (send_message_to_connection(&wdNode->server_socket,pkt) == true)
-		return true;
-	ereport(DEBUG1,
-			(errmsg("sending packet to node \"%s\" failed, closing connection", wdNode->nodeName)));
-	return false;
+	bool ret;
+
+	ret = send_message_to_connection(&wdNode->client_socket,pkt);
+
+	if (ret == false)
+	{
+		ret = send_message_to_connection(&wdNode->server_socket,pkt);
+	}
+	if (ret)
+	{
+		/* we only update the last sent time if reply for packet is expected */
+		switch (pkt->type) {
+			case WD_ADD_NODE_MESSAGE:
+			case WD_REQ_INFO_MESSAGE:
+			case WD_REPLICATE_VARIABLE_REQUEST:
+			case WD_ASK_FOR_POOL_CONFIG:
+			case WD_STAND_FOR_COORDINATOR_MESSAGE:
+			case WD_DECLARE_COORDINATOR_MESSAGE:
+				gettimeofday(&wdNode->last_sent_time, NULL);
+		break;
+			default:
+    break;
+		}
+	}
+	else
+	{
+		ereport(DEBUG1,
+				(errmsg("sending packet %c to node \"%s\" failed", pkt->type,wdNode->nodeName)));
+	}
+	return ret;
 }
 
 /*
@@ -3254,11 +3389,11 @@ static int send_message(WatchdogNode* wdNode, WDPacketData *pkt)
 			return 1;
 		return 0;
 	}
-	/* NULL means send to all nodes */
+	/* NULL means send to all reachable odes */
 	for (i=0; i< g_cluster.remoteNodeCount; i++)
 	{
 		wdNode = &(g_cluster.remoteNodes[i]);
-		if (send_message_to_node(wdNode,pkt))
+		if (is_node_reachable(wdNode) && send_message_to_node(wdNode,pkt))
 			count++;
 	}
 	return count;
@@ -3287,6 +3422,101 @@ static void service_internal_command(void)
 	}
 }
 
+static void node_lost_while_ipc_command(WatchdogNode* wdNode)
+{
+	List* ipcCommands_to_del = NIL;
+	ListCell *lc;
+	foreach(lc, g_cluster.ipc_commands)
+	{
+
+		WDIPCCommandData* ipcCommand = lfirst(lc);
+		if (ipcCommand)
+		{
+			if (ipcCommand->type == WD_REPLICATE_VARIABLE_REQUEST)
+			{
+				if (node_lost_during_replicate_command_processing(wdNode, ipcCommand) == IPC_CMD_COMPLETE)
+				{
+					ipcCommands_to_del = lappend(ipcCommands_to_del,ipcCommand);
+				}
+			}
+			else if (ipcCommand->type == WD_FAILOVER_CMD_SYNC_REQUEST ||
+					 ipcCommand->type == WD_GET_MASTER_DATA_REQUEST)
+			{
+				int i;
+				for (i=0; i< g_cluster.remoteNodeCount; i++)
+				{
+					WDCommandNodeResult* nodeResult = &ipcCommand->nodeResults[i];
+					if (nodeResult->wdNode == wdNode)
+					{
+						if (nodeResult->cmdState == COMMAND_STATE_SENT)
+						{
+							ereport(LOG,
+									(errmsg("remote node \"%s\" lost while ipc command was in progress ",wdNode->nodeName)));
+							/* since the node is down so we may never get the results back */
+							write_ipc_command_with_result_data(ipcCommand, WD_IPC_CMD_CLUSTER_IN_TRAN, NULL, 0);
+							ipcCommands_to_del = lappend(ipcCommands_to_del,ipcCommand);
+
+							break;
+						}
+					}
+				}
+			}
+		}
+	}
+	/* delete completed commands */
+	foreach(lc, ipcCommands_to_del)
+	{
+		WDIPCCommandData* ipcCommand = lfirst(lc);
+		cleanUpIPCCommand(ipcCommand);
+	}
+}
+
+
+
+static void service_ipc_commands(void)
+{
+	ListCell *lc;
+	foreach(lc, g_cluster.ipc_commands)
+	{
+		WDIPCCommandData* ipcCommand = lfirst(lc);
+
+		if (ipcCommand && ipcCommand->sendTo_error_count)
+		{
+			int i;
+			for (i=0; i< g_cluster.remoteNodeCount; i++)
+			{
+				WDCommandNodeResult* nodeResult = &ipcCommand->nodeResults[i];
+				if (nodeResult->cmdState == COMMAND_STATE_SEND_ERROR)
+				{
+					if (is_node_reachable(nodeResult->wdNode))
+					{
+						WDPacketData wdPacket;
+
+						ereport(LOG,
+								(errmsg("remote node \"%s\" is reachable again, resending the command packet ",nodeResult->wdNode->nodeName)));
+
+						init_wd_packet(&wdPacket);
+						set_message_type(&wdPacket, ipcCommand->type);
+						set_message_commandID(&wdPacket,ipcCommand->internal_command_id);
+						set_message_data(&wdPacket,ipcCommand->data_buf, ipcCommand->data_len);
+
+						if (send_message_to_node(nodeResult->wdNode, &wdPacket) == true)
+						{
+							nodeResult->cmdState = COMMAND_STATE_SENT;
+							ipcCommand->sendTo_error_count--;
+							ipcCommand->sendTo_count++;
+							if (ipcCommand->sendTo_error_count == 0)
+								break;
+						}
+					}
+				}
+			}
+		}
+	}
+}
+
+
+
 static bool watchdog_internal_command_packet_processor(WatchdogNode* wdNode, WDPacketData* pkt)
 {
 	int i;
@@ -3332,7 +3562,6 @@ static bool watchdog_internal_command_packet_processor(WatchdogNode* wdNode, WDP
 
 	if (g_cluster.currentCommand.commandReplyFromCount >= g_cluster.currentCommand.commandSendToCount)
 	{
-		g_cluster.currentCommand.commandFinished = true;
 		if (pkt->type == WD_REJECT_MESSAGE || pkt->type == WD_ERROR_MESSAGE)
 			g_cluster.currentCommand.commandStatus = COMMAND_FINISHED_NODE_REJECTED;
 		else
@@ -3342,7 +3571,6 @@ static bool watchdog_internal_command_packet_processor(WatchdogNode* wdNode, WDP
 	else if (pkt->type == WD_REJECT_MESSAGE || pkt->type == WD_ERROR_MESSAGE)
 	{
 		/* Error or reject message by any node imidiately finishes the command */
-		g_cluster.currentCommand.commandFinished = true;
 		g_cluster.currentCommand.commandStatus = COMMAND_FINISHED_NODE_REJECTED;
 		watchdog_state_machine(WD_EVENT_COMMAND_FINISHED, wdNode, pkt);
 	}
@@ -3350,65 +3578,26 @@ static bool watchdog_internal_command_packet_processor(WatchdogNode* wdNode, WDP
 	return true; /* do not process this packet further */
 }
 
+static bool is_cluster_command_in_progress(void)
+{
+	return g_cluster.currentCommand.commandStatus == COMMAND_IN_PROGRESS;
+}
 
 static void check_for_current_command_timeout(void)
 {
 	struct timeval currTime;
-	if (g_cluster.currentCommand.commandStatus != COMMAND_IN_PROGRESS ||
-		g_cluster.currentCommand.commandFinished != 0)
+
+	if (!is_cluster_command_in_progress())
 		return;
 
 	gettimeofday(&currTime,NULL);
 	if (WD_TIME_DIFF_SEC(currTime,g_cluster.currentCommand.commandTime) >=  g_cluster.currentCommand.commandTimeoutSecs)
 	{
-		g_cluster.currentCommand.commandFinished = true;
 		g_cluster.currentCommand.commandStatus = COMMAND_FINISHED_TIMEOUT;
 		watchdog_state_machine(WD_EVENT_COMMAND_FINISHED, NULL, NULL);
 	}
 }
 
-static char get_current_command_resultant_message_type(void)
-{
-	char res = WD_ACCEPT_MESSAGE;
-	int i;
-	if (g_cluster.currentCommand.commandFinished == 0)
-		return WD_NO_MESSAGE;
-	if (g_cluster.currentCommand.sendToNode == NULL)
-	{
-		/* The command was for all nodes */
-		for (i = 0; i< g_cluster.remoteNodeCount; i++)
-		{
-			WDCommandNodeResult* nodeRes = &g_cluster.currentCommand.nodeResults[i];
-			if (nodeRes->cmdState != COMMAND_STATE_REPLIED)
-				continue;
-			if (nodeRes->result_type != WD_ACCEPT_MESSAGE && nodeRes->result_type != WD_INFO_MESSAGE)
-			{
-				/* failed */
-				if (res != WD_ERROR_MESSAGE)
-					res = nodeRes->result_type;
-			}
-		}
-	}
-	else
-	{
-		if (g_cluster.currentCommand.commandSendToCount == 0) /* We failed to send to any node */
-			return WD_ERROR_MESSAGE;
-		if (g_cluster.currentCommand.commandReplyFromCount == 0) /* We got no reply */
-			return WD_ERROR_MESSAGE;
-		for (i = 0; i< g_cluster.remoteNodeCount; i++)
-		{
-			WDCommandNodeResult* nodeRes = &g_cluster.currentCommand.nodeResults[i];
-			if (nodeRes->wdNode == g_cluster.currentCommand.sendToNode)
-			{
-				if (nodeRes->cmdState != COMMAND_STATE_REPLIED)
-					return WD_ERROR_MESSAGE;
-				return nodeRes->result_type;
-			}
-		}
-		return WD_ERROR_MESSAGE;
-	}
-	return res;
-}
 
 static void clear_current_command(void)
 {
@@ -3458,7 +3647,7 @@ static int issue_watchdog_internal_command(WatchdogNode* wdNode, WDPacketData *p
 				if (send_message_to_node(nodeResult->wdNode, pkt) == false)
 				{
 					ereport(DEBUG1,
-							(errmsg("failed to send watchdog internla command packet %s",nodeResult->wdNode->nodeName),
+							(errmsg("failed to send watchdog internal command packet %s",nodeResult->wdNode->nodeName),
 							 errdetail("saving the packet. will try to resend it if connection recovers")));
 
 					/* failed to send. May be try again later */
@@ -3506,7 +3695,6 @@ static int issue_watchdog_internal_command(WatchdogNode* wdNode, WDPacketData *p
 		memcpy(g_cluster.currentCommand.packet.data,pkt->data,pkt->len);
 		g_cluster.currentCommand.packet.len = pkt->len;
 	}
-	g_cluster.currentCommand.commandFinished = false;
 	return g_cluster.currentCommand.commandSendToCount;
 }
 
@@ -3535,7 +3723,7 @@ static bool service_lost_connections(void)
 	for (i = 0; i< g_cluster.remoteNodeCount; i++)
 	{
 		WatchdogNode* wdNode = &(g_cluster.remoteNodes[i]);
-		if (wdNode->state == WD_SHUTDOWN)
+		if (wdNode->state == WD_SHUTDOWN || wdNode->state == WD_DEAD)
 			continue;
 
 		if (is_socket_connection_connected(&wdNode->client_socket) == false)
@@ -3553,7 +3741,6 @@ static bool service_lost_connections(void)
 					watchdog_state_machine(WD_EVENT_NEW_OUTBOUND_CONNECTION, wdNode, NULL);
 					ret = true;
 				}
-
 			}
 		}
 	}
@@ -3627,6 +3814,12 @@ static int send_cluster_command(WatchdogNode* wdNode, char type, int timeout_sec
 	WDPacketData *pkt = get_message_of_type(type);
 	if (pkt)
 	{
+		/* If the old command is in progress and we are about to issue a new command
+		 * finish the old command with timeout
+		 */
+		if (is_cluster_command_in_progress())
+			watchdog_state_machine(WD_EVENT_COMMAND_FINISHED, NULL, NULL);
+
 		ret = issue_watchdog_internal_command(wdNode, pkt, timeout_sec);
 		free_packet(pkt);
 	}
@@ -3834,6 +4027,10 @@ static int watchdog_state_machine(WD_EVENTS event, WatchdogNode* wdNode, WDPacke
 	
 	if (event == WD_EVENT_REMOTE_NODE_LOST)
 	{
+		/* close all socket connections to the node */
+		close_socket_connection(&wdNode->client_socket);
+		close_socket_connection(&wdNode->server_socket);
+
 		if (wdNode->state == WD_SHUTDOWN)
 		{
 			ereport(LOG,
@@ -3851,6 +4048,10 @@ static int watchdog_state_machine(WD_EVENTS event, WatchdogNode* wdNode, WDPacke
 					(errmsg("watchdog cluster has lost the coordinator node")));
 			g_cluster.masterNode = NULL;
 		}
+		/* clear the wait timer on the node */
+		wdNode->last_sent_time.tv_sec = 0;
+		wdNode->last_sent_time.tv_usec = 0;
+		node_lost_while_ipc_command(wdNode);
 	}
 	else if (event == WD_EVENT_PACKET_RCV)
 	{
@@ -3999,7 +4200,7 @@ static int watchdog_state_machine_loading(WD_EVENTS event, WatchdogNode* wdNode,
 				}
 			}
 			free_packet(addPkt);
-			set_timeout(4);
+			set_timeout(MAX_SECS_WAIT_FOR_REPLY_FROM_NODE);
 		}
 			break;
 
@@ -4086,8 +4287,9 @@ static int watchdog_state_machine_joining(WD_EVENTS event, WatchdogNode* wdNode,
 	{
 		case WD_EVENT_WD_STATE_CHANGED:
 			g_cluster.masterNode = NULL;
+			try_connecting_with_all_unreachable_nodes();
 			send_cluster_command(NULL, WD_REQ_INFO_MESSAGE, 5);
-			set_timeout(5);
+			set_timeout(MAX_SECS_WAIT_FOR_REPLY_FROM_NODE);
 			break;
 
 		case WD_EVENT_TIMEOUT:
@@ -4215,7 +4417,7 @@ static int watchdog_state_machine_initializing(WD_EVENTS event, WatchdogNode* wd
 					if (wdNode->state == WD_ADD_MESSAGE_SENT)
 						ereport(FATAL,
 							(return_code(POOL_EXIT_FATAL),
-								errmsg("Add to watchdog cluster request is rejected by node \"%s:%d\"",wdNode->hostname,wdNode->wd_port),
+								errmsg("add to watchdog cluster request is rejected by node \"%s:%d\"",wdNode->hostname,wdNode->wd_port),
 								 errhint("check the watchdog configurations.")));
 					break;
 				default:
@@ -4237,9 +4439,9 @@ static int watchdog_state_machine_standForCord(WD_EVENTS event, WatchdogNode* wd
 	switch (event)
 	{
 		case WD_EVENT_WD_STATE_CHANGED:
-			send_cluster_command(NULL, WD_STAND_FOR_COORDINATOR_MESSAGE, 5);
+			send_cluster_command(NULL, WD_STAND_FOR_COORDINATOR_MESSAGE, 4);
 			/* wait for 5 seconds if someone rejects us*/
-			set_timeout(5);
+			set_timeout(MAX_SECS_WAIT_FOR_REPLY_FROM_NODE);
 			break;
 
 		case WD_EVENT_COMMAND_FINISHED:
@@ -4358,8 +4560,8 @@ static int watchdog_state_machine_coordinator(WD_EVENTS event, WatchdogNode* wdN
 		case WD_EVENT_WD_STATE_CHANGED:
 		{
 			int i;
-			send_cluster_command(NULL, WD_DECLARE_COORDINATOR_MESSAGE, 5);
-			set_timeout(10);
+			send_cluster_command(NULL, WD_DECLARE_COORDINATOR_MESSAGE, 4);
+			set_timeout(MAX_SECS_WAIT_FOR_REPLY_FROM_NODE);
 			ereport(LOG,
 					(errmsg("I am announcing my self as master/coordinator watchdog node")));
 
@@ -4510,8 +4712,15 @@ static int watchdog_state_machine_coordinator(WD_EVENTS event, WatchdogNode* wdN
 			break;
 
 		case WD_EVENT_TIMEOUT:
-			send_cluster_command(NULL, WD_IAM_COORDINATOR_MESSAGE, BEACON_MESSAGE_INTERVAL_SECONDS);
-			set_timeout(BEACON_MESSAGE_INTERVAL_SECONDS);
+			if (is_cluster_command_in_progress())
+			{
+				set_timeout(MAX_SECS_WAIT_FOR_REPLY_FROM_NODE);
+			}
+			else
+			{
+				send_cluster_command(NULL, WD_IAM_COORDINATOR_MESSAGE, 5);
+				set_timeout(BEACON_MESSAGE_INTERVAL_SECONDS);
+			}
 			break;
 
 		case WD_EVENT_REMOTE_NODE_LOST:
@@ -4785,7 +4994,7 @@ static int watchdog_state_machine_voting(WD_EVENTS event, WatchdogNode* wdNode,
 	{
 		case WD_EVENT_WD_STATE_CHANGED:
 			clear_current_command();
-			set_timeout(6);
+			set_timeout(MAX_SECS_WAIT_FOR_REPLY_FROM_NODE);
 			break;
 			
 		case WD_EVENT_TIMEOUT:
@@ -5103,8 +5312,12 @@ static IPC_CMD_PREOCESS_RES execute_replicate_command(WDIPCCommandData* ipcComma
 {
 	int i;
 	IPC_CMD_PREOCESS_RES res;
-	
 	WDPacketData wdPacket;
+
+	/* save the hassel if I am the only alive node */
+	if (get_cluster_node_count() == 0)
+		return IPC_CMD_COMPLETE;
+
 	init_wd_packet(&wdPacket);
 	set_message_type(&wdPacket, WD_REPLICATE_VARIABLE_REQUEST);
 	set_next_commandID_in_message(&wdPacket);
@@ -5112,21 +5325,38 @@ static IPC_CMD_PREOCESS_RES execute_replicate_command(WDIPCCommandData* ipcComma
 	
 	allocate_resultNodes_in_IPCCommand(ipcCommand);
 	ipcCommand->sendTo_count = 0;
+	ipcCommand->sendTo_error_count = 0;
 	ipcCommand->reply_from_count = 0;
 	ipcCommand->internal_command_id = wdPacket.command_id;
 	ipcCommand->type = wdPacket.type;
-	
+
 	for (i=0; i< g_cluster.remoteNodeCount; i++)
 	{
 		WDCommandNodeResult* nodeResult = &ipcCommand->nodeResults[i];
-		if (send_message_to_node(nodeResult->wdNode, &wdPacket) == true)
+		if (nodeResult->wdNode->state == WD_DEAD || nodeResult->wdNode->state == WD_LOST || nodeResult->wdNode->state == WD_SHUTDOWN)
+		{
+			ereport(DEBUG2,
+					(errmsg("not sending ipc command packet to DEAD node \"%s\"",nodeResult->wdNode->nodeName)));
+			/* Do not send to dead node */
+			nodeResult->cmdState = COMMAND_STATE_DO_NOT_SEND;
+		}
+		else if (send_message_to_node(nodeResult->wdNode, &wdPacket) == true)
 		{
 			nodeResult->cmdState = COMMAND_STATE_SENT;
 			ipcCommand->sendTo_count++;
 		}
 		else
+		{
 			nodeResult->cmdState = COMMAND_STATE_SEND_ERROR;
+			ipcCommand->sendTo_error_count++;
+		}
 	}
+	/* Our first priority is to send to all alive connected nodes
+	 * so if send error count is not zero return IPC_PROCESSING and retry
+	 * sending later in the cycle
+	 */
+	return IPC_CMD_PROCESSING;
+
 	/*
 	 * The current quorum status of standby node can be out of sync,
 	 * so update it before making any decision on command success
@@ -5306,6 +5536,56 @@ static bool process_wd_command_function(WatchdogNode* wdNode, WDPacketData* pkt,
 	return true;
 }
 
+static IPC_CMD_PREOCESS_RES node_lost_during_replicate_command_processing(WatchdogNode* wdNode, WDIPCCommandData* ipcCommand)
+{
+	int i;
+	/* this command is needed to be sent to all connected nodes */
+	for (i=0; i< g_cluster.remoteNodeCount; i++)
+	{
+		WDCommandNodeResult* nodeResult = &ipcCommand->nodeResults[i];
+		if (nodeResult->wdNode != wdNode)
+			continue;
+
+		/* if the command state is sent and the node that is lost
+		 * so we will never get the reply back from that node
+		 */
+		if (nodeResult->cmdState == COMMAND_STATE_SENT)
+		{
+			nodeResult->cmdState = COMMAND_STATE_DO_NOT_SEND;
+			ereport(WARNING,
+					(errmsg("watchdog node \"%s\" was lost during processing of pgpool-II replicate command",wdNode->nodeName)));
+			if (ipcCommand->sendTo_count)
+				ipcCommand->sendTo_count--;
+		}
+		break;
+	}
+
+	if (ipcCommand->reply_from_count >= ipcCommand->sendTo_count)
+	{
+		/*
+		 * we have received results from all nodes
+		 * analyze the result
+		 */
+		char res_type = WD_IPC_CMD_RESULT_OK;
+
+		for (i=0; i< g_cluster.remoteNodeCount; i++)
+		{
+			WDCommandNodeResult* nodeResult = &ipcCommand->nodeResults[i];
+			if (nodeResult->cmdState == COMMAND_STATE_REPLIED &&
+				nodeResult->result_type != WD_ACCEPT_MESSAGE)
+			{
+				res_type = WD_IPC_CMD_RESULT_BAD;
+				break;
+			}
+		}
+		if (write_ipc_command_with_result_data(ipcCommand, res_type, NULL, 0) == false)
+			ereport(LOG,
+					(errmsg("failed to forward replicate command results to IPC command socket")));
+
+		return IPC_CMD_COMPLETE;
+	}
+	return IPC_CMD_PROCESSING;
+}
 
 static bool reply_is_received_for_pgpool_replicate_command(WatchdogNode* wdNode, WDPacketData* pkt, WDIPCCommandData* ipcCommand)
 {
@@ -5328,13 +5608,26 @@ static bool reply_is_received_for_pgpool_replicate_command(WatchdogNode* wdNode,
 				(errmsg("unable to find result node for pgpool-II replicate command packet received from watchdog node \"%s\"",wdNode->nodeName)));
 		return true;
 	}
+
 	nodeResult->result_type = pkt->type;
 	nodeResult->cmdState = COMMAND_STATE_REPLIED;
 	ipcCommand->reply_from_count++;
+
 	ereport(DEBUG2,
 		(errmsg("watchdog node \"%s\" has replied for pgpool-II replicate command packet",wdNode->nodeName),
 			 errdetail("command was sent to %d nodes and %d nodes have replied to it",ipcCommand->sendTo_count,ipcCommand->reply_from_count)));
 
+	/* reject message from any node finishes the command */
+	if (pkt->type != WD_ACCEPT_MESSAGE)
+	{
+
+		if (write_ipc_command_with_result_data(ipcCommand, WD_IPC_CMD_RESULT_BAD, NULL, 0) == false)
+			ereport(LOG,
+					(errmsg("failed to forward message to IPC command socket")));
+		cleanUpIPCCommand(ipcCommand);
+		return true;
+	}
+
 	if (ipcCommand->reply_from_count >= ipcCommand->sendTo_count)
 	{
 		/*
diff --git a/src/watchdog/wd_commands.c b/src/watchdog/wd_commands.c
index 0133267..851c156 100644
--- a/src/watchdog/wd_commands.c
+++ b/src/watchdog/wd_commands.c
@@ -214,7 +214,7 @@ issue_command_to_watchdog(char type, int timeout_sec, char* data, int data_len,
 				if (socket_read(sock, &res_type, 1 ,0) <=0)
 				{
 					ereport(WARNING,
-						(errmsg("error reading from IPC command socket"),
+						(errmsg("error reading from IPC command socket for ipc command %c",type),
 							 errdetail("read from socket failed with error \"%s\"",strerror(errno))));
 					close(sock);
 					return result;
@@ -223,7 +223,7 @@ issue_command_to_watchdog(char type, int timeout_sec, char* data, int data_len,
 				if (socket_read(sock, &res_length, sizeof(int), 0) <= 0)
 				{
 					ereport(WARNING,
-						(errmsg("error reading from IPC command socket"),
+						(errmsg("error reading from IPC command socket for ipc command %c",type),
 							 errdetail("read from socket failed with error \"%s\"",strerror(errno))));
 					close(sock);
 					return result;
@@ -242,7 +242,7 @@ issue_command_to_watchdog(char type, int timeout_sec, char* data, int data_len,
 						pfree(result->data);
 						pfree(result);
 						ereport(DEBUG1,
-							(errmsg("error reading from IPC command socket"),
+							(errmsg("error reading from IPC command socket for ipc command %c",type),
 								 errdetail("read from socket failed with error \"%s\"",strerror(errno))));
 						close(sock);
 						return NULL;
@@ -509,25 +509,34 @@ wd_send_failover_sync_command(char* syncReqType, enum WDFailoverLocks lockID)
 
 	pfree(json_data);
 
-	if (result == NULL || result->length <= 0)
+	if (result == NULL)
 	{
 		ereport(WARNING,
-			(errmsg("watchdog failed to send failover command"),
+			(errmsg("watchdog failover interlocking command failed"),
 				 errdetail("issue command to watchdog returned NULL")));
 		return FAILOVER_RES_ERROR;
 	}
 	else if (result->type == WD_IPC_CMD_TIMEOUT)
 	{
 		ereport(WARNING,
-			(errmsg("watchdog failed to send failover command"),
+			(errmsg("watchdog failover interlocking command failed"),
 				 errdetail("ipc command timeout")));
 		FreeCmdResult(result);
 		return FAILOVER_RES_ERROR;
 	}
+	if (result->type == WD_IPC_CMD_CLUSTER_IN_TRAN)
+	{
+		ereport(WARNING,
+				(errmsg("watchdog failover interlocking command failed"),
+				 errdetail("watchdog cluster is not in stable state"),
+					errhint("try again when the cluster is fully initialized")));
+		return FAILOVER_RES_TRANSITION;
+	}
+
 	if (result->length <= 0)
 	{
 		ereport(WARNING,
-			(errmsg("watchdog failed to send failover command"),
+			(errmsg("watchdog failover interlocking command failed"),
 				 errdetail("issue command to watchdog returned no data")));
 		FreeCmdResult(result);
 		return FAILOVER_RES_ERROR;
@@ -577,7 +586,7 @@ wd_degenerate_backend_set(int *node_id_set, int count)
 													   WD_DEFAULT_IPC_COMMAND_TIMEOUT,
 													   func, strlen(func), true);
 	pfree(func);
-	
+
 	if (result == NULL)
 	{
 		ereport(WARNING,
@@ -623,7 +632,7 @@ wd_promote_backend(int node_id)
 	/* if promote packet is received already, do nothing */
 	if (wd_chk_node_mask_for_promote_req(&n,1))
 		return COMMAND_OK;
-	
+
 	func = get_wd_node_function_json(WD_FUNCTION_PROMOTE_REQUEST,&n, 1,
 									 shared_key?*shared_key:0,pool_config->wd_authkey);
 	result = issue_command_to_watchdog(WD_FUNCTION_COMMAND ,
@@ -822,12 +831,12 @@ static WDFailoverCMDResults wd_issue_failover_lock_command(char* syncReqType, en
 {
 	WDFailoverCMDResults res;
 	int x;
-	for (x=0; x < MAX_SEC_WAIT_FOR_CLUSTER_TRANSATION; x++)
+	for (x=0; x <= MAX_SEC_WAIT_FOR_CLUSTER_TRANSATION/2; x++)
 	{
 		res = wd_send_failover_sync_command(syncReqType, lockID);
 		if (res != FAILOVER_RES_TRANSITION)
 			break;
-		sleep(1);
+		sleep(2);
 	}
 	return res;
 }
failover_standby_fix.diff (45,682 bytes)

Muhammad Usama

2016-09-27 23:30

developer   ~0001090

Hi

Sorry for the delayed response. The issue was a little more complex than origin anticipated. Can you please try the attached a patch (failover_standby_fix.diff) if it solves the issue.

Not that you need to apply the patch on the current head of master branch and if it fixes the problem I will also back port it to pgpool-II 3.5 branch.

Thanks
 

supp_k

2016-09-28 00:46

reporter   ~0001092

Yes, the issue is fixed. Quick tests detects the problem disappeared. Can you please apply it to branch 3.5?


Another problem appeares on the new build "pgpool-II version 3.5.4 (ekieboshi)".
It seems something has changed in authentication. The new build returns error: "Caused by: org.postgresql.util.PSQLException: ERROR: MD5 authentication is unsupported in replication and master-slave modes." Can be happen because the versions is alpha?

supp_k

2016-09-28 01:15

reporter   ~0001093

Sorry for the mistake - auth error is detected in "pgpool-II version 3.6-alpha1 (subaruboshi)"

t-ishii

2016-09-28 09:44

developer   ~0001096

Regarding the auth problem, yes I recently changed auth module. Please report it using new bug report.

Dang Minh Huong

2016-09-28 15:49

reporter   ~0001104

Thanks for fixing this issue.
Our customer also have facing with this issue too.
When will this patch is applied to STABLE versions sir?

I also think, this is a critical problem.
Is it should be a minor version (for example 3.5.5?) release here sir?

supp_k

2016-09-28 16:09

reporter   ~0001105

There is one remark regarding fixing of the issue. Now I see that in the environment with 3 installations of Pgpool the failover_command is executed by 2 survived Pgpool instances. Should it happen or the only one Pgpool instance should trigger the failover_command?

Muhammad Usama

2016-09-28 17:15

developer   ~0001106

Hi
Thanks for taking the time out for testing.
No failover_command should only be executed by one node only. Can you share the steps to reproduce and the log files for the scenario.

supp_k

2016-09-28 22:01

reporter   ~0001107

Please see the attached file.
The environment consists of:
Server 1) Head Pgpool server
Server 2) Pgpool A + PostgreSQL
Server 3) Pgpool A + PostgreSQL

Server 3 is killed and from Head & A server's logs one can see the failover_command is executed twice.

data.tar.gz (245,760 bytes)

Muhammad Usama

2016-09-28 22:37

developer   ~0001108

Many thanks.

Can you please try out the revised attached patch (failover_standby_fix_v2).

failover_standby_fix_v2.diff (45,544 bytes)
diff --git a/src/include/pool.h b/src/include/pool.h
index 65b3c92..2a0cf56 100644
--- a/src/include/pool.h
+++ b/src/include/pool.h
@@ -373,7 +373,7 @@ extern int my_master_node_id;
 #define ACCEPT_FD_SEM			5
 #define MAX_REQUEST_QUEUE_SIZE	10
 
-#define MAX_SEC_WAIT_FOR_CLUSTER_TRANSATION 6 /* time in seconds to keep retrying for a
+#define MAX_SEC_WAIT_FOR_CLUSTER_TRANSATION 10 /* time in seconds to keep retrying for a
 											   * watchdog command if the cluster is not
 											   * in stable state */
 
diff --git a/src/include/watchdog/watchdog.h b/src/include/watchdog/watchdog.h
index e7c9770..fd986a4 100644
--- a/src/include/watchdog/watchdog.h
+++ b/src/include/watchdog/watchdog.h
@@ -94,6 +94,7 @@ typedef struct SocketConnection
 	struct			timeval tv;		/* connect time of socket */
 	char			addr[48];		/* ip address of socket connection*/
 	WD_SOCK_STATE	sock_state;		/* current state of socket */
+	bool			eof;
 }SocketConnection;
 
 typedef struct WatchdogNode
@@ -103,6 +104,7 @@ typedef struct WatchdogNode
 	struct timeval last_rcv_time;			/* timestamp when last packet
 											 * was received from the node
 											 */
+	struct timeval last_sent_time;
 	char nodeName[WD_MAX_HOST_NAMELEN];		/* name of this node */
 	char hostname[WD_MAX_HOST_NAMELEN];		/* host name */
 	int wd_port;							/* watchdog port */
diff --git a/src/include/watchdog/wd_ipc_commands.h b/src/include/watchdog/wd_ipc_commands.h
index d5dd148..17eadab 100644
--- a/src/include/watchdog/wd_ipc_commands.h
+++ b/src/include/watchdog/wd_ipc_commands.h
@@ -31,7 +31,7 @@
 
 typedef enum WdCommandResult
 {
-	CLUSTER_IN_TRANSATIONING,
+	CLUSTER_IN_TRANSATIONING = 0,
 	COMMAND_OK,
 	COMMAND_FAILED,
 	COMMAND_TIMEOUT
diff --git a/src/main/pgpool_main.c b/src/main/pgpool_main.c
index cd490be..39bc6dd 100644
--- a/src/main/pgpool_main.c
+++ b/src/main/pgpool_main.c
@@ -1195,7 +1195,7 @@ bool degenerate_backend_set_ex(int *node_id_set, int count, bool error, bool tes
 			 * Is proceeding to failover is the right choice ???
 			 */
 			ereport(NOTICE,
-					(errmsg("rdegenerate backend request for %d node(s) from pid [%d], But cluster is not in stable state"
+					(errmsg("degenerate backend request for %d node(s) from pid [%d], But cluster is not in stable state"
 							, node_count, getpid())));
 		}
 
diff --git a/src/utils/pool_stream.c b/src/utils/pool_stream.c
index e1336d3..9fa9db1 100644
--- a/src/utils/pool_stream.c
+++ b/src/utils/pool_stream.c
@@ -1173,9 +1173,32 @@ int socket_read(int fd, void* buf, size_t len, int timeout)
 {
 	int ret, read_len;
 	read_len = 0;
+	struct timeval timeoutval;
+	fd_set readmask;
+	int fds;
 
 	while (read_len < len)
 	{
+		FD_ZERO(&readmask);
+		FD_SET(fd, &readmask);
+
+		timeoutval.tv_sec = timeout;
+		timeoutval.tv_usec = 0;
+
+		fds = select(fd+1, &readmask, NULL, NULL, timeout?&timeoutval:NULL);
+		if (fds == -1)
+		{
+			if (errno == EAGAIN || errno == EINTR)
+				continue;
+
+			ereport(WARNING,
+					(errmsg("select failed with error: \"%s\"", strerror(errno))));
+			return -1;
+		}
+		else if (fds == 0)
+		{
+			return -2;
+		}
 		ret = read(fd, buf + read_len, (len - read_len));
 		if(ret < 0)
 		{
diff --git a/src/watchdog/watchdog.c b/src/watchdog/watchdog.c
index 791efa6..2339cf9 100644
--- a/src/watchdog/watchdog.c
+++ b/src/watchdog/watchdog.c
@@ -77,6 +77,9 @@ typedef enum IPC_CMD_PREOCESS_RES
 
 #define BEACON_MESSAGE_INTERVAL_SECONDS		10 /* interval between beacon messages */
 
+#define MAX_SECS_WAIT_FOR_REPLY_FROM_NODE	5
+
+
 
 
 #define WD_NO_MESSAGE						0
@@ -210,6 +213,7 @@ typedef struct WDIPCCommandData
 	char				*data_buf;
 	
 	unsigned int	sendTo_count;
+	unsigned int	sendTo_error_count;
 	unsigned int	reply_from_count;
 	unsigned int	timeout_secs;
 
@@ -260,7 +264,6 @@ typedef struct WDCommandData
 	struct timeval			commandTime;
 	unsigned int			commandSendToCount;
 	unsigned int			commandReplyFromCount;
-	int						commandFinished;
 	int						partial_sent;
 }WDCommandData;
 
@@ -316,6 +319,9 @@ static void try_connecting_with_all_unreachable_nodes(void);
 static bool connect_to_node(WatchdogNode* wdNode);
 static bool is_socket_connection_connected(SocketConnection* conn);
 
+static bool is_node_reachable(WatchdogNode* wdNode);
+static void service_unreachable_nodes(void);
+
 static int update_successful_outgoing_cons(fd_set* wmask, int pending_fds_count);
 static int prepare_fds(fd_set* rmask, fd_set* wmask, fd_set* emask);
 
@@ -335,11 +341,13 @@ static WDPacketData* read_packet(SocketConnection* conn);
 static WDPacketData* get_message_of_type(char type);
 
 static int issue_watchdog_internal_command(WatchdogNode* wdNode, WDPacketData *pkt, int timeout_sec);
-static char get_current_command_resultant_message_type(void);
 static void check_for_current_command_timeout(void);
+static bool is_cluster_command_in_progress(void);
 static bool watchdog_internal_command_packet_processor(WatchdogNode* wdNode, WDPacketData* pkt);
 static bool service_lost_connections(void);
 static void service_internal_command(void);
+static void service_ipc_commands(void);
+static void node_lost_while_ipc_command(WatchdogNode* wdNode);
 
 static unsigned int get_next_commandID(void);
 static WatchdogNode* parse_node_info_message(WDPacketData* pkt, char **authkey);
@@ -396,13 +404,14 @@ static WDIPCCommandData* get_wd_IPC_command_from_reply(WDPacketData* pkt);
 static WDIPCCommandData* get_wd_IPC_command_from_socket(int sock);
 
 static IPC_CMD_PREOCESS_RES process_IPC_command(WDIPCCommandData* ipcCommand);
-static IPC_CMD_PREOCESS_RES process_IPC_nodeStatusChange_command(WDIPCCommandData* IPCCommand);
-static IPC_CMD_PREOCESS_RES process_IPC_nodeList_command(WDIPCCommandData* IPCCommand);
-static IPC_CMD_PREOCESS_RES process_IPC_replicate_variable(WDIPCCommandData* IPCCommand);
-static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandData *IPCCommand);
-static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandData *IPCCommand);
+static IPC_CMD_PREOCESS_RES process_IPC_nodeStatusChange_command(WDIPCCommandData* ipcCommand);
+static IPC_CMD_PREOCESS_RES process_IPC_nodeList_command(WDIPCCommandData* ipcCommand);
+static IPC_CMD_PREOCESS_RES process_IPC_replicate_variable(WDIPCCommandData* ipcCommand);
+static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandData *ipcCommand);
+static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandData *ipcCommand);
 static IPC_CMD_PREOCESS_RES execute_replicate_command(WDIPCCommandData* ipcCommand);
-static bool write_ipc_command_with_result_data(WDIPCCommandData* IPCCommand, char type, char* data, int len);
+static IPC_CMD_PREOCESS_RES node_lost_during_replicate_command_processing(WatchdogNode* wdNode, WDIPCCommandData* ipcCommand);
+static bool write_ipc_command_with_result_data(WDIPCCommandData* ipcCommand, char type, char* data, int len);
 
 static int node_has_requested_for_interlocking(WatchdogNode* wdNode, WDPacketData* pkt);
 static bool node_has_resigned_from_interlocking(WatchdogNode* wdNode, WDPacketData* pkt);
@@ -437,6 +446,7 @@ static void print_received_packet_info(WDPacketData* pkt,WatchdogNode* wdNode);
 static void update_interface_status(void);
 static bool any_interface_available(void);
 static WDPacketData* process_data_request(WatchdogNode* wdNode, WDPacketData* pkt);
+static void allocate_resultNodes_in_IPCCommand(WDIPCCommandData* ipcCommand);
 
 /* global variables */
 wd_cluster g_cluster;
@@ -713,7 +723,6 @@ wd_create_recv_socket(int port)
 			(errmsg("failed to create watchdog receive socket"),
 				 errdetail("listen failed with reason: \"%s\"", strerror(saved_errno))));
 	}
-
 	return sock;
 }
 
@@ -834,8 +843,15 @@ try_connecting_with_all_unreachable_nodes(void)
 		if (wdNode->client_socket.sock_state != WD_SOCK_WAITING_FOR_CONNECT && wdNode->client_socket.sock_state != WD_SOCK_CONNECTED &&
 			wdNode->server_socket.sock_state != WD_SOCK_WAITING_FOR_CONNECT && wdNode->server_socket.sock_state != WD_SOCK_CONNECTED)
 		{
-			wdNode->state = WD_DEAD;
+			if (wdNode->state == WD_SHUTDOWN)
+				continue;
 			connect_to_node(wdNode);
+			if (wdNode->client_socket.sock_state == WD_SOCK_CONNECTED)
+			{
+				ereport(LOG,
+						(errmsg("connection to the remote node \"%s\" is restored",wdNode->nodeName)));
+				watchdog_state_machine(WD_EVENT_NEW_OUTBOUND_CONNECTION, wdNode, NULL);
+			}
 		}
 	}
 }
@@ -1026,7 +1042,6 @@ watchdog_main(void)
 			if (WD_TIME_DIFF_SEC(ref_time,g_tm_set_time) >=  g_timeout_sec)
 			{
 				timeout_event = true;
-				g_timeout_sec = 0;
 			}
 		}
 		if (select_ret > 0)
@@ -1037,21 +1052,71 @@ watchdog_main(void)
 			processed_fds += read_sockets(&rmask,(select_ret - processed_fds));
 		}
 
-		if (timeout_event)
-			watchdog_state_machine(WD_EVENT_TIMEOUT, NULL, NULL);
 		if (WD_TIME_DIFF_SEC(ref_time,g_tm_set_time) >=  1)
+		{
 			process_wd_func_commands_for_timer_events();
+		}
+
+		if (timeout_event)
+		{
+			g_timeout_sec = 0;
+			watchdog_state_machine(WD_EVENT_TIMEOUT, NULL, NULL);
+		}
 
 		check_for_current_command_timeout();
 
 		if (service_lost_connections() == true)
+		{
 			service_internal_command();
+			service_ipc_commands();
+		}
+
+		service_unreachable_nodes();
 
 		update_connected_node_count();
 	}
 	return 0;
 }
 
+/* remove the unreachable nodes from cluster */
+static void service_unreachable_nodes(void)
+{
+	int i;
+	struct timeval currTime;
+	gettimeofday(&currTime,NULL);
+
+	for (i = 0; i< g_cluster.remoteNodeCount; i++)
+	{
+		WatchdogNode* wdNode = &(g_cluster.remoteNodes[i]);
+		if (wdNode->state == WD_DEAD || wdNode->state == WD_LOST || wdNode->state == WD_SHUTDOWN)
+			continue;
+
+		if (is_node_reachable(wdNode) || wdNode->client_socket.sock_state == WD_SOCK_WAITING_FOR_CONNECT)
+		{
+			/* check if we are waiting for reply from this node */
+			if (wdNode->last_sent_time.tv_sec > 0)
+			{
+				if (WD_TIME_DIFF_SEC(currTime,wdNode->last_sent_time) >=  MAX_SECS_WAIT_FOR_REPLY_FROM_NODE)
+				{
+					ereport(LOG,
+						(errmsg("remote node \"%s\" is not replying..",wdNode->nodeName),
+							 errdetail("marking the node as lost")));
+					/* mark the node as lost */
+					watchdog_state_machine(WD_EVENT_REMOTE_NODE_LOST, wdNode, NULL);
+				}
+			}
+		}
+		else
+		{
+			ereport(LOG,
+				(errmsg("remote node \"%s\" is not reachable",wdNode->nodeName),
+					 errdetail("marking the node as lost")));
+			watchdog_state_machine(WD_EVENT_REMOTE_NODE_LOST, wdNode, NULL);
+		}
+	}
+}
+
+
 static int
 wd_create_command_server_socket(void)
 {
@@ -1233,8 +1298,17 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 				if (pkt)
 				{
 					watchdog_state_machine(WD_EVENT_PACKET_RCV, wdNode, pkt);
+					/* since a packet is received reset last sent time */
+					wdNode->last_sent_time.tv_sec = 0;
+					wdNode->last_sent_time.tv_usec = 0;
 					free_packet(pkt);
 				}
+				else
+				{
+					ereport(LOG,
+							(errmsg("client socket of %s is closed", wdNode->nodeName)));
+				}
+
 				count++;
 				if (count >= pending_fds_count)
 					return count;
@@ -1250,9 +1324,16 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 				if (pkt)
 				{
 					watchdog_state_machine(WD_EVENT_PACKET_RCV, wdNode, pkt);
+					/* since a packet is received reset last sent time */
+					wdNode->last_sent_time.tv_sec = 0;
+					wdNode->last_sent_time.tv_usec = 0;
 					free_packet(pkt);
 				}
-				
+				else
+				{
+					ereport(LOG,
+							(errmsg("server socket of %s is closed", wdNode->nodeName)));
+				}
 				count++;
 				if (count >= pending_fds_count)
 					return count;
@@ -1283,7 +1364,7 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 					print_watchdog_node_info(tempNode);
 					authenticated = verify_authhash_for_node(tempNode, authkey);
 					ereport(DEBUG1,
-							(errmsg("ADD NODE MESSAGE from Hostname:\"%s\" PORT:%d pgpool_port:%d",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port)));
+							(errmsg("ADD NODE MESSAGE from hostname:\"%s\" PORT:%d pgpool_port:%d",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port)));
 					/* verify this node */
 					if (authenticated)
 					{
@@ -1311,7 +1392,7 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 						{
 							/* reply with node info message */
 							ereport(NOTICE,
-									(errmsg("New node joined the cluster Hostname:\"%s\" PORT:%d pgpool_port:%d",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port)));
+									(errmsg("New node joined the cluster hostname:\"%s\" port:%d pgpool_port:%d",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port)));
 
 							watchdog_state_machine(WD_EVENT_PACKET_RCV, wdNode, pkt);
 						}
@@ -1324,7 +1405,7 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 					else
 					{
 						ereport(NOTICE,
-								(errmsg("authentication failed for add node from Hostname:\"%s\" PORT:%d pgpool_port:%d",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port),
+								(errmsg("authentication failed for add node from hostname:\"%s\" port:%d pgpool_port:%d",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port),
 								 errdetail("make sure wd_authkey configuration is same on all nodes")));
 					}
 
@@ -1398,10 +1479,10 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 	{
 		g_cluster.ipc_command_socks = list_delete_int(g_cluster.ipc_command_socks,lfirst_int(lc));
 	}
-	
+
 	list_free(socks_to_del);
 	socks_to_del = NULL;
-	
+
 	if (count >= pending_fds_count)
 		return count;
 	
@@ -1431,7 +1512,6 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 	list_free(socks_to_del);
 	socks_to_del = NULL;
 
-
 	/* Finally check if something waits us on interface monitoring socket */
 	if (g_cluster.network_monitor_sock > 0 &&  FD_ISSET(g_cluster.network_monitor_sock, rmask))
 	{
@@ -1440,7 +1520,7 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 		if (read_interface_change_event(g_cluster.network_monitor_sock, &link_event, &deleted))
 		{
 			ereport(DEBUG1,
-					(errmsg("Network event received"),
+					(errmsg("network event received"),
 						errdetail("deleted = %s Link change event = %s",
 							deleted?"YES":"NO",
 							link_event?"YES":"NO")));
@@ -1464,22 +1544,25 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 	return count;
 }
 
-static bool write_ipc_command_with_result_data(WDIPCCommandData* IPCCommand, char type, char* data, int len)
+static bool write_ipc_command_with_result_data(WDIPCCommandData* ipcCommand, char type, char* data, int len)
 {
 	int send_len = 0;
-	if (socket_write(IPCCommand->issueing_sock, &type, 1) < 0)
+	if (socket_write(ipcCommand->issueing_sock, &type, 1) < 0)
+	{
 		return false;
-
+	}
 	if (len > 0)
 	{
 		send_len = htonl(len);
-		if (socket_write(IPCCommand->issueing_sock, &send_len, sizeof(int)) < 0)
+		if (socket_write(ipcCommand->issueing_sock, &send_len, sizeof(int)) < 0)
 			return false;
-		if (socket_write(IPCCommand->issueing_sock, data, len) < 0)
+		if (socket_write(ipcCommand->issueing_sock, data, len) < 0)
 			return false;
 	}
-	else if (socket_write(IPCCommand->issueing_sock, &send_len, sizeof(int)) < 0)
+	else if (socket_write(ipcCommand->issueing_sock, &send_len, sizeof(int)) < 0)
+	{
 		return false;
+	}
 	return true;
 }
 
@@ -1633,17 +1716,17 @@ static IPC_CMD_PREOCESS_RES process_IPC_command(WDIPCCommandData* ipcCommand)
 }
 
 
-static IPC_CMD_PREOCESS_RES process_IPC_nodeList_command(WDIPCCommandData* IPCCommand)
+static IPC_CMD_PREOCESS_RES process_IPC_nodeList_command(WDIPCCommandData* ipcCommand)
 {
 	/* get the json for node list */
 	JsonNode* jNode = NULL;
 	int NodeID = -1;
 	bool ret;
 
-	if (IPCCommand->data_len <= 0 || IPCCommand->data_buf == NULL)
+	if (ipcCommand->data_len <= 0 || ipcCommand->data_buf == NULL)
 		return IPC_CMD_ERROR;
 
-	json_value *root = json_parse(IPCCommand->data_buf,IPCCommand->data_len);
+	json_value *root = json_parse(ipcCommand->data_buf,ipcCommand->data_len);
 	/* The root node must be object */
 	if (root == NULL || root->type != json_object)
 	{
@@ -1661,7 +1744,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_nodeList_command(WDIPCCommandData* IPCCo
 
 	json_value_free(root);
 	jNode = get_node_list_json(NodeID);
-	ret = write_ipc_command_with_result_data(IPCCommand, WD_IPC_CMD_RESULT_OK,
+	ret = write_ipc_command_with_result_data(ipcCommand, WD_IPC_CMD_RESULT_OK,
 											 jw_get_json_string(jNode), jw_get_json_length(jNode) +1);
 	jw_destroy(jNode);
 	if (ret == false)
@@ -1669,17 +1752,17 @@ static IPC_CMD_PREOCESS_RES process_IPC_nodeList_command(WDIPCCommandData* IPCCo
 	return IPC_CMD_COMPLETE;
 }
 
-static IPC_CMD_PREOCESS_RES process_IPC_nodeStatusChange_command(WDIPCCommandData* IPCCommand)
+static IPC_CMD_PREOCESS_RES process_IPC_nodeStatusChange_command(WDIPCCommandData* ipcCommand)
 {
 	int nodeStatus;
 	int nodeID;
 	char *message;
 	bool ret;
 
-	if (IPCCommand->data_len <= 0 || IPCCommand->data_buf == NULL)
+	if (ipcCommand->data_len <= 0 || ipcCommand->data_buf == NULL)
 		return IPC_CMD_ERROR;
 
-	ret = parse_node_status_json(IPCCommand->data_buf, IPCCommand->data_len, &nodeID, &nodeStatus, &message);
+	ret = parse_node_status_json(ipcCommand->data_buf, ipcCommand->data_len, &nodeID, &nodeStatus, &message);
 	
 	if (ret == false)
 	{
@@ -1754,14 +1837,14 @@ static bool fire_node_status_event(int nodeID, int nodeStatus)
 
 
 
-static IPC_CMD_PREOCESS_RES process_IPC_replicate_variable(WDIPCCommandData* IPCCommand)
+static IPC_CMD_PREOCESS_RES process_IPC_replicate_variable(WDIPCCommandData* ipcCommand)
 {
 	char res_type = WD_IPC_CMD_RESULT_BAD;
 
 	if (get_local_node_state() == WD_STANDBY ||
 		get_local_node_state() == WD_COORDINATOR)
 	{
-		IPC_CMD_PREOCESS_RES execute_res = execute_replicate_command(IPCCommand);
+		IPC_CMD_PREOCESS_RES execute_res = execute_replicate_command(ipcCommand);
 
 		if (execute_res == IPC_CMD_COMPLETE)
 		{
@@ -1785,7 +1868,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_replicate_variable(WDIPCCommandData* IPC
 		res_type = WD_IPC_CMD_CLUSTER_IN_TRAN;
 	}
 
-	if (write_ipc_command_with_result_data(IPCCommand, res_type, NULL, 0))
+	if (write_ipc_command_with_result_data(ipcCommand, res_type, NULL, 0))
 	{
 		/*
 		 * This is the complete lifecycle of command.
@@ -1797,7 +1880,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_replicate_variable(WDIPCCommandData* IPC
 	return IPC_CMD_ERROR;
 }
 
-static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandData *IPCCommand)
+static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandData *ipcCommand)
 {
 	char res_type = WD_IPC_CMD_RESULT_BAD;
 	/*
@@ -1807,7 +1890,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandDat
 	ereport(LOG,
 			(errmsg("processing master node data request from IPC socket")));
 
-	IPCCommand->type = WD_GET_MASTER_DATA_REQUEST;
+	ipcCommand->type = WD_GET_MASTER_DATA_REQUEST;
 	if (get_local_node_state() == WD_STANDBY)
 	{
 		/* I am a standby node, Just forward the request to coordinator */
@@ -1816,9 +1899,10 @@ static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandDat
 		init_wd_packet(&wdPacket);
 		set_message_type(&wdPacket, WD_GET_MASTER_DATA_REQUEST);
 		set_next_commandID_in_message(&wdPacket);
-		set_message_data(&wdPacket, IPCCommand->data_buf , IPCCommand->data_len);
+		set_message_data(&wdPacket, ipcCommand->data_buf , ipcCommand->data_len);
 		/* save the command ID */
-		IPCCommand->internal_command_id = wdPacket.command_id;
+		ipcCommand->internal_command_id = wdPacket.command_id;
+		allocate_resultNodes_in_IPCCommand(ipcCommand);
 		if (send_message(g_cluster.masterNode, &wdPacket) <= 0)
 		{
 			ereport(LOG,
@@ -1829,6 +1913,20 @@ static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandDat
 		}
 		else
 		{
+			int i;
+			for (i=0; i< g_cluster.remoteNodeCount; i++)
+			{
+				WDCommandNodeResult* nodeResult = &ipcCommand->nodeResults[i];
+				if (nodeResult->wdNode == g_cluster.masterNode)
+				{
+					nodeResult->cmdState = COMMAND_STATE_SENT;
+				}
+				else
+				{
+					nodeResult->cmdState = COMMAND_STATE_DO_NOT_SEND;
+				}
+			}
+
 			/*
 			 * we need to wait for the result
 			 */
@@ -1849,7 +1947,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandDat
 		res_type = WD_IPC_CMD_CLUSTER_IN_TRAN;
 	}
 
-	if (write_ipc_command_with_result_data(IPCCommand, res_type, NULL, 0))
+	if (write_ipc_command_with_result_data(ipcCommand, res_type, NULL, 0))
 	{
 		/*
 		 * This is the complete lifecycle of command.
@@ -1861,7 +1959,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandDat
 
 }
 
-static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandData *IPCCommand)
+static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandData *ipcCommand)
 {
 	char res_type = WD_IPC_CMD_RESULT_BAD;
 	/*
@@ -1871,7 +1969,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandDat
 	ereport(LOG,
 		(errmsg("processing sync request from IPC socket")));
 
-	IPCCommand->type = WD_FAILOVER_CMD_SYNC_REQUEST;
+	ipcCommand->type = WD_FAILOVER_CMD_SYNC_REQUEST;
 	if (get_local_node_state() == WD_STANDBY)
 	{
 		/* I am a standby node, Just forward the request to coordinator */
@@ -1880,9 +1978,10 @@ static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandDat
 		init_wd_packet(&wdPacket);
 		set_message_type(&wdPacket, WD_FAILOVER_CMD_SYNC_REQUEST);
 		set_next_commandID_in_message(&wdPacket);
-		set_message_data(&wdPacket, IPCCommand->data_buf , IPCCommand->data_len);
+		set_message_data(&wdPacket, ipcCommand->data_buf , ipcCommand->data_len);
 		/* save the command ID */
-		IPCCommand->internal_command_id = wdPacket.command_id;
+		ipcCommand->internal_command_id = wdPacket.command_id;
+		allocate_resultNodes_in_IPCCommand(ipcCommand);
 		if (send_message(g_cluster.masterNode, &wdPacket) <= 0)
 		{
 			ereport(LOG,
@@ -1893,6 +1992,19 @@ static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandDat
 		}
 		else
 		{
+			int i;
+			for (i=0; i< g_cluster.remoteNodeCount; i++)
+			{
+				WDCommandNodeResult* nodeResult = &ipcCommand->nodeResults[i];
+				if (nodeResult->wdNode == g_cluster.masterNode)
+				{
+					nodeResult->cmdState = COMMAND_STATE_SENT;
+				}
+				else
+				{
+					nodeResult->cmdState = COMMAND_STATE_DO_NOT_SEND;
+				}
+			}
 			/*
 			 * we need to wait for the result
 			 */
@@ -1908,7 +2020,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandDat
 		/*
 		 * If I am coordinator, Just process the request locally
 		 */
-		process_failover_command_sync_requests(g_cluster.localNode, NULL, IPCCommand);
+		process_failover_command_sync_requests(g_cluster.localNode, NULL, ipcCommand);
 		return IPC_CMD_COMPLETE;
 	}
 	else /* we are not in any stable state at the moment */
@@ -1916,7 +2028,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandDat
 		res_type = WD_IPC_CMD_CLUSTER_IN_TRAN;
 	}
 
-	if (write_ipc_command_with_result_data(IPCCommand, res_type, NULL, 0))
+	if (write_ipc_command_with_result_data(ipcCommand, res_type, NULL, 0))
 	{
 		/*
 		 * This is the complete lifecycle of command.
@@ -3184,7 +3296,7 @@ static int standard_packet_processor(WatchdogNode* wdNode, WDPacketData* pkt)
 		case WD_IAM_COORDINATOR_MESSAGE:
 		{
 			/*
-			 * if the message is received from coordinator reply with infor,
+			 * if the message is received from coordinator reply with info,
 			 * otherwise reject
 			 */
 			if (g_cluster.masterNode != NULL && wdNode != g_cluster.masterNode)
@@ -3230,13 +3342,32 @@ static bool send_message_to_connection(SocketConnection* conn, WDPacketData *pkt
 
 static bool send_message_to_node(WatchdogNode* wdNode, WDPacketData *pkt)
 {
-	if (send_message_to_connection(&wdNode->client_socket,pkt) == true)
-		return true;
-	if (send_message_to_connection(&wdNode->server_socket,pkt) == true)
-		return true;
-	ereport(DEBUG1,
-			(errmsg("sending packet to node \"%s\" failed, closing connection", wdNode->nodeName)));
-	return false;
+	bool ret;
+
+	ret = send_message_to_connection(&wdNode->client_socket,pkt);
+
+	if (ret == false)
+	{
+		ret = send_message_to_connection(&wdNode->server_socket,pkt);
+	}
+	if (ret)
+	{
+		/* we only update the last sent time if reply for packet is expected */
+		switch (pkt->type) {
+			case WD_REPLICATE_VARIABLE_REQUEST:
+			case WD_FAILOVER_CMD_SYNC_REQUEST:
+				gettimeofday(&wdNode->last_sent_time, NULL);
+		break;
+			default:
+    break;
+		}
+	}
+	else
+	{
+		ereport(DEBUG1,
+				(errmsg("sending packet %c to node \"%s\" failed", pkt->type,wdNode->nodeName)));
+	}
+	return ret;
 }
 
 /*
@@ -3254,11 +3385,11 @@ static int send_message(WatchdogNode* wdNode, WDPacketData *pkt)
 			return 1;
 		return 0;
 	}
-	/* NULL means send to all nodes */
+	/* NULL means send to all reachable odes */
 	for (i=0; i< g_cluster.remoteNodeCount; i++)
 	{
 		wdNode = &(g_cluster.remoteNodes[i]);
-		if (send_message_to_node(wdNode,pkt))
+		if (is_node_reachable(wdNode) && send_message_to_node(wdNode,pkt))
 			count++;
 	}
 	return count;
@@ -3287,6 +3418,101 @@ static void service_internal_command(void)
 	}
 }
 
+static void node_lost_while_ipc_command(WatchdogNode* wdNode)
+{
+	List* ipcCommands_to_del = NIL;
+	ListCell *lc;
+	foreach(lc, g_cluster.ipc_commands)
+	{
+
+		WDIPCCommandData* ipcCommand = lfirst(lc);
+		if (ipcCommand)
+		{
+			if (ipcCommand->type == WD_REPLICATE_VARIABLE_REQUEST)
+			{
+				if (node_lost_during_replicate_command_processing(wdNode, ipcCommand) == IPC_CMD_COMPLETE)
+				{
+					ipcCommands_to_del = lappend(ipcCommands_to_del,ipcCommand);
+				}
+			}
+			else if (ipcCommand->type == WD_FAILOVER_CMD_SYNC_REQUEST ||
+					 ipcCommand->type == WD_GET_MASTER_DATA_REQUEST)
+			{
+				int i;
+				for (i=0; i< g_cluster.remoteNodeCount; i++)
+				{
+					WDCommandNodeResult* nodeResult = &ipcCommand->nodeResults[i];
+					if (nodeResult->wdNode == wdNode)
+					{
+						if (nodeResult->cmdState == COMMAND_STATE_SENT)
+						{
+							ereport(LOG,
+									(errmsg("remote node \"%s\" lost while ipc command was in progress ",wdNode->nodeName)));
+							/* since the node is down so we may never get the results back */
+							write_ipc_command_with_result_data(ipcCommand, WD_IPC_CMD_CLUSTER_IN_TRAN, NULL, 0);
+							ipcCommands_to_del = lappend(ipcCommands_to_del,ipcCommand);
+
+							break;
+						}
+					}
+				}
+			}
+		}
+	}
+	/* delete completed commands */
+	foreach(lc, ipcCommands_to_del)
+	{
+		WDIPCCommandData* ipcCommand = lfirst(lc);
+		cleanUpIPCCommand(ipcCommand);
+	}
+}
+
+
+
+static void service_ipc_commands(void)
+{
+	ListCell *lc;
+	foreach(lc, g_cluster.ipc_commands)
+	{
+		WDIPCCommandData* ipcCommand = lfirst(lc);
+
+		if (ipcCommand && ipcCommand->sendTo_error_count)
+		{
+			int i;
+			for (i=0; i< g_cluster.remoteNodeCount; i++)
+			{
+				WDCommandNodeResult* nodeResult = &ipcCommand->nodeResults[i];
+				if (nodeResult->cmdState == COMMAND_STATE_SEND_ERROR)
+				{
+					if (is_node_reachable(nodeResult->wdNode))
+					{
+						WDPacketData wdPacket;
+
+						ereport(LOG,
+								(errmsg("remote node \"%s\" is reachable again, resending the command packet ",nodeResult->wdNode->nodeName)));
+
+						init_wd_packet(&wdPacket);
+						set_message_type(&wdPacket, ipcCommand->type);
+						set_message_commandID(&wdPacket,ipcCommand->internal_command_id);
+						set_message_data(&wdPacket,ipcCommand->data_buf, ipcCommand->data_len);
+
+						if (send_message_to_node(nodeResult->wdNode, &wdPacket) == true)
+						{
+							nodeResult->cmdState = COMMAND_STATE_SENT;
+							ipcCommand->sendTo_error_count--;
+							ipcCommand->sendTo_count++;
+							if (ipcCommand->sendTo_error_count == 0)
+								break;
+						}
+					}
+				}
+			}
+		}
+	}
+}
+
+
+
 static bool watchdog_internal_command_packet_processor(WatchdogNode* wdNode, WDPacketData* pkt)
 {
 	int i;
@@ -3332,7 +3558,6 @@ static bool watchdog_internal_command_packet_processor(WatchdogNode* wdNode, WDP
 
 	if (g_cluster.currentCommand.commandReplyFromCount >= g_cluster.currentCommand.commandSendToCount)
 	{
-		g_cluster.currentCommand.commandFinished = true;
 		if (pkt->type == WD_REJECT_MESSAGE || pkt->type == WD_ERROR_MESSAGE)
 			g_cluster.currentCommand.commandStatus = COMMAND_FINISHED_NODE_REJECTED;
 		else
@@ -3342,7 +3567,6 @@ static bool watchdog_internal_command_packet_processor(WatchdogNode* wdNode, WDP
 	else if (pkt->type == WD_REJECT_MESSAGE || pkt->type == WD_ERROR_MESSAGE)
 	{
 		/* Error or reject message by any node imidiately finishes the command */
-		g_cluster.currentCommand.commandFinished = true;
 		g_cluster.currentCommand.commandStatus = COMMAND_FINISHED_NODE_REJECTED;
 		watchdog_state_machine(WD_EVENT_COMMAND_FINISHED, wdNode, pkt);
 	}
@@ -3350,65 +3574,26 @@ static bool watchdog_internal_command_packet_processor(WatchdogNode* wdNode, WDP
 	return true; /* do not process this packet further */
 }
 
+static bool is_cluster_command_in_progress(void)
+{
+	return g_cluster.currentCommand.commandStatus == COMMAND_IN_PROGRESS;
+}
 
 static void check_for_current_command_timeout(void)
 {
 	struct timeval currTime;
-	if (g_cluster.currentCommand.commandStatus != COMMAND_IN_PROGRESS ||
-		g_cluster.currentCommand.commandFinished != 0)
+
+	if (!is_cluster_command_in_progress())
 		return;
 
 	gettimeofday(&currTime,NULL);
 	if (WD_TIME_DIFF_SEC(currTime,g_cluster.currentCommand.commandTime) >=  g_cluster.currentCommand.commandTimeoutSecs)
 	{
-		g_cluster.currentCommand.commandFinished = true;
 		g_cluster.currentCommand.commandStatus = COMMAND_FINISHED_TIMEOUT;
 		watchdog_state_machine(WD_EVENT_COMMAND_FINISHED, NULL, NULL);
 	}
 }
 
-static char get_current_command_resultant_message_type(void)
-{
-	char res = WD_ACCEPT_MESSAGE;
-	int i;
-	if (g_cluster.currentCommand.commandFinished == 0)
-		return WD_NO_MESSAGE;
-	if (g_cluster.currentCommand.sendToNode == NULL)
-	{
-		/* The command was for all nodes */
-		for (i = 0; i< g_cluster.remoteNodeCount; i++)
-		{
-			WDCommandNodeResult* nodeRes = &g_cluster.currentCommand.nodeResults[i];
-			if (nodeRes->cmdState != COMMAND_STATE_REPLIED)
-				continue;
-			if (nodeRes->result_type != WD_ACCEPT_MESSAGE && nodeRes->result_type != WD_INFO_MESSAGE)
-			{
-				/* failed */
-				if (res != WD_ERROR_MESSAGE)
-					res = nodeRes->result_type;
-			}
-		}
-	}
-	else
-	{
-		if (g_cluster.currentCommand.commandSendToCount == 0) /* We failed to send to any node */
-			return WD_ERROR_MESSAGE;
-		if (g_cluster.currentCommand.commandReplyFromCount == 0) /* We got no reply */
-			return WD_ERROR_MESSAGE;
-		for (i = 0; i< g_cluster.remoteNodeCount; i++)
-		{
-			WDCommandNodeResult* nodeRes = &g_cluster.currentCommand.nodeResults[i];
-			if (nodeRes->wdNode == g_cluster.currentCommand.sendToNode)
-			{
-				if (nodeRes->cmdState != COMMAND_STATE_REPLIED)
-					return WD_ERROR_MESSAGE;
-				return nodeRes->result_type;
-			}
-		}
-		return WD_ERROR_MESSAGE;
-	}
-	return res;
-}
 
 static void clear_current_command(void)
 {
@@ -3458,7 +3643,7 @@ static int issue_watchdog_internal_command(WatchdogNode* wdNode, WDPacketData *p
 				if (send_message_to_node(nodeResult->wdNode, pkt) == false)
 				{
 					ereport(DEBUG1,
-							(errmsg("failed to send watchdog internla command packet %s",nodeResult->wdNode->nodeName),
+							(errmsg("failed to send watchdog internal command packet %s",nodeResult->wdNode->nodeName),
 							 errdetail("saving the packet. will try to resend it if connection recovers")));
 
 					/* failed to send. May be try again later */
@@ -3506,7 +3691,6 @@ static int issue_watchdog_internal_command(WatchdogNode* wdNode, WDPacketData *p
 		memcpy(g_cluster.currentCommand.packet.data,pkt->data,pkt->len);
 		g_cluster.currentCommand.packet.len = pkt->len;
 	}
-	g_cluster.currentCommand.commandFinished = false;
 	return g_cluster.currentCommand.commandSendToCount;
 }
 
@@ -3535,7 +3719,7 @@ static bool service_lost_connections(void)
 	for (i = 0; i< g_cluster.remoteNodeCount; i++)
 	{
 		WatchdogNode* wdNode = &(g_cluster.remoteNodes[i]);
-		if (wdNode->state == WD_SHUTDOWN)
+		if (wdNode->state == WD_SHUTDOWN || wdNode->state == WD_DEAD)
 			continue;
 
 		if (is_socket_connection_connected(&wdNode->client_socket) == false)
@@ -3553,7 +3737,6 @@ static bool service_lost_connections(void)
 					watchdog_state_machine(WD_EVENT_NEW_OUTBOUND_CONNECTION, wdNode, NULL);
 					ret = true;
 				}
-
 			}
 		}
 	}
@@ -3627,6 +3810,12 @@ static int send_cluster_command(WatchdogNode* wdNode, char type, int timeout_sec
 	WDPacketData *pkt = get_message_of_type(type);
 	if (pkt)
 	{
+		/* If the old command is in progress and we are about to issue a new command
+		 * finish the old command with timeout
+		 */
+		if (is_cluster_command_in_progress())
+			watchdog_state_machine(WD_EVENT_COMMAND_FINISHED, NULL, NULL);
+
 		ret = issue_watchdog_internal_command(wdNode, pkt, timeout_sec);
 		free_packet(pkt);
 	}
@@ -3834,6 +4023,10 @@ static int watchdog_state_machine(WD_EVENTS event, WatchdogNode* wdNode, WDPacke
 	
 	if (event == WD_EVENT_REMOTE_NODE_LOST)
 	{
+		/* close all socket connections to the node */
+		close_socket_connection(&wdNode->client_socket);
+		close_socket_connection(&wdNode->server_socket);
+
 		if (wdNode->state == WD_SHUTDOWN)
 		{
 			ereport(LOG,
@@ -3851,6 +4044,10 @@ static int watchdog_state_machine(WD_EVENTS event, WatchdogNode* wdNode, WDPacke
 					(errmsg("watchdog cluster has lost the coordinator node")));
 			g_cluster.masterNode = NULL;
 		}
+		/* clear the wait timer on the node */
+		wdNode->last_sent_time.tv_sec = 0;
+		wdNode->last_sent_time.tv_usec = 0;
+		node_lost_while_ipc_command(wdNode);
 	}
 	else if (event == WD_EVENT_PACKET_RCV)
 	{
@@ -3999,7 +4196,7 @@ static int watchdog_state_machine_loading(WD_EVENTS event, WatchdogNode* wdNode,
 				}
 			}
 			free_packet(addPkt);
-			set_timeout(4);
+			set_timeout(MAX_SECS_WAIT_FOR_REPLY_FROM_NODE);
 		}
 			break;
 
@@ -4086,8 +4283,9 @@ static int watchdog_state_machine_joining(WD_EVENTS event, WatchdogNode* wdNode,
 	{
 		case WD_EVENT_WD_STATE_CHANGED:
 			g_cluster.masterNode = NULL;
+			try_connecting_with_all_unreachable_nodes();
 			send_cluster_command(NULL, WD_REQ_INFO_MESSAGE, 5);
-			set_timeout(5);
+			set_timeout(MAX_SECS_WAIT_FOR_REPLY_FROM_NODE);
 			break;
 
 		case WD_EVENT_TIMEOUT:
@@ -4215,7 +4413,7 @@ static int watchdog_state_machine_initializing(WD_EVENTS event, WatchdogNode* wd
 					if (wdNode->state == WD_ADD_MESSAGE_SENT)
 						ereport(FATAL,
 							(return_code(POOL_EXIT_FATAL),
-								errmsg("Add to watchdog cluster request is rejected by node \"%s:%d\"",wdNode->hostname,wdNode->wd_port),
+								errmsg("add to watchdog cluster request is rejected by node \"%s:%d\"",wdNode->hostname,wdNode->wd_port),
 								 errhint("check the watchdog configurations.")));
 					break;
 				default:
@@ -4237,9 +4435,9 @@ static int watchdog_state_machine_standForCord(WD_EVENTS event, WatchdogNode* wd
 	switch (event)
 	{
 		case WD_EVENT_WD_STATE_CHANGED:
-			send_cluster_command(NULL, WD_STAND_FOR_COORDINATOR_MESSAGE, 5);
+			send_cluster_command(NULL, WD_STAND_FOR_COORDINATOR_MESSAGE, 4);
 			/* wait for 5 seconds if someone rejects us*/
-			set_timeout(5);
+			set_timeout(MAX_SECS_WAIT_FOR_REPLY_FROM_NODE);
 			break;
 
 		case WD_EVENT_COMMAND_FINISHED:
@@ -4358,8 +4556,8 @@ static int watchdog_state_machine_coordinator(WD_EVENTS event, WatchdogNode* wdN
 		case WD_EVENT_WD_STATE_CHANGED:
 		{
 			int i;
-			send_cluster_command(NULL, WD_DECLARE_COORDINATOR_MESSAGE, 5);
-			set_timeout(10);
+			send_cluster_command(NULL, WD_DECLARE_COORDINATOR_MESSAGE, 4);
+			set_timeout(MAX_SECS_WAIT_FOR_REPLY_FROM_NODE);
 			ereport(LOG,
 					(errmsg("I am announcing my self as master/coordinator watchdog node")));
 
@@ -4510,8 +4708,15 @@ static int watchdog_state_machine_coordinator(WD_EVENTS event, WatchdogNode* wdN
 			break;
 
 		case WD_EVENT_TIMEOUT:
-			send_cluster_command(NULL, WD_IAM_COORDINATOR_MESSAGE, BEACON_MESSAGE_INTERVAL_SECONDS);
-			set_timeout(BEACON_MESSAGE_INTERVAL_SECONDS);
+			if (is_cluster_command_in_progress())
+			{
+				set_timeout(MAX_SECS_WAIT_FOR_REPLY_FROM_NODE);
+			}
+			else
+			{
+				send_cluster_command(NULL, WD_IAM_COORDINATOR_MESSAGE, 5);
+				set_timeout(BEACON_MESSAGE_INTERVAL_SECONDS);
+			}
 			break;
 
 		case WD_EVENT_REMOTE_NODE_LOST:
@@ -4785,7 +4990,7 @@ static int watchdog_state_machine_voting(WD_EVENTS event, WatchdogNode* wdNode,
 	{
 		case WD_EVENT_WD_STATE_CHANGED:
 			clear_current_command();
-			set_timeout(6);
+			set_timeout(MAX_SECS_WAIT_FOR_REPLY_FROM_NODE);
 			break;
 			
 		case WD_EVENT_TIMEOUT:
@@ -5103,8 +5308,12 @@ static IPC_CMD_PREOCESS_RES execute_replicate_command(WDIPCCommandData* ipcComma
 {
 	int i;
 	IPC_CMD_PREOCESS_RES res;
-	
 	WDPacketData wdPacket;
+
+	/* save the hassel if I am the only alive node */
+	if (get_cluster_node_count() == 0)
+		return IPC_CMD_COMPLETE;
+
 	init_wd_packet(&wdPacket);
 	set_message_type(&wdPacket, WD_REPLICATE_VARIABLE_REQUEST);
 	set_next_commandID_in_message(&wdPacket);
@@ -5112,21 +5321,38 @@ static IPC_CMD_PREOCESS_RES execute_replicate_command(WDIPCCommandData* ipcComma
 	
 	allocate_resultNodes_in_IPCCommand(ipcCommand);
 	ipcCommand->sendTo_count = 0;
+	ipcCommand->sendTo_error_count = 0;
 	ipcCommand->reply_from_count = 0;
 	ipcCommand->internal_command_id = wdPacket.command_id;
 	ipcCommand->type = wdPacket.type;
-	
+
 	for (i=0; i< g_cluster.remoteNodeCount; i++)
 	{
 		WDCommandNodeResult* nodeResult = &ipcCommand->nodeResults[i];
-		if (send_message_to_node(nodeResult->wdNode, &wdPacket) == true)
+		if (nodeResult->wdNode->state == WD_DEAD || nodeResult->wdNode->state == WD_LOST || nodeResult->wdNode->state == WD_SHUTDOWN)
+		{
+			ereport(DEBUG2,
+					(errmsg("not sending ipc command packet to DEAD node \"%s\"",nodeResult->wdNode->nodeName)));
+			/* Do not send to dead node */
+			nodeResult->cmdState = COMMAND_STATE_DO_NOT_SEND;
+		}
+		else if (send_message_to_node(nodeResult->wdNode, &wdPacket) == true)
 		{
 			nodeResult->cmdState = COMMAND_STATE_SENT;
 			ipcCommand->sendTo_count++;
 		}
 		else
+		{
 			nodeResult->cmdState = COMMAND_STATE_SEND_ERROR;
+			ipcCommand->sendTo_error_count++;
+		}
 	}
+	/* Our first priority is to send to all alive connected nodes
+	 * so if send error count is not zero return IPC_PROCESSING and retry
+	 * sending later in the cycle
+	 */
+	return IPC_CMD_PROCESSING;
+
 	/*
 	 * The current quorum status of standby node can be out of sync,
 	 * so update it before making any decision on command success
@@ -5306,6 +5532,56 @@ static bool process_wd_command_function(WatchdogNode* wdNode, WDPacketData* pkt,
 	return true;
 }
 
+static IPC_CMD_PREOCESS_RES node_lost_during_replicate_command_processing(WatchdogNode* wdNode, WDIPCCommandData* ipcCommand)
+{
+	int i;
+	/* this command is needed to be sent to all connected nodes */
+	for (i=0; i< g_cluster.remoteNodeCount; i++)
+	{
+		WDCommandNodeResult* nodeResult = &ipcCommand->nodeResults[i];
+		if (nodeResult->wdNode != wdNode)
+			continue;
+
+		/* if the command state is sent and the node that is lost
+		 * so we will never get the reply back from that node
+		 */
+		if (nodeResult->cmdState == COMMAND_STATE_SENT)
+		{
+			nodeResult->cmdState = COMMAND_STATE_DO_NOT_SEND;
+			ereport(WARNING,
+					(errmsg("watchdog node \"%s\" was lost during processing of pgpool-II replicate command",wdNode->nodeName)));
+			if (ipcCommand->sendTo_count)
+				ipcCommand->sendTo_count--;
+		}
+		break;
+	}
+
+	if (ipcCommand->reply_from_count >= ipcCommand->sendTo_count)
+	{
+		/*
+		 * we have received results from all nodes
+		 * analyze the result
+		 */
+		char res_type = WD_IPC_CMD_RESULT_OK;
+
+		for (i=0; i< g_cluster.remoteNodeCount; i++)
+		{
+			WDCommandNodeResult* nodeResult = &ipcCommand->nodeResults[i];
+			if (nodeResult->cmdState == COMMAND_STATE_REPLIED &&
+				nodeResult->result_type != WD_ACCEPT_MESSAGE)
+			{
+				res_type = WD_IPC_CMD_RESULT_BAD;
+				break;
+			}
+		}
+		if (write_ipc_command_with_result_data(ipcCommand, res_type, NULL, 0) == false)
+			ereport(LOG,
+					(errmsg("failed to forward replicate command results to IPC command socket")));
+
+		return IPC_CMD_COMPLETE;
+	}
+	return IPC_CMD_PROCESSING;
+}
 
 static bool reply_is_received_for_pgpool_replicate_command(WatchdogNode* wdNode, WDPacketData* pkt, WDIPCCommandData* ipcCommand)
 {
@@ -5328,13 +5604,26 @@ static bool reply_is_received_for_pgpool_replicate_command(WatchdogNode* wdNode,
 				(errmsg("unable to find result node for pgpool-II replicate command packet received from watchdog node \"%s\"",wdNode->nodeName)));
 		return true;
 	}
+
 	nodeResult->result_type = pkt->type;
 	nodeResult->cmdState = COMMAND_STATE_REPLIED;
 	ipcCommand->reply_from_count++;
+
 	ereport(DEBUG2,
 		(errmsg("watchdog node \"%s\" has replied for pgpool-II replicate command packet",wdNode->nodeName),
 			 errdetail("command was sent to %d nodes and %d nodes have replied to it",ipcCommand->sendTo_count,ipcCommand->reply_from_count)));
 
+	/* reject message from any node finishes the command */
+	if (pkt->type != WD_ACCEPT_MESSAGE)
+	{
+
+		if (write_ipc_command_with_result_data(ipcCommand, WD_IPC_CMD_RESULT_BAD, NULL, 0) == false)
+			ereport(LOG,
+					(errmsg("failed to forward message to IPC command socket")));
+		cleanUpIPCCommand(ipcCommand);
+		return true;
+	}
+
 	if (ipcCommand->reply_from_count >= ipcCommand->sendTo_count)
 	{
 		/*
diff --git a/src/watchdog/wd_commands.c b/src/watchdog/wd_commands.c
index 0133267..851c156 100644
--- a/src/watchdog/wd_commands.c
+++ b/src/watchdog/wd_commands.c
@@ -214,7 +214,7 @@ issue_command_to_watchdog(char type, int timeout_sec, char* data, int data_len,
 				if (socket_read(sock, &res_type, 1 ,0) <=0)
 				{
 					ereport(WARNING,
-						(errmsg("error reading from IPC command socket"),
+						(errmsg("error reading from IPC command socket for ipc command %c",type),
 							 errdetail("read from socket failed with error \"%s\"",strerror(errno))));
 					close(sock);
 					return result;
@@ -223,7 +223,7 @@ issue_command_to_watchdog(char type, int timeout_sec, char* data, int data_len,
 				if (socket_read(sock, &res_length, sizeof(int), 0) <= 0)
 				{
 					ereport(WARNING,
-						(errmsg("error reading from IPC command socket"),
+						(errmsg("error reading from IPC command socket for ipc command %c",type),
 							 errdetail("read from socket failed with error \"%s\"",strerror(errno))));
 					close(sock);
 					return result;
@@ -242,7 +242,7 @@ issue_command_to_watchdog(char type, int timeout_sec, char* data, int data_len,
 						pfree(result->data);
 						pfree(result);
 						ereport(DEBUG1,
-							(errmsg("error reading from IPC command socket"),
+							(errmsg("error reading from IPC command socket for ipc command %c",type),
 								 errdetail("read from socket failed with error \"%s\"",strerror(errno))));
 						close(sock);
 						return NULL;
@@ -509,25 +509,34 @@ wd_send_failover_sync_command(char* syncReqType, enum WDFailoverLocks lockID)
 
 	pfree(json_data);
 
-	if (result == NULL || result->length <= 0)
+	if (result == NULL)
 	{
 		ereport(WARNING,
-			(errmsg("watchdog failed to send failover command"),
+			(errmsg("watchdog failover interlocking command failed"),
 				 errdetail("issue command to watchdog returned NULL")));
 		return FAILOVER_RES_ERROR;
 	}
 	else if (result->type == WD_IPC_CMD_TIMEOUT)
 	{
 		ereport(WARNING,
-			(errmsg("watchdog failed to send failover command"),
+			(errmsg("watchdog failover interlocking command failed"),
 				 errdetail("ipc command timeout")));
 		FreeCmdResult(result);
 		return FAILOVER_RES_ERROR;
 	}
+	if (result->type == WD_IPC_CMD_CLUSTER_IN_TRAN)
+	{
+		ereport(WARNING,
+				(errmsg("watchdog failover interlocking command failed"),
+				 errdetail("watchdog cluster is not in stable state"),
+					errhint("try again when the cluster is fully initialized")));
+		return FAILOVER_RES_TRANSITION;
+	}
+
 	if (result->length <= 0)
 	{
 		ereport(WARNING,
-			(errmsg("watchdog failed to send failover command"),
+			(errmsg("watchdog failover interlocking command failed"),
 				 errdetail("issue command to watchdog returned no data")));
 		FreeCmdResult(result);
 		return FAILOVER_RES_ERROR;
@@ -577,7 +586,7 @@ wd_degenerate_backend_set(int *node_id_set, int count)
 													   WD_DEFAULT_IPC_COMMAND_TIMEOUT,
 													   func, strlen(func), true);
 	pfree(func);
-	
+
 	if (result == NULL)
 	{
 		ereport(WARNING,
@@ -623,7 +632,7 @@ wd_promote_backend(int node_id)
 	/* if promote packet is received already, do nothing */
 	if (wd_chk_node_mask_for_promote_req(&n,1))
 		return COMMAND_OK;
-	
+
 	func = get_wd_node_function_json(WD_FUNCTION_PROMOTE_REQUEST,&n, 1,
 									 shared_key?*shared_key:0,pool_config->wd_authkey);
 	result = issue_command_to_watchdog(WD_FUNCTION_COMMAND ,
@@ -822,12 +831,12 @@ static WDFailoverCMDResults wd_issue_failover_lock_command(char* syncReqType, en
 {
 	WDFailoverCMDResults res;
 	int x;
-	for (x=0; x < MAX_SEC_WAIT_FOR_CLUSTER_TRANSATION; x++)
+	for (x=0; x <= MAX_SEC_WAIT_FOR_CLUSTER_TRANSATION/2; x++)
 	{
 		res = wd_send_failover_sync_command(syncReqType, lockID);
 		if (res != FAILOVER_RES_TRANSITION)
 			break;
-		sleep(1);
+		sleep(2);
 	}
 	return res;
 }

supp_k

2016-09-30 00:11

reporter   ~0001111

Hi,

the patch doesn't solve the problem. The failover_command is still triggered by both survived nodes nodes. In my scenario the Pgpool master node was killed.

See the logs:

Server 1:
LOG: server socket of Linux_srv-2181113.aqa.int.zone_5432 is closed
LOG: remote node "Linux_srv-2181113.aqa.int.zone_5432" is not reachable
DETAIL: marking the node as lost
LOG: remote node "Linux_srv-2181113.aqa.int.zone_5432" is lost
LOG: watchdog cluster has lost the coordinator node
LOG: watchdog node state changed from [STANDBY] to [JOINING]
LOG: watchdog node state changed from [JOINING] to [INITIALIZING]
LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER]
LOG: watchdog node state changed from [STANDING FOR MASTER] to [PARTICIPATING IN ELECTION]
LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [INITIALIZING]
LOG: watchdog node state changed from [INITIALIZING] to [STANDBY]
LOG: successfully joined the watchdog cluster as standby node
DETAIL: our join coordinator request is accepted by cluster leader node "Linux_srv-2181107.aqa.int.zon
e_5432"
LOG: failed to connect to PostgreSQL server on "b.db.node:15432" using INET socket
DETAIL: select() system call failed with an error "Interrupted system call"
ERROR: failed to make persistent db connection
DETAIL: connection to host:"b.db.node:15432" failed
LOG: received degenerate backend request for node_id: 1 from pid [13113]
LOG: connect_inet_domain_socket: select() interrupted by certain signal. retrying...
LOG: failed to connect to PostgreSQL server on "b.db.node:15432" using INET socket
DETAIL: select() system call failed with an error "Interrupted system call"
ERROR: failed to make persistent db connection
DETAIL: connection to host:"b.db.node:15432" failed
LOG: setting backend node 1 status to NODE DOWN
LOG: received degenerate backend request for node_id: 1 from pid [13111]
LOG: new IPC connection received
LOG: new IPC connection received
LOG: processing sync request from IPC socket
LOG: sync request from IPC socket is forwarded to master watchdog node "Linux_srv-2181107.aqa.int.zone_5432"
DETAIL: waiting for the reply from master node...
LOG: starting degeneration. shutdown host b.db.node(15432)
LOG: Restart all children
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: execute command: /etc/pgpool-II/failover.sh 1 b.db.node 15432 /var/lib/pgsql/9.5/data 0 a.db.node 0 0 15432 /var/lib/pgsql/9.5/data
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3



Server 2:
LOG: read from socket failed, remote end closed the connection
LOG: server socket of Linux_srv-2181113.aqa.int.zone_5432 is closed
LOG: remote node "Linux_srv-2181113.aqa.int.zone_5432" is not reachable
DETAIL: marking the node as lost
LOG: remote node "Linux_srv-2181113.aqa.int.zone_5432" is lost
LOG: watchdog cluster has lost the coordinator node
LOG: watchdog node state changed from [STANDBY] to [JOINING]
LOG: watchdog node state changed from [JOINING] to [INITIALIZING]
LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER]
LOG: watchdog node state changed from [STANDING FOR MASTER] to [MASTER]
LOG: I am announcing my self as master/coordinator watchdog node
LOG: I am the cluster leader node
DETAIL: our declare coordinator message is accepted by all nodes
LOG: I am the cluster leader node. Starting escalation process
LOG: escalation process started with PID:9137
LOG: watchdog: escalation started
WARNING: interface is ignored: Operation not permitted
LOG: failback event detected
DETAIL: restarting myself
LOG: selecting backend connection
DETAIL: failback event detected, discarding existing connections
LOG: child process with pid: 8806 exits with status 256
LOG: fork a new child process with pid: 9141
LOG: watchdog bringing up delegate IP, 'if_up_cmd' succeeded
LOG: watchdog escalation process with pid: 9137 exit with SUCCESS.
LOG: failed to connect to PostgreSQL server on "b.db.node:15432" using INET socket
DETAIL: select() system call failed with an error "Interrupted system call"
ERROR: failed to make persistent db connection
DETAIL: connection to host:"b.db.node:15432" failed
LOG: trying connecting to PostgreSQL server on "b.db.node:15432" by INET socket
DETAIL: timed out. retrying...
LOG: failed to connect to PostgreSQL server on "b.db.node:15432" using INET socket
DETAIL: select() system call failed with an error "Interrupted system call"
ERROR: failed to make persistent db connection
DETAIL: connection to host:"b.db.node:15432" failed
LOG: setting backend node 1 status to NODE DOWN
LOG: received degenerate backend request for node_id: 1 from pid [8799]
LOG: new IPC connection received
LOG: new IPC connection received
LOG: processing sync request from IPC socket
LOG: local pgpool-II node "Linux_srv-2181107.aqa.int.zone_5432" is requesting to become a lock holder
LOG: local pgpool-II node "Linux_srv-2181107.aqa.int.zone_5432" is the lock holder
LOG: starting degeneration. shutdown host b.db.node(15432)
LOG: Restart all children
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: execute command: /etc/pgpool-II/failover.sh 1 b.db.node 15432 /var/lib/pgsql/9.5/data 0 a.db.node
 0 0 15432 /var/lib/pgsql/9.5/data
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3
LOG: child process received shutdown request signal 3

z0rb1n0

2016-09-30 01:24

reporter   ~0001112

@supp_k: are failover_command's %d and %P set to the same values in both pgpools? You may be experiencing a double-detection problem.

Speaking of which, is there any particular reason all the pgpools perform the health checks to begin with, and not just the current watchdog master?

supp_k

2016-09-30 16:23

reporter   ~0001114

z0rb1n0,


yes the commands are identical. It can be seen from the log information in my previous message. Identical commands are triggered.

supp_k

2016-10-05 20:26

reporter   ~0001116

Guys,

do you need any additional information? When will you be able to fix the issue?

gabrimonfa

2016-10-10 20:00

reporter   ~0001117

I think this is a quite important issue.
When will you be able to fix the issue?

supp_k

2016-10-13 18:07

reporter   ~0001118

Guys,

kindly ask you to review this problem. Without this issue fixed we can't go live and even automatic tests failing. Full verification of our solution which works on the basis of Pgpool can't be performed.

If you need any assistance or help with testing then we are ready and want to help you with it.


Also we would kindly ask you to refresh the yum repository version, because our customers need the fresh official build.


Best Regards,
Sergey.

Muhammad Usama

2016-11-01 00:33

developer  

wd_rewamp_failover.diff (159,721 bytes)
diff --git a/src/include/pool.h b/src/include/pool.h
index 65b3c92..e2a9cdb 100644
--- a/src/include/pool.h
+++ b/src/include/pool.h
@@ -373,7 +373,7 @@ extern int my_master_node_id;
 #define ACCEPT_FD_SEM			5
 #define MAX_REQUEST_QUEUE_SIZE	10
 
-#define MAX_SEC_WAIT_FOR_CLUSTER_TRANSATION 6 /* time in seconds to keep retrying for a
+#define MAX_SEC_WAIT_FOR_CLUSTER_TRANSATION 10 /* time in seconds to keep retrying for a
 											   * watchdog command if the cluster is not
 											   * in stable state */
 
@@ -406,6 +406,7 @@ typedef enum {
 typedef struct {
 	POOL_REQUEST_KIND	kind;		/* request kind */
 	unsigned char request_details;	/* option flags kind */
+	unsigned int wd_failover_id;	/* watchdog ID for this failover operation */
 	int node_id[MAX_NUM_BACKENDS];	/* request node id */
 	int count;						/* request node ids count */
 }POOL_REQUEST_NODE;
@@ -512,7 +513,7 @@ extern char remote_port[];	/* client port */
 /*
  * public functions
  */
-extern bool register_node_operation_request(POOL_REQUEST_KIND kind, int* node_id_set, int count, bool switch_over);
+extern bool register_node_operation_request(POOL_REQUEST_KIND kind, int* node_id_set, int count, bool switch_over, unsigned int wd_failover_id);
 extern char *get_config_file_name(void);
 extern char *get_hba_file_name(void);
 extern void do_child(int *fds);
@@ -543,10 +544,10 @@ extern void NoticeResponse(POOL_CONNECTION *frontend,
 								  POOL_CONNECTION_POOL *backend);
 
 extern void notice_backend_error(int node_id, bool switch_over);
-extern void degenerate_backend_set(int *node_id_set, int count, bool switch_over);
-extern bool degenerate_backend_set_ex(int *node_id_set, int count, bool error, bool test_only, bool switch_over);
-extern void promote_backend(int node_id);
-extern void send_failback_request(int node_id, bool throw_error);
+extern void degenerate_backend_set(int *node_id_set, int count, bool switch_over, unsigned int wd_failover_id);
+extern bool degenerate_backend_set_ex(int *node_id_set, int count, bool error, bool test_only, bool switch_over, unsigned int wd_failover_id);
+extern void promote_backend(int node_id, unsigned int wd_failover_id);
+extern void send_failback_request(int node_id, bool throw_error, unsigned int wd_failover_id);
 
 
 extern void pool_set_timeout(int timeoutval);
diff --git a/src/include/watchdog/watchdog.h b/src/include/watchdog/watchdog.h
index e7c9770..8699a4c 100644
--- a/src/include/watchdog/watchdog.h
+++ b/src/include/watchdog/watchdog.h
@@ -103,6 +103,9 @@ typedef struct WatchdogNode
 	struct timeval last_rcv_time;			/* timestamp when last packet
 											 * was received from the node
 											 */
+	struct timeval last_sent_time;			/* timestamp when last packet
+											 * was sent on the node
+											 */
 	char nodeName[WD_MAX_HOST_NAMELEN];		/* name of this node */
 	char hostname[WD_MAX_HOST_NAMELEN];		/* host name */
 	int wd_port;							/* watchdog port */
diff --git a/src/include/watchdog/wd_ipc_commands.h b/src/include/watchdog/wd_ipc_commands.h
index d5dd148..259ddce 100644
--- a/src/include/watchdog/wd_ipc_commands.h
+++ b/src/include/watchdog/wd_ipc_commands.h
@@ -7,7 +7,7 @@
  * pgpool: a language independent connection pool server for PostgreSQL
  * written by Tatsuo Ishii
  *
- * Copyright (c) 2003-2015	PgPool Global Development Group
+ * Copyright (c) 2003-2016	PgPool Global Development Group
  *
  * Permission to use, copy, modify, and distribute this software and
  * its documentation for any purpose and without fee is hereby
@@ -50,19 +50,11 @@ extern char* get_watchdog_ipc_address(void);
 extern unsigned int* get_ipc_shared_key(void);
 
 
-extern int wd_set_node_mask_for_failback_req(int *node_id_set, int count);
-extern int wd_set_node_mask_for_degenerate_req(int *node_id_set, int count);
-extern int wd_set_node_mask_for_promote_req(int *node_id_set, int count);
-extern int wd_chk_node_mask_for_failback_req(int *node_id_set, int count);
-extern int wd_chk_node_mask_for_degenerate_req(int *node_id_set, int count);
-extern int wd_chk_node_mask_for_promote_req(int *node_id_set, int count);
-
-
 extern WdCommandResult wd_start_recovery(void);
 extern WdCommandResult wd_end_recovery(void);
-extern WdCommandResult wd_send_failback_request(int node_id);
-extern WdCommandResult wd_degenerate_backend_set(int *node_id_set, int count);
-extern WdCommandResult wd_promote_backend(int node_id);
+extern WDFailoverCMDResults wd_send_failback_request(int node_id, unsigned int *wd_failover_id);
+extern WDFailoverCMDResults wd_degenerate_backend_set(int *node_id_set, int count, unsigned int *wd_failover_id);
+extern WDFailoverCMDResults wd_promote_backend(int node_id, unsigned int *wd_failover_id);
 
 extern WDPGBackendStatus* get_pg_backend_status_from_master_wd_node(void);
 
@@ -72,11 +64,11 @@ extern WDIPCCmdResult* issue_command_to_watchdog(char type, int timeout_sec, cha
 
 
 /* functions for failover commands interlocking */
-extern WDFailoverCMDResults wd_end_failover_interlocking(void);
-extern WDFailoverCMDResults wd_start_failover_interlocking(void);
-extern WDFailoverCMDResults wd_failover_lock_release(enum WDFailoverLocks lock);
-extern WDFailoverCMDResults wd_failover_lock_status(enum WDFailoverLocks lock);
-extern void wd_wait_until_command_complete_or_timeout(enum WDFailoverLocks lock);
+extern WDFailoverCMDResults wd_end_failover_interlocking(unsigned int wd_failover_id);
+extern WDFailoverCMDResults wd_start_failover_interlocking(unsigned int wd_failover_id);
+extern WDFailoverCMDResults wd_failover_lock_release(enum WDFailoverLocks lock, unsigned int wd_failover_id);
+extern WDFailoverCMDResults wd_failover_lock_status(enum WDFailoverLocks lock, unsigned int wd_failover_id);
+extern void wd_wait_until_command_complete_or_timeout(enum WDFailoverLocks lock, unsigned int wd_failover_id);
 
 
 
diff --git a/src/include/watchdog/wd_ipc_defines.h b/src/include/watchdog/wd_ipc_defines.h
index 9301f2a..3f72066 100644
--- a/src/include/watchdog/wd_ipc_defines.h
+++ b/src/include/watchdog/wd_ipc_defines.h
@@ -6,7 +6,7 @@
  * pgpool: a language independent connection pool server for PostgreSQL
  * written by Tatsuo Ishii
  *
- * Copyright (c) 2003-2015	PgPool Global Development Group
+ * Copyright (c) 2003-2016	PgPool Global Development Group
  *
  * Permission to use, copy, modify, and distribute this software and
  * its documentation for any purpose and without fee is hereby
@@ -46,7 +46,16 @@ typedef enum WDFailoverCMDResults
 	FAILOVER_RES_UNLOCKED,				/* the lock is not acquired */
 	FAILOVER_RES_LOCKED,				/* lock is acquired */
 	FAILOVER_RES_SUCCESS,
-	FAILOVER_RES_NO_LOCKHOLDER
+	FAILOVER_RES_NO_LOCKHOLDER,
+	FAILOVER_RES_NO_LOCKHOLDER_BUT_WAIT,
+	
+	FAILOVER_RES_PROCEED,
+	FAILOVER_RES_WILL_BE_DONE,
+	FAILOVER_RES_NOT_ALLOWED,
+	FAILOVER_RES_INVALID_FUNCTION,
+	FAILOVER_RES_ALREADY_ISSUED,
+	FAILOVER_RES_MASTER_REJECTED,
+	FAILOVER_RES_TIMEOUT
 }WDFailoverCMDResults;
 
 
@@ -61,7 +70,8 @@ typedef enum WDFailoverCMDResults
 #define WD_IPC_CMD_RESULT_OK				'7'
 #define WD_IPC_CMD_TIMEOUT					'8'
 
-#define WD_FUNCTION_COMMAND					'f'
+#define WD_IPC_FAILOVER_COMMAND				'f'
+#define WD_IPC_ONLINE_RECOVERY_COMMAND		'r'
 #define WD_FAILOVER_CMD_SYNC_REQUEST		's'
 #define WD_GET_MASTER_DATA_REQUEST			'd'
 
@@ -79,6 +89,8 @@ typedef enum WDFailoverCMDResults
 #define WD_REQ_FAILOVER_RELEASE_LOCK	"RELEASE_LOCK"
 #define WD_REQ_FAILOVER_LOCK_STATUS		"CHECK_LOCKED"
 
+#define WD_FAILOVER_RESULT_KEY			"FAILOVER_COMMAND_RESULT"
+#define WD_FAILOVER_ID_KEY				"FAILOVER_COMMAND_ID"
 
 
 #define WD_IPC_AUTH_KEY			"IPCAuthKey"	/* JSON data key for authentication.
diff --git a/src/include/watchdog/wd_json_data.h b/src/include/watchdog/wd_json_data.h
index e8f662d..1efe48b 100644
--- a/src/include/watchdog/wd_json_data.h
+++ b/src/include/watchdog/wd_json_data.h
@@ -64,7 +64,7 @@ extern WDNodeInfo* get_WDNodeInfo_from_wd_node_json(json_value* source);
 
 extern char* get_wd_node_function_json(char* func_name, int *node_id_set, int count, unsigned int sharedKey, char* authKey);
 extern bool parse_wd_node_function_json(char* json_data, int data_len, char** func_name, int **node_id_set, int *count);
-extern char* get_wd_simple_error_message_json(char* message);
+extern char* get_wd_simple_message_json(char* message);
 
 extern WDPGBackendStatus* get_pg_backend_node_status_from_json(char* json_data, int data_len);
 extern char* get_backend_node_status_json(WatchdogNode* wdNode);
diff --git a/src/main/pgpool_main.c b/src/main/pgpool_main.c
index cd490be..95d3c68 100644
--- a/src/main/pgpool_main.c
+++ b/src/main/pgpool_main.c
@@ -577,7 +577,7 @@ process_backend_health_check_failure(int health_check_node_id, int retrycnt)
 			ereport(LOG,
 					(errmsg("setting backend node %d status to NODE DOWN", health_check_node_id)));
 			health_check_timer_expired = 0;
-			degenerate_backend_set(&health_check_node_id, 1, partial_failover);
+			degenerate_backend_set(&health_check_node_id, 1, partial_failover, 0);
 			return 2;
 			/* need to distribute this info to children ??*/
 		}
@@ -591,7 +591,7 @@ process_backend_health_check_failure(int health_check_node_id, int retrycnt)
  * This function enqueues the failover/failback requests, and fires the failover() if the function
  * is not already executing
  */
-bool register_node_operation_request(POOL_REQUEST_KIND kind, int* node_id_set, int count, bool switch_over)
+bool register_node_operation_request(POOL_REQUEST_KIND kind, int* node_id_set, int count, bool switch_over , unsigned int wd_failover_id)
 {
 	bool failover_in_progress;
 	pool_sigset_t oldmask;
@@ -617,6 +617,7 @@ bool register_node_operation_request(POOL_REQUEST_KIND kind, int* node_id_set, i
 	Req_info->request_queue_tail++;
 	index = Req_info->request_queue_tail % MAX_REQUEST_QUEUE_SIZE;
 	Req_info->request[index].kind = kind;
+	Req_info->request[index].wd_failover_id = wd_failover_id;
 
 	/* Set switch over flag if requested */
 	if (switch_over)
@@ -1100,7 +1101,7 @@ void notice_backend_error(int node_id, bool switch_over)
 	}
 	else
 	{
-		degenerate_backend_set(&n, 1, switch_over);
+		degenerate_backend_set(&n, 1, switch_over, 0);
 	}
 }
 
@@ -1120,9 +1121,11 @@ void notice_backend_error(int node_id, bool switch_over)
  *				For test_only case function returs false or throws an error as
  *				soon as first non complient node in node_id_set is found
  * switch_over: if set, the request is originated by switch over, not errors.
+ *
+ * wd_failover_id: The watchdog internal ID for this failover
  */
 bool degenerate_backend_set_ex(int *node_id_set, int count, bool error, bool test_only,
-							   bool switch_over)
+							   bool switch_over, unsigned int wd_failover_id)
 {
 	int i;
 	int node_id[MAX_NUM_BACKENDS];
@@ -1172,36 +1175,41 @@ bool degenerate_backend_set_ex(int *node_id_set, int count, bool error, bool tes
 
 	if (node_count)
 	{
-		WdCommandResult res = COMMAND_OK;
+		WDFailoverCMDResults res = FAILOVER_RES_PROCEED;
 		/* If this was only a test. Inform the caller without doing anything */
 		if(test_only)
 			return true;
 
-		if (pool_config->use_watchdog)
+		if (pool_config->use_watchdog && wd_failover_id == 0)
 		{
 			int x;
 			for (x=0; x < MAX_SEC_WAIT_FOR_CLUSTER_TRANSATION; x++)
 			{
-				res = wd_degenerate_backend_set(node_id_set, count);
-				if (res != CLUSTER_IN_TRANSATIONING)
+				res = wd_degenerate_backend_set(node_id_set, count, &wd_failover_id);
+				if (res != FAILOVER_RES_TRANSITION)
 					break;
 				sleep(1);
 			}
 		}
-		if (res == CLUSTER_IN_TRANSATIONING)
+		if (res == FAILOVER_RES_TRANSITION)
 		{
 			/*
 			 * What to do when cluster is still not stable
 			 * Is proceeding to failover is the right choice ???
 			 */
 			ereport(NOTICE,
-					(errmsg("rdegenerate backend request for %d node(s) from pid [%d], But cluster is not in stable state"
+					(errmsg("received degenerate backend request for %d node(s) from pid [%d], But cluster is not in stable state"
 							, node_count, getpid())));
 		}
-
-		if (res != COMMAND_FAILED)
+		if (res == FAILOVER_RES_PROCEED)
 		{
-			register_node_operation_request(NODE_DOWN_REQUEST, node_id, node_count, switch_over);
+			register_node_operation_request(NODE_DOWN_REQUEST, node_id, node_count, switch_over, wd_failover_id);
+		}
+		else if (res == FAILOVER_RES_WILL_BE_DONE)
+		{
+			ereport(LOG,
+					(errmsg("degenerate backend request for %d node(s) from pid [%d], will be handled by watchdog"
+							, node_count, getpid())));
 		}
 		else
 		{
@@ -1218,15 +1226,16 @@ bool degenerate_backend_set_ex(int *node_id_set, int count, bool error, bool tes
  * wrapper over degenerate_backend_set_ex function to register
  * NODE down operation request
  */
-void degenerate_backend_set(int *node_id_set, int count, bool switch_over)
+void degenerate_backend_set(int *node_id_set, int count, bool switch_over, unsigned int wd_failover_id)
 {
-	degenerate_backend_set_ex(node_id_set, count, false, false, switch_over);
+	degenerate_backend_set_ex(node_id_set, count, false, false, switch_over, wd_failover_id);
 }
 
 /* send promote node request using SIGUSR1 */
-void promote_backend(int node_id)
+void promote_backend(int node_id, unsigned int wd_failover_id)
 {
-	WdCommandResult res = COMMAND_OK;
+	WDFailoverCMDResults res = FAILOVER_RES_PROCEED;
+
 
 	if (!MASTER_SLAVE || pool_config->master_slave_sub_mode != STREAM_MODE)
 	{
@@ -1250,18 +1259,18 @@ void promote_backend(int node_id)
 					node_id, getpid())));
 
 	/* If this was only a test. Inform the caller without doing anything */
-	if (pool_config->use_watchdog)
+	if (pool_config->use_watchdog && wd_failover_id == 0)
 	{
 		int x;
 		for (x=0; x < MAX_SEC_WAIT_FOR_CLUSTER_TRANSATION; x++)
 		{
-			res = wd_promote_backend(node_id);
-			if (res != CLUSTER_IN_TRANSATIONING)
+			res = wd_promote_backend(node_id, &wd_failover_id);
+			if (res != FAILOVER_RES_TRANSITION)
 				break;
 			sleep(1);
 		}
 	}
-	if (res == CLUSTER_IN_TRANSATIONING)
+	if (res == FAILOVER_RES_TRANSITION)
 	{
 		/*
 		 * What to do when cluster is still not stable
@@ -1272,9 +1281,15 @@ void promote_backend(int node_id)
 						, node_id, getpid())));
 	}
 
-	if (res != COMMAND_FAILED)
+	if (res == FAILOVER_RES_PROCEED)
 	{
-		register_node_operation_request(PROMOTE_NODE_REQUEST, &node_id, 1, false);
+		register_node_operation_request(PROMOTE_NODE_REQUEST, &node_id, 1, false, wd_failover_id);
+	}
+	else if (res == FAILOVER_RES_WILL_BE_DONE)
+	{
+		ereport(LOG,
+				(errmsg("promote backend request for node_id: %d from pid [%d], will be handled by watchdog"
+						, node_id, getpid())));
 	}
 	else
 	{
@@ -1285,9 +1300,9 @@ void promote_backend(int node_id)
 }
 
 /* send failback request using SIGUSR1 */
-void send_failback_request(int node_id,bool throw_error)
+void send_failback_request(int node_id,bool throw_error, unsigned int wd_failover_id)
 {
-	WdCommandResult res = COMMAND_OK;
+	WDFailoverCMDResults res = FAILOVER_RES_PROCEED;
 
     if (node_id < 0 || node_id >= MAX_NUM_BACKENDS ||
 		(RAW_MODE && BACKEND_INFO(node_id).backend_status != CON_DOWN && VALID_BACKEND(node_id)))
@@ -1304,22 +1319,22 @@ void send_failback_request(int node_id,bool throw_error)
 	}
 
 	ereport(LOG,
-			(errmsg("received failback request for node_id: %d from pid [%d]",
-					node_id, getpid())));
+			(errmsg("received failback request for node_id: %d from pid [%d] wd_failover_id [%d]",
+					node_id, getpid(),wd_failover_id)));
 
 	/* If this was only a test. Inform the caller without doing anything */
-	if (pool_config->use_watchdog)
+	if (pool_config->use_watchdog && wd_failover_id == 0)
 	{
 		int x;
 		for (x=0; x < MAX_SEC_WAIT_FOR_CLUSTER_TRANSATION; x++)
 		{
-			res = wd_send_failback_request(node_id);
-			if (res != CLUSTER_IN_TRANSATIONING)
+			res = wd_send_failback_request(node_id, &wd_failover_id);
+			if (res != FAILOVER_RES_TRANSITION)
 				break;
 			sleep(1);
 		}
 	}
-	if (res == CLUSTER_IN_TRANSATIONING)
+	if (res == FAILOVER_RES_TRANSITION)
 	{
 		/*
 		 * What to do when cluster is still not stable
@@ -1330,10 +1345,17 @@ void send_failback_request(int node_id,bool throw_error)
 						, node_id, getpid())));
 	}
 
-	if (res != COMMAND_FAILED)
+	if (res == FAILOVER_RES_PROCEED)
+	{
+		register_node_operation_request(NODE_UP_REQUEST, &node_id, 1, false, wd_failover_id);
+	}
+	else if (res == FAILOVER_RES_WILL_BE_DONE)
 	{
-		register_node_operation_request(NODE_UP_REQUEST, &node_id, 1, false);
+		ereport(LOG,
+				(errmsg("failback request for node_id: %d from pid [%d], will be handled by watchdog"
+						, node_id, getpid())));
 	}
+
 	else
 	{
 		ereport(throw_error?ERROR:LOG,
@@ -1560,6 +1582,7 @@ static void failover(void)
 		int node_id_set[MAX_NUM_BACKENDS];
 		int node_count;
 		unsigned char request_details;
+		unsigned int wd_failover_id;
 		WDFailoverCMDResults wdInterlockingRes;
 
 		pool_semaphore_lock(REQUEST_INFO_SEM);
@@ -1579,7 +1602,7 @@ static void failover(void)
 		reqkind = Req_info->request[queue_index].kind;
 		request_details = Req_info->request[queue_index].request_details;
 		node_count = Req_info->request[queue_index].count;
-
+		wd_failover_id = Req_info->request[queue_index].wd_failover_id;
 		pool_semaphore_unlock(REQUEST_INFO_SEM);
 
 		ereport(DEBUG1,
@@ -1593,7 +1616,7 @@ static void failover(void)
 		}
 
 		/* start watchdog interlocking */
-		wdInterlockingRes = wd_start_failover_interlocking();
+		wdInterlockingRes = wd_start_failover_interlocking(wd_failover_id);
 
 		/*
 		 * if not in replication mode/master slave mode, we treat this a restart request.
@@ -1620,7 +1643,7 @@ static void failover(void)
 							(errmsg("invalid failback request, status: [%d] of node id : %d is invalid for failback",BACKEND_INFO(node_id).backend_status,node_id)));
 
 				if (wdInterlockingRes == FAILOVER_RES_I_AM_LOCK_HOLDER)
-					wd_end_failover_interlocking();
+					wd_end_failover_interlocking(wd_failover_id);
 
 				continue;
 			}
@@ -1652,7 +1675,7 @@ static void failover(void)
 			{
 				trigger_failover_command(node_id, pool_config->failback_command,
 											MASTER_NODE_ID, get_next_master_node(), PRIMARY_NODE_ID);
-				wd_failover_lock_release(FAILBACK_LOCK);
+				wd_failover_lock_release(FAILBACK_LOCK, wd_failover_id);
 			}
 			else
 			{
@@ -1661,7 +1684,7 @@ static void failover(void)
 				 * so we need to wait till the one who is executing the command
 				 * finish with it.
 				 */
-				wd_wait_until_command_complete_or_timeout(FAILBACK_LOCK);
+				wd_wait_until_command_complete_or_timeout(FAILBACK_LOCK,wd_failover_id);
 			}
 		}
 		else if (reqkind == PROMOTE_NODE_REQUEST)
@@ -1678,7 +1701,7 @@ static void failover(void)
 				ereport(LOG,
 						(errmsg("failover: no backends are promoted")));
 				if (wdInterlockingRes == FAILOVER_RES_I_AM_LOCK_HOLDER)
-					wd_end_failover_interlocking();
+					wd_end_failover_interlocking(wd_failover_id);
 				continue;
 			}
 		}
@@ -1712,7 +1735,7 @@ static void failover(void)
 						(errmsg("failover: no backends are degenerated")));
 
 				if (wdInterlockingRes == FAILOVER_RES_I_AM_LOCK_HOLDER)
-					wd_end_failover_interlocking();
+					wd_end_failover_interlocking(wd_failover_id);
 
 				continue;
 			}
@@ -1838,11 +1861,11 @@ static void failover(void)
 					trigger_failover_command(i, pool_config->failover_command,
 												MASTER_NODE_ID, new_master, PRIMARY_NODE_ID);
 			}
-			wd_failover_lock_release(FAILOVER_LOCK);
+			wd_failover_lock_release(FAILOVER_LOCK, wd_failover_id);
 		}
 		else
 		{
-			wd_wait_until_command_complete_or_timeout(FAILOVER_LOCK);
+			wd_wait_until_command_complete_or_timeout(FAILOVER_LOCK, wd_failover_id);
 		}
 
 	/* no need to wait since it will be done in reap_handler */
@@ -1937,11 +1960,11 @@ static void failover(void)
 				follow_pid = fork_follow_child(Req_info->master_node_id, new_primary,
 											Req_info->primary_node_id);
 			}
-			wd_failover_lock_release(FOLLOW_MASTER_LOCK);
+			wd_failover_lock_release(FOLLOW_MASTER_LOCK, wd_failover_id);
 		}
 		else
 		{
-			wd_wait_until_command_complete_or_timeout(FOLLOW_MASTER_LOCK);
+			wd_wait_until_command_complete_or_timeout(FOLLOW_MASTER_LOCK, wd_failover_id);
 
 		}
 
@@ -2031,7 +2054,7 @@ static void failover(void)
 		kill(worker_pid, SIGUSR1);
 
 		if (wdInterlockingRes == FAILOVER_RES_I_AM_LOCK_HOLDER)
-			wd_end_failover_interlocking();
+			wd_end_failover_interlocking(wd_failover_id);
 
 		if (reqkind == NODE_UP_REQUEST)
 		{
diff --git a/src/pcp_con/pcp_worker.c b/src/pcp_con/pcp_worker.c
index 5fdfdf0..900ad7f 100644
--- a/src/pcp_con/pcp_worker.c
+++ b/src/pcp_con/pcp_worker.c
@@ -502,14 +502,14 @@ static int pool_detach_node(int node_id, bool gracefully)
 {
 	if (!gracefully)
 	{
-		degenerate_backend_set_ex(&node_id, 1, true, false, true);
+		degenerate_backend_set_ex(&node_id, 1, true, false, true, 0);
 		return 0;
 	}
 
 	/* Check if the NODE DOWN can be executed on
 	 * the given node id.
 	 */
-	degenerate_backend_set_ex(&node_id, 1, true, true, true);
+	degenerate_backend_set_ex(&node_id, 1, true, true, true, 0);
 
 	/*
 	 * Wait until all frontends exit
@@ -529,7 +529,7 @@ static int pool_detach_node(int node_id, bool gracefully)
 	/*
 	 * Now all frontends have gone. Let's do failover.
 	 */
-	degenerate_backend_set_ex(&node_id, 1, true, false, true);
+	degenerate_backend_set_ex(&node_id, 1, true, false, true, 0);
 
 	/*
 	 * Wait for failover completed.
@@ -556,7 +556,7 @@ static int pool_promote_node(int node_id, bool gracefully)
 {
 	if (!gracefully)
 	{
-		promote_backend(node_id);	/* send promote request */
+		promote_backend(node_id, false);	/* send promote request */
 		return 0;
 	}
 
@@ -576,7 +576,7 @@ static int pool_promote_node(int node_id, bool gracefully)
 	/*
 	 * Now all frontends have gone. Let's do failover.
 	 */
-	promote_backend(node_id);		/* send promote request */
+	promote_backend(node_id, false);		/* send promote request */
 
 	/*
 	 * Wait for failover completed.
@@ -910,7 +910,7 @@ process_attach_node(PCP_CONNECTION *frontend,char *buf)
 			(errmsg("PCP: processing attach node"),
 			 errdetail("attaching Node ID %d", node_id)));
 
-	send_failback_request(node_id,true);
+	send_failback_request(node_id,true, false);
 
 	pcp_write(frontend, "c", 1);
 	wsize = htonl(sizeof(code) + sizeof(int));
diff --git a/src/pcp_con/recovery.c b/src/pcp_con/recovery.c
index 260f0c6..f2f57f5 100644
--- a/src/pcp_con/recovery.c
+++ b/src/pcp_con/recovery.c
@@ -147,7 +147,7 @@ void start_recovery(int recovery_node)
 		pcp_worker_wakeup_request = 0;
 
 		/* send failback request to pgpool parent */
-		send_failback_request(recovery_node,false);
+		send_failback_request(recovery_node,false, false);
 
 		/* wait for failback */
 		failback_wait_count = 0;
diff --git a/src/protocol/pool_process_query.c b/src/protocol/pool_process_query.c
index 81f8647..58e75e9 100644
--- a/src/protocol/pool_process_query.c
+++ b/src/protocol/pool_process_query.c
@@ -3698,7 +3698,7 @@ void read_kind_from_backend(POOL_CONNECTION *frontend, POOL_CONNECTION_POOL *bac
 
 		if (pool_config->replication_stop_on_mismatch)
 		{
-			degenerate_backend_set(degenerate_node, degenerate_node_num, false);
+			degenerate_backend_set(degenerate_node, degenerate_node_num, false, 0);
             retcode = 1;
 		}
         ereport(FATAL,
diff --git a/src/protocol/pool_proto_modules.c b/src/protocol/pool_proto_modules.c
index 9549bbb..c4115f8 100644
--- a/src/protocol/pool_proto_modules.c
+++ b/src/protocol/pool_proto_modules.c
@@ -500,7 +500,7 @@ POOL_STATUS SimpleQuery(POOL_CONNECTION *frontend,
 
 			ereport(DEBUG1,
 					(errmsg("Query: sending SIGUSR1 signal to parent")));
-			register_node_operation_request(CLOSE_IDLE_REQUEST, NULL, 0, false);
+			register_node_operation_request(CLOSE_IDLE_REQUEST, NULL, 0, false, 0);
 
 			/* we need to loop over here since we will get USR1 signal while sleeping */
 			while (stime > 0)
@@ -1632,7 +1632,7 @@ POOL_STATUS ReadyForQuery(POOL_CONNECTION *frontend,
 
 				free_string(msg);
 
-				degenerate_backend_set(victim_nodes, number_of_nodes, true);
+				degenerate_backend_set(victim_nodes, number_of_nodes, true, 0);
 				child_exit(POOL_EXIT_AND_RESTART);
 			}
 			else
diff --git a/src/utils/pool_stream.c b/src/utils/pool_stream.c
index e1336d3..36713ea 100644
--- a/src/utils/pool_stream.c
+++ b/src/utils/pool_stream.c
@@ -1173,9 +1173,32 @@ int socket_read(int fd, void* buf, size_t len, int timeout)
 {
 	int ret, read_len;
 	read_len = 0;
-
+	struct timeval timeoutval;
+	fd_set readmask;
+	int fds;
+	
 	while (read_len < len)
 	{
+		FD_ZERO(&readmask);
+		FD_SET(fd, &readmask);
+		
+		timeoutval.tv_sec = timeout;
+		timeoutval.tv_usec = 0;
+		
+		fds = select(fd+1, &readmask, NULL, NULL, timeout?&timeoutval:NULL);
+		if (fds == -1)
+		{
+			if (errno == EAGAIN || errno == EINTR)
+				continue;
+			
+			ereport(WARNING,
+					(errmsg("select failed with error: \"%s\"", strerror(errno))));
+			return -1;
+		}
+		else if (fds == 0)
+		{
+			return -2;
+		}
 		ret = read(fd, buf + read_len, (len - read_len));
 		if(ret < 0)
 		{
diff --git a/src/watchdog/watchdog.c b/src/watchdog/watchdog.c
index 791efa6..52197ff 100644
--- a/src/watchdog/watchdog.c
+++ b/src/watchdog/watchdog.c
@@ -62,7 +62,9 @@ typedef enum IPC_CMD_PREOCESS_RES
 {
 	IPC_CMD_COMPLETE,
 	IPC_CMD_PROCESSING,
-	IPC_CMD_ERROR
+	IPC_CMD_ERROR,
+	IPC_CMD_OK,
+	IPC_CMD_TRY_AGAIN
 }IPC_CMD_PREOCESS_RES;
 
 
@@ -77,6 +79,10 @@ typedef enum IPC_CMD_PREOCESS_RES
 
 #define BEACON_MESSAGE_INTERVAL_SECONDS		10 /* interval between beacon messages */
 
+#define	MAX_SECS_WAIT_FOR_REPLY_FROM_NODE	5  /* time in seconds to wait for the reply from
+												* remote watchdog node
+												*/
+
 
 
 #define WD_NO_MESSAGE						0
@@ -95,10 +101,11 @@ typedef enum IPC_CMD_PREOCESS_RES
 #define WD_REJECT_MESSAGE					'R'
 #define WD_STAND_FOR_COORDINATOR_MESSAGE	'S'
 #define WD_INTERUNLOCKING_REQUEST			'U'
-#define WD_REPLICATE_VARIABLE_REQUEST		'V'
+#define WD_REMOTE_FAILOVER_REQUEST			'V'
 #define WD_INFORM_I_AM_GOING_DOWN			'X'
 #define WD_ASK_FOR_POOL_CONFIG				'Y'
 #define WD_POOL_CONFIG_DATA					'Z'
+#define WD_CMD_REPLY_IN_DATA				'-'
 
 typedef struct packet_types
 {
@@ -123,7 +130,9 @@ packet_types all_packet_types[] = {
 	{WD_STAND_FOR_COORDINATOR_MESSAGE, "STAND FOR COORDINATOR"},
 	
 	{WD_INTERUNLOCKING_REQUEST, "INTERUNLOCKING REQUEST"},
-	{WD_REPLICATE_VARIABLE_REQUEST, "REPLICATE VARIABLE REQUEST"},
+	{WD_REMOTE_FAILOVER_REQUEST, "REPLICATE FAILOVER REQUEST"},
+	{WD_IPC_ONLINE_RECOVERY_COMMAND, "ONLINE RECOVERY REQUEST"},
+	{WD_IPC_FAILOVER_COMMAND, "FAILOVER FUNCTION COMMAND"},
 	{WD_INFORM_I_AM_GOING_DOWN, "INFORM I AM GOING DOWN"},
 	{WD_ASK_FOR_POOL_CONFIG, "ASK FOR POOL CONFIG"},
 	{WD_POOL_CONFIG_DATA, "CONFIG DATA"},
@@ -198,7 +207,15 @@ typedef struct WDCommandNodeResult
 	char*	result_data;
 }WDCommandNodeResult;
 
+typedef enum WDCommandSource
+{
+	COMMAND_SOURCE_IPC,
+	COMMAND_SOURCE_LOCAL,
+	COMMAND_SOURCE_REMOTE,
+	COMMAND_SOURCE_INTERNAL
+}WDCommandSource;
 
+/*
 typedef struct WDIPCCommandData
 {
 	MemoryContext		memoryContext;
@@ -215,6 +232,7 @@ typedef struct WDIPCCommandData
 
 	WDCommandNodeResult*	nodeResults;
 }WDIPCCommandData;
+ */
 
 typedef struct WDFunctionCommandData
 {
@@ -247,21 +265,28 @@ typedef enum WDCommandStatus
 	COMMAND_IN_PROGRESS,
 	COMMAND_FINISHED_TIMEOUT,
 	COMMAND_FINISHED_ALL_REPLIED,
-	COMMAND_FINISHED_NODE_REJECTED
+	COMMAND_FINISHED_NODE_REJECTED,
+	COMMAND_FINISHED_SEND_FAILED
 }WDCommandStatus;
 
 typedef struct WDCommandData
 {
-	WDPacketData			packet;
+	WDPacketData			sourcePacket;
+	WDPacketData			commandPacket;
 	WDCommandNodeResult		*nodeResults;
 	WatchdogNode			*sendToNode;	/* NULL means send to all */
 	WDCommandStatus			commandStatus;
 	unsigned int			commandTimeoutSecs;
 	struct timeval			commandTime;
 	unsigned int			commandSendToCount;
+	unsigned int			commandSendToErrorCount;
 	unsigned int			commandReplyFromCount;
-	int						commandFinished;
-	int						partial_sent;
+	WDCommandSource			commandSource;
+	int						sourceIPCSocket;	/* Only valid for COMMAND_SOURCE_IPC */
+	WatchdogNode			*sourceWdNode;		/* Only valid for COMMAND_SOURCE_REMOTE */
+	char					*errorMessage;
+	MemoryContext			memoryContext;
+	void (*commandCompleteFunc)(struct WDCommandData* command);
 }WDCommandData;
 
 typedef struct WDInterfaceStatus
@@ -297,8 +322,34 @@ typedef struct wd_cluster
 	List			*ipc_commands;
 	List			*wd_timer_commands;
 	List			*wdInterfaceToMonitor;
+	List			*wdCurrentFailovers;
 }wd_cluster;
 
+typedef struct WDFailoverObject
+{
+	int id;
+	POOL_REQUEST_KIND reqKind;
+	int nodesCount;
+	unsigned int failoverID;
+	int *nodeList;
+	WatchdogNode* wdRequestingNode;
+	struct timeval	startTime;
+	int state;
+	MemoryContext memContext;
+}WDFailoverObject;
+static IPC_CMD_PREOCESS_RES process_failover_command_on_coordinator(WDCommandData* ipcCommand);
+static void process_remote_failover_command_on_coordinator(WatchdogNode* wdNode, WDPacketData* pkt);
+static IPC_CMD_PREOCESS_RES process_IPC_failover_command_on_coordinator(WDCommandData* ipcCommand);
+static WDFailoverObject* get_failover_object(POOL_REQUEST_KIND reqKind, int nodesCount, int *nodeList);
+static bool remove_failover_object_by_id(unsigned int failoverID);
+static WDFailoverObject* get_failover_object_by_id(unsigned int failoverID);
+static IPC_CMD_PREOCESS_RES process_IPC_failover_command(WDCommandData* ipcCommand);
+
+static int send_command_packet_to_remote_nodes(WDCommandData* ipcCommand, bool source_included);
+static void wd_command_is_complete(WDCommandData* ipcCommand);
+static IPC_CMD_PREOCESS_RES wd_command_processor_for_node_lost_event(WDCommandData* ipcCommand, WatchdogNode* wdLostNode);
+static bool is_cluster_command_in_progress(void);
+
 volatile sig_atomic_t reload_config_signal = 0;
 volatile sig_atomic_t sigchld_request = 0;
 
@@ -316,6 +367,13 @@ static void try_connecting_with_all_unreachable_nodes(void);
 static bool connect_to_node(WatchdogNode* wdNode);
 static bool is_socket_connection_connected(SocketConnection* conn);
 
+static void service_unreachable_nodes(void);
+
+static void allocate_resultNodes_in_IPCCommand(WDCommandData* ipcCommand);
+static bool is_node_active_and_reachable(WatchdogNode* wdNode);
+static bool is_node_active(WatchdogNode* wdNode);
+static bool is_node_reachable(WatchdogNode* wdNode);
+
 static int update_successful_outgoing_cons(fd_set* wmask, int pending_fds_count);
 static int prepare_fds(fd_set* rmask, fd_set* wmask, fd_set* emask);
 
@@ -335,10 +393,10 @@ static WDPacketData* read_packet(SocketConnection* conn);
 static WDPacketData* get_message_of_type(char type);
 
 static int issue_watchdog_internal_command(WatchdogNode* wdNode, WDPacketData *pkt, int timeout_sec);
-static char get_current_command_resultant_message_type(void);
 static void check_for_current_command_timeout(void);
 static bool watchdog_internal_command_packet_processor(WatchdogNode* wdNode, WDPacketData* pkt);
 static bool service_lost_connections(void);
+static void service_ipc_commands(void);
 static void service_internal_command(void);
 
 static unsigned int get_next_commandID(void);
@@ -346,7 +404,7 @@ static WatchdogNode* parse_node_info_message(WDPacketData* pkt, char **authkey);
 static int update_quorum_status(void);
 static int get_mimimum_nodes_required_for_quorum(void);
 
-static bool write_packet_to_socket(int sock, WDPacketData* pkt);
+static bool write_packet_to_socket(int sock, WDPacketData* pkt, bool ipcPacket);
 static int read_sockets(fd_set* rmask,int pending_fds_count);
 static void set_timeout(unsigned int sec);
 static int wd_create_command_server_socket(void);
@@ -381,8 +439,8 @@ static int watchdog_state_machine_loading(WD_EVENTS event, WatchdogNode* wdNode,
 static int watchdog_state_machine(WD_EVENTS event, WatchdogNode* wdNode, WDPacketData* pkt);
 static int watchdog_state_machine_nw_error(WD_EVENTS event, WatchdogNode* wdNode, WDPacketData* pkt);
 
-static void cleanUpIPCCommand(WDIPCCommandData* ipcCommand);
-static bool read_ipc_command_and_process(int socket, bool *remove_socket);
+static void cleanUpIPCCommand(WDCommandData* ipcCommand);
+static bool read_ipc_socket_and_process(int socket, bool *remove_socket);
 
 static JsonNode* get_node_list_json(int id);
 static bool add_nodeinfo_to_json(JsonNode* jNode, WatchdogNode* node);
@@ -390,34 +448,38 @@ static bool fire_node_status_event(int nodeID, int nodeStatus);
 static void resign_from_escalated_node(void);
 static void start_escalated_node(void);
 static void init_wd_packet(WDPacketData* pkt);
+static void wd_packet_shallow_copy(WDPacketData* srcPkt, WDPacketData* dstPkt);
 static bool wd_commands_packet_processor(WD_EVENTS event, WatchdogNode* wdNode, WDPacketData* pkt);
 
-static WDIPCCommandData* get_wd_IPC_command_from_reply(WDPacketData* pkt);
-static WDIPCCommandData* get_wd_IPC_command_from_socket(int sock);
+static WDCommandData* get_wd_IPC_command_from_reply(WDPacketData* pkt);
+static WDCommandData* get_wd_IPC_command_from_socket(int sock);
 
-static IPC_CMD_PREOCESS_RES process_IPC_command(WDIPCCommandData* ipcCommand);
-static IPC_CMD_PREOCESS_RES process_IPC_nodeStatusChange_command(WDIPCCommandData* IPCCommand);
-static IPC_CMD_PREOCESS_RES process_IPC_nodeList_command(WDIPCCommandData* IPCCommand);
-static IPC_CMD_PREOCESS_RES process_IPC_replicate_variable(WDIPCCommandData* IPCCommand);
-static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandData *IPCCommand);
-static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandData *IPCCommand);
-static IPC_CMD_PREOCESS_RES execute_replicate_command(WDIPCCommandData* ipcCommand);
-static bool write_ipc_command_with_result_data(WDIPCCommandData* IPCCommand, char type, char* data, int len);
+static IPC_CMD_PREOCESS_RES process_IPC_command(WDCommandData* ipcCommand);
+static IPC_CMD_PREOCESS_RES process_IPC_nodeStatusChange_command(WDCommandData* ipcCommand);
+static IPC_CMD_PREOCESS_RES process_IPC_nodeList_command(WDCommandData* ipcCommand);
+static IPC_CMD_PREOCESS_RES process_IPC_online_recovery(WDCommandData* ipcCommand);
+static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDCommandData *ipcCommand);
+static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDCommandData *ipcCommand);
+static bool write_ipc_command_with_result_data(WDCommandData* ipcCommand, char type, char* data, int len);
 
 static int node_has_requested_for_interlocking(WatchdogNode* wdNode, WDPacketData* pkt);
 static bool node_has_resigned_from_interlocking(WatchdogNode* wdNode, WDPacketData* pkt);
 
 static void process_wd_func_commands_for_timer_events(void);
 static void add_wd_command_for_timer_events(unsigned int expire_secs, bool need_tics, WDFunctionCommandData* wd_func_command);
-static bool reply_is_received_for_pgpool_replicate_command(WatchdogNode* wdNode, WDPacketData* pkt, WDIPCCommandData* ipcCommand);
-static bool process_wd_command_function(WatchdogNode* wdNode, WDPacketData* pkt, char* func_name, int node_count, int* node_id_list);
-static bool process_pgpool_replicate_command(WatchdogNode* wdNode, WDPacketData* pkt);
-
-static void process_failover_command_sync_requests(WatchdogNode* wdNode, WDPacketData* pkt, WDIPCCommandData* ipcCommand);
-static WDFailoverCMDResults node_is_asking_for_failover_end(WatchdogNode* wdNode, WDPacketData* pkt);
-static WDFailoverCMDResults node_is_asking_for_failover_start(WatchdogNode* wdNode, WDPacketData* pkt);
-static WDFailoverCMDResults node_is_asking_for_failover_lock_status(WatchdogNode* wdNode, WDPacketData* pkt, WDFailoverLock failoverLock);
-static WDFailoverCMDResults node_is_asking_for_failover_lock_release(WatchdogNode* wdNode, WDPacketData* pkt, WDFailoverLock failoverLock);
+static bool reply_is_received_for_pgpool_replicate_command(WatchdogNode* wdNode, WDPacketData* pkt, WDCommandData* ipcCommand);
+static void process_wd_command_function(WatchdogNode* wdNode, WDPacketData* pkt, char* func_name, int node_count, int* node_id_list, unsigned int failover_id);
+static void process_pgpool_remote_failover_command(WatchdogNode* wdNode, WDPacketData* pkt);
+static void process_remote_online_recovery_command(WatchdogNode* wdNode, WDPacketData* pkt);
+
+
+static IPC_CMD_PREOCESS_RES process_failover_locking_requests_on_cordinator(WDCommandData* ipcCommand);
+static WDFailoverCMDResults node_is_asking_for_failover_end(WatchdogNode* wdNode, WDPacketData* pkt, unsigned int failoverID);
+static WDFailoverCMDResults node_is_asking_for_failover_start(WatchdogNode* wdNode, WDPacketData* pkt, unsigned int failoverID);
+static WDFailoverCMDResults node_is_asking_for_failover_lock_status(WatchdogNode* wdNode, WDPacketData* pkt,
+																	WDFailoverLock failoverLock, unsigned int failoverID);
+static WDFailoverCMDResults node_is_asking_for_failover_lock_release(WatchdogNode* wdNode, WDPacketData* pkt,
+																	 WDFailoverLock failoverLock, unsigned int failoverID);
 static void wd_system_will_go_down(int code, Datum arg);
 static void verify_pool_configurations(WatchdogNode* wdNode, POOL_CONFIG* config);
 
@@ -431,7 +493,7 @@ static pid_t watchdog_main(void);
 static pid_t fork_watchdog_child(void);
 static void cluster_in_stable_state(void);
 static bool check_IPC_client_authentication(json_value *rootObj, bool internal_client_only);
-static bool check_and_report_IPC_authentication(WDIPCCommandData* ipcCommand);
+static bool check_and_report_IPC_authentication(WDCommandData* ipcCommand);
 
 static void print_received_packet_info(WDPacketData* pkt,WatchdogNode* wdNode);
 static void update_interface_status(void);
@@ -834,8 +896,15 @@ try_connecting_with_all_unreachable_nodes(void)
 		if (wdNode->client_socket.sock_state != WD_SOCK_WAITING_FOR_CONNECT && wdNode->client_socket.sock_state != WD_SOCK_CONNECTED &&
 			wdNode->server_socket.sock_state != WD_SOCK_WAITING_FOR_CONNECT && wdNode->server_socket.sock_state != WD_SOCK_CONNECTED)
 		{
-			wdNode->state = WD_DEAD;
+			if (wdNode->state == WD_SHUTDOWN)
+				continue;
 			connect_to_node(wdNode);
+			if (wdNode->client_socket.sock_state == WD_SOCK_CONNECTED)
+			{
+				ereport(LOG,
+						(errmsg("connection to the remote node \"%s\" is restored",wdNode->nodeName)));
+				watchdog_state_machine(WD_EVENT_NEW_OUTBOUND_CONNECTION, wdNode, NULL);
+			}
 		}
 	}
 }
@@ -1036,17 +1105,27 @@ watchdog_main(void)
 			processed_fds += update_successful_outgoing_cons(&wmask,(select_ret - processed_fds));
 			processed_fds += read_sockets(&rmask,(select_ret - processed_fds));
 		}
-
-		if (timeout_event)
-			watchdog_state_machine(WD_EVENT_TIMEOUT, NULL, NULL);
 		if (WD_TIME_DIFF_SEC(ref_time,g_tm_set_time) >=  1)
+		{
 			process_wd_func_commands_for_timer_events();
-
+		}
+		
+		if (timeout_event)
+		{
+			g_timeout_sec = 0;
+			watchdog_state_machine(WD_EVENT_TIMEOUT, NULL, NULL);
+		}
+		
 		check_for_current_command_timeout();
-
+		
 		if (service_lost_connections() == true)
+		{
 			service_internal_command();
-
+			service_ipc_commands();
+		}
+		
+		service_unreachable_nodes();
+		
 		update_connected_node_count();
 	}
 	return 0;
@@ -1233,8 +1312,17 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 				if (pkt)
 				{
 					watchdog_state_machine(WD_EVENT_PACKET_RCV, wdNode, pkt);
+					/* since a packet is received reset last sent time */
+					wdNode->last_sent_time.tv_sec = 0;
+					wdNode->last_sent_time.tv_usec = 0;
 					free_packet(pkt);
 				}
+				else
+				{
+					ereport(LOG,
+							(errmsg("client socket of %s is closed", wdNode->nodeName)));
+				}
+
 				count++;
 				if (count >= pending_fds_count)
 					return count;
@@ -1250,9 +1338,17 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 				if (pkt)
 				{
 					watchdog_state_machine(WD_EVENT_PACKET_RCV, wdNode, pkt);
+					/* since a packet is received reset last sent time */
+					wdNode->last_sent_time.tv_sec = 0;
+					wdNode->last_sent_time.tv_usec = 0;
 					free_packet(pkt);
 				}
-				
+				else
+				{
+					ereport(LOG,
+							(errmsg("outbound socket of %s is closed", wdNode->nodeName)));
+				}
+
 				count++;
 				if (count >= pending_fds_count)
 					return count;
@@ -1283,7 +1379,7 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 					print_watchdog_node_info(tempNode);
 					authenticated = verify_authhash_for_node(tempNode, authkey);
 					ereport(DEBUG1,
-							(errmsg("ADD NODE MESSAGE from Hostname:\"%s\" PORT:%d pgpool_port:%d",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port)));
+							(errmsg("ADD NODE MESSAGE from hostname:\"%s\" port:%d pgpool_port:%d",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port)));
 					/* verify this node */
 					if (authenticated)
 					{
@@ -1311,27 +1407,27 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 						{
 							/* reply with node info message */
 							ereport(NOTICE,
-									(errmsg("New node joined the cluster Hostname:\"%s\" PORT:%d pgpool_port:%d",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port)));
+									(errmsg("New node joined the cluster hostname:\"%s\" port:%d pgpool_port:%d",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port)));
 
 							watchdog_state_machine(WD_EVENT_PACKET_RCV, wdNode, pkt);
 						}
 						else
 							ereport(NOTICE,
-								(errmsg("add node from Hostname:\"%s\" PORT:%d pgpool_port:%d rejected.",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port),
+								(errmsg("add node from hostname:\"%s\" port:%d pgpool_port:%d rejected.",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port),
 									 errdetail("verify the other watchdog node configurations")));
 
 					}
 					else
 					{
 						ereport(NOTICE,
-								(errmsg("authentication failed for add node from Hostname:\"%s\" PORT:%d pgpool_port:%d",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port),
+								(errmsg("authentication failed for add node from hostname:\"%s\" port:%d pgpool_port:%d",tempNode->hostname,tempNode->wd_port,tempNode->pgpool_port),
 								 errdetail("make sure wd_authkey configuration is same on all nodes")));
 					}
 
 					if (found == false || authenticated == false)
 					{
 						/* reply with reject message, We do not need to go to state processor */
-						/* For now, create a empty temp node. TODO*/
+						/* For now, create a empty temp node.*/
 						WatchdogNode tmpNode;
 						tmpNode.client_socket = *conn;
 						tmpNode.client_socket.sock_state = WD_SOCK_CONNECTED;
@@ -1372,18 +1468,18 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 		if (command_sock > 0 &&  FD_ISSET(command_sock, rmask))
 		{
 			bool remove_sock = false;
-			read_ipc_command_and_process(command_sock, &remove_sock);
+			read_ipc_socket_and_process(command_sock, &remove_sock);
 			if (remove_sock)
 			{
 				/* Also locate the command if it has this socket */
-				WDIPCCommandData* ipcCommand = get_wd_IPC_command_from_socket(command_sock);
+				WDCommandData* ipcCommand = get_wd_IPC_command_from_socket(command_sock);
 				if (ipcCommand)
 				{
 					/* special case we want to remove the socket from
 					 * ipc_command_sock list manually, so mark the issuing socket
 					 * of ipcComman to invalid value
 					 */
-					ipcCommand->issueing_sock = -1;
+					ipcCommand->sourceIPCSocket = -1;
 				}
 				close(command_sock);
 				socks_to_del = lappend_int(socks_to_del,command_sock);
@@ -1411,7 +1507,7 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 		if (notify_sock > 0 &&  FD_ISSET(notify_sock, rmask))
 		{
 			bool remove_sock = false;
-			read_ipc_command_and_process(notify_sock, &remove_sock);
+			read_ipc_socket_and_process(notify_sock, &remove_sock);
 			if (remove_sock)
 			{
 				close(notify_sock);
@@ -1440,7 +1536,7 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 		if (read_interface_change_event(g_cluster.network_monitor_sock, &link_event, &deleted))
 		{
 			ereport(DEBUG1,
-					(errmsg("Network event received"),
+					(errmsg("network event received"),
 						errdetail("deleted = %s Link change event = %s",
 							deleted?"YES":"NO",
 							link_event?"YES":"NO")));
@@ -1464,31 +1560,49 @@ static int read_sockets(fd_set* rmask,int pending_fds_count)
 	return count;
 }
 
-static bool write_ipc_command_with_result_data(WDIPCCommandData* IPCCommand, char type, char* data, int len)
+static bool write_ipc_command_with_result_data(WDCommandData* ipcCommand, char type, char* data, int len)
 {
-	int send_len = 0;
-	if (socket_write(IPCCommand->issueing_sock, &type, 1) < 0)
-		return false;
-
-	if (len > 0)
+	WDPacketData pkt;
+	pkt.data = data;
+	pkt.len = len;
+	pkt.type = type;
+	if (ipcCommand == NULL || ipcCommand->commandSource != COMMAND_SOURCE_IPC || ipcCommand->sourceIPCSocket <= 0)
 	{
-		send_len = htonl(len);
-		if (socket_write(IPCCommand->issueing_sock, &send_len, sizeof(int)) < 0)
-			return false;
-		if (socket_write(IPCCommand->issueing_sock, data, len) < 0)
-			return false;
-	}
-	else if (socket_write(IPCCommand->issueing_sock, &send_len, sizeof(int)) < 0)
+		ereport(DEBUG1,
+				(errmsg("not replying to IPC, Invalid IPC command.")));
 		return false;
-	return true;
+	}
+	return write_packet_to_socket(ipcCommand->sourceIPCSocket, &pkt, true);
+}
+
+static WDCommandData* create_command_object(int packet_data_length)
+{
+	MemoryContext mCxt, oldCxt;
+	WDCommandData*	wdCommand;
+	/* wd command lives in its own memory context */
+	mCxt = AllocSetContextCreate(TopMemoryContext,
+								 "WDCommand",
+								 ALLOCSET_SMALL_MINSIZE,
+								 ALLOCSET_SMALL_INITSIZE,
+								 ALLOCSET_SMALL_MAXSIZE);
+	oldCxt = MemoryContextSwitchTo(mCxt);
+
+	wdCommand = palloc0(sizeof(WDCommandData));
+	wdCommand->memoryContext = mCxt;
+	if (packet_data_length > 0)
+		wdCommand->sourcePacket.data = palloc(packet_data_length);
+	wdCommand->commandPacket.type = WD_NO_MESSAGE;
+	wdCommand->sourcePacket.type = WD_NO_MESSAGE;
+	MemoryContextSwitchTo(oldCxt);
+	return wdCommand;
 }
 
-static bool read_ipc_command_and_process(int sock, bool *remove_socket)
+static bool read_ipc_socket_and_process(int sock, bool *remove_socket)
 {
 	char type;
-	IPC_CMD_PREOCESS_RES res;
 	int data_len,ret;
-	WDIPCCommandData* IPCCommand = NULL;
+	WDCommandData*	ipcCommand;
+	IPC_CMD_PREOCESS_RES res;
 
 	*remove_socket = true;
 
@@ -1517,73 +1631,93 @@ static bool read_ipc_command_and_process(int sock, bool *remove_socket)
 
 	data_len = ntohl(data_len);
 	/* see if we have enough information to process this command */
-	MemoryContext mCxt, oldCxt;
-	mCxt = AllocSetContextCreate(TopMemoryContext,
-								 "WDIPCCommand",
-								 ALLOCSET_SMALL_MINSIZE,
-								 ALLOCSET_SMALL_INITSIZE,
-								 ALLOCSET_SMALL_MAXSIZE);
-	oldCxt = MemoryContextSwitchTo(mCxt);
-	
-	IPCCommand = palloc0(sizeof(WDIPCCommandData));
-	
-	IPCCommand->issueing_sock = sock;
-	IPCCommand->type = type;
-	gettimeofday(&IPCCommand->issue_time, NULL);
+	ipcCommand = create_command_object(data_len);
+	ipcCommand->sourceIPCSocket = sock;
+	ipcCommand->commandSource = COMMAND_SOURCE_IPC;
+	ipcCommand->sourcePacket.type = type;
+	ipcCommand->sourcePacket.len = data_len;
+	gettimeofday(&ipcCommand->commandTime, NULL);
 
 	if (data_len > 0)
 	{
-		IPCCommand->data_buf = palloc(data_len);
-		if (socket_read(sock, IPCCommand->data_buf , data_len, 0) <= 0)
+		if (socket_read(sock, ipcCommand->sourcePacket.data , data_len, 0) <= 0)
 		{
 			ereport(LOG,
 				(errmsg("error reading IPC from socket"),
 					 errdetail("read from socket failed with error \"%s\"",strerror(errno))));
-			MemoryContextDelete(mCxt);
 			return false;
 		}
 	}
-	else
-		IPCCommand->data_buf = NULL;
-
-	IPCCommand->nodeResults = NULL;
-	IPCCommand->memoryContext = mCxt;
-	IPCCommand->data_len = data_len;
-	MemoryContextSwitchTo(oldCxt);
 
-	res = process_IPC_command(IPCCommand);
+	res = process_IPC_command(ipcCommand);
 	if (res == IPC_CMD_PROCESSING)
 	{
 		/*
 		 * The command still needs further processing
 		 * store it in the list
 		 */
+		MemoryContext oldCxt;
 		*remove_socket = false;
 		oldCxt = MemoryContextSwitchTo(TopMemoryContext);
-		g_cluster.ipc_commands = lappend(g_cluster.ipc_commands,IPCCommand);
+		g_cluster.ipc_commands = lappend(g_cluster.ipc_commands,ipcCommand);
 		MemoryContextSwitchTo(oldCxt);
 		return true;
 	}
-	if (res == IPC_CMD_ERROR)
-		ereport(NOTICE,
-				(errmsg("error processing IPC from socket")));
+	else if (res != IPC_CMD_COMPLETE)
+	{
+		char res_type;
+		char *data = NULL;
+		int data_len = 0;
+		switch (res) {
+			case IPC_CMD_TRY_AGAIN:
+				res_type = WD_IPC_CMD_CLUSTER_IN_TRAN;
+				break;
+			case IPC_CMD_ERROR:
+				ereport(NOTICE,
+						(errmsg("error processing IPC from socket")));
+				res_type = WD_IPC_CMD_RESULT_BAD;
+				break;
+			case IPC_CMD_OK:
+				res_type = WD_IPC_CMD_RESULT_OK;
+				break;
+			default:
+				res_type = WD_IPC_CMD_RESULT_BAD;
+				ereport(NOTICE,
+						(errmsg("unexpected IPC processing result")));
+				break;
+		}
+		if (ipcCommand->errorMessage)
+		{
+			data = get_wd_simple_message_json(ipcCommand->errorMessage);
+			data_len = strlen(data) + 1;
+		}
+		
+		if (write_ipc_command_with_result_data(ipcCommand, res_type, data, data_len))
+		{
+			ereport(NOTICE,
+					(errmsg("error writing to IPC socket")));
+		}
+		if (data)
+			pfree(data);
+	}
 	
-	/* Delete the ipcCommand structure,
+	/* Delete the Command structure,
 	 * it is as simple as to delete the memory context
 	 */
-	MemoryContextDelete(mCxt);
+	MemoryContextDelete(ipcCommand->memoryContext);
 	return (res != IPC_CMD_ERROR);
 }
 
-static IPC_CMD_PREOCESS_RES process_IPC_command(WDIPCCommandData* ipcCommand)
+static IPC_CMD_PREOCESS_RES process_IPC_command(WDCommandData* ipcCommand)
 {
 	/* authenticate the client first */
 	if (check_and_report_IPC_authentication(ipcCommand) == false)
 	{
+		/* authentication error is already reported to the caller*/
 		return IPC_CMD_ERROR;
 	}
 
-	switch(ipcCommand->type)
+	switch(ipcCommand->sourcePacket.type)
 	{
 
 		case WD_NODE_STATUS_CHANGE_COMMAND:
@@ -1592,7 +1726,7 @@ static IPC_CMD_PREOCESS_RES process_IPC_command(WDIPCCommandData* ipcCommand)
 
 		case WD_REGISTER_FOR_NOTIFICATION:
 			/* Add this socket to the notify socket list*/
-			g_cluster.notify_clients = lappend_int(g_cluster.notify_clients, ipcCommand->issueing_sock);
+			g_cluster.notify_clients = lappend_int(g_cluster.notify_clients, ipcCommand->sourceIPCSocket);
 			/* The command is completed successfully */
 			return IPC_CMD_COMPLETE;
 			break;
@@ -1601,8 +1735,11 @@ static IPC_CMD_PREOCESS_RES process_IPC_command(WDIPCCommandData* ipcCommand)
 			return process_IPC_nodeList_command(ipcCommand);
 			break;
 
-		case WD_FUNCTION_COMMAND:
-			return process_IPC_replicate_variable(ipcCommand);
+		case WD_IPC_FAILOVER_COMMAND:
+			return process_IPC_failover_command(ipcCommand);
+
+		case WD_IPC_ONLINE_RECOVERY_COMMAND:
+			return process_IPC_online_recovery(ipcCommand);
 			break;
 
 		case WD_FAILOVER_CMD_SYNC_REQUEST:
@@ -1612,38 +1749,24 @@ static IPC_CMD_PREOCESS_RES process_IPC_command(WDIPCCommandData* ipcCommand)
 			return process_IPC_data_request_from_master(ipcCommand);
 
 		default:
-		{
-			char* error_json;
-			ereport(LOG,
-					(errmsg("invalid IPC command type %c",ipcCommand->type)));
-
-			error_json = get_wd_simple_error_message_json("unknown IPC command type");
-			if (write_ipc_command_with_result_data(ipcCommand, WD_IPC_CMD_RESULT_BAD,
-												   error_json, strlen(error_json) +1))
-			{
-				ereport(LOG,
-						(errmsg("failed to forward error message of process node list command to IPC socket")));
-			}
-			pfree(error_json);
-
-		}
+			ipcCommand->errorMessage = MemoryContextStrdup(ipcCommand->memoryContext,"unknown IPC command type");
 			break;
 	}
 	return IPC_CMD_ERROR;
 }
 
 
-static IPC_CMD_PREOCESS_RES process_IPC_nodeList_command(WDIPCCommandData* IPCCommand)
+static IPC_CMD_PREOCESS_RES process_IPC_nodeList_command(WDCommandData* ipcCommand)
 {
 	/* get the json for node list */
 	JsonNode* jNode = NULL;
 	int NodeID = -1;
 	bool ret;
 
-	if (IPCCommand->data_len <= 0 || IPCCommand->data_buf == NULL)
+	if (ipcCommand->sourcePacket.len <= 0 || ipcCommand->sourcePacket.data == NULL)
 		return IPC_CMD_ERROR;
 
-	json_value *root = json_parse(IPCCommand->data_buf,IPCCommand->data_len);
+	json_value *root = json_parse(ipcCommand->sourcePacket.data,ipcCommand->sourcePacket.len);
 	/* The root node must be object */
 	if (root == NULL || root->type != json_object)
 	{
@@ -1661,25 +1784,23 @@ static IPC_CMD_PREOCESS_RES process_IPC_nodeList_command(WDIPCCommandData* IPCCo
 
 	json_value_free(root);
 	jNode = get_node_list_json(NodeID);
-	ret = write_ipc_command_with_result_data(IPCCommand, WD_IPC_CMD_RESULT_OK,
+	ret = write_ipc_command_with_result_data(ipcCommand, WD_IPC_CMD_RESULT_OK,
 											 jw_get_json_string(jNode), jw_get_json_length(jNode) +1);
 	jw_destroy(jNode);
-	if (ret == false)
-		return IPC_CMD_ERROR;
 	return IPC_CMD_COMPLETE;
 }
 
-static IPC_CMD_PREOCESS_RES process_IPC_nodeStatusChange_command(WDIPCCommandData* IPCCommand)
+static IPC_CMD_PREOCESS_RES process_IPC_nodeStatusChange_command(WDCommandData* ipcCommand)
 {
 	int nodeStatus;
 	int nodeID;
 	char *message;
 	bool ret;
 
-	if (IPCCommand->data_len <= 0 || IPCCommand->data_buf == NULL)
+	if (ipcCommand->sourcePacket.len <= 0 || ipcCommand->sourcePacket.data == NULL)
 		return IPC_CMD_ERROR;
 
-	ret = parse_node_status_json(IPCCommand->data_buf, IPCCommand->data_len, &nodeID, &nodeStatus, &message);
+	ret = parse_node_status_json(ipcCommand->sourcePacket.data, ipcCommand->sourcePacket.len, &nodeID, &nodeStatus, &message);
 	
 	if (ret == false)
 	{
@@ -1752,342 +1873,591 @@ static bool fire_node_status_event(int nodeID, int nodeStatus)
 	return true;
 }
 
-
-
-static IPC_CMD_PREOCESS_RES process_IPC_replicate_variable(WDIPCCommandData* IPCCommand)
+static WDFailoverObject* get_failover_object_by_id(unsigned int failoverID)
 {
-	char res_type = WD_IPC_CMD_RESULT_BAD;
-
-	if (get_local_node_state() == WD_STANDBY ||
-		get_local_node_state() == WD_COORDINATOR)
+	ListCell *lc;
+	foreach(lc, g_cluster.wdCurrentFailovers)
 	{
-		IPC_CMD_PREOCESS_RES execute_res = execute_replicate_command(IPCCommand);
-
-		if (execute_res == IPC_CMD_COMPLETE)
+		WDFailoverObject* failoverObj = lfirst(lc);
+		if (failoverObj)
 		{
-			res_type = WD_IPC_CMD_RESULT_OK;
-		}
-		else if (execute_res == IPC_CMD_ERROR)
-		{
-			res_type = WD_IPC_CMD_RESULT_BAD;
-		}
-		else /* IPC_CMD_PROCESSING*/
-		{
-			/*
-			 * Just return from the function, Do not reply back to requester at the moment
-			 * as we still need to further process this command
-			 */
-			return execute_res;
+			if (failoverObj->failoverID == failoverID)
+			{
+				return failoverObj;
+			}
 		}
 	}
-	else /* we are not in any stable state at the moment */
+	return NULL;
+}
+static bool remove_failover_object_by_id(unsigned int failoverID)
+{
+	WDFailoverObject* failoverObj = get_failover_object_by_id(failoverID);
+	if (failoverObj)
 	{
-		res_type = WD_IPC_CMD_CLUSTER_IN_TRAN;
+		ereport(DEBUG2,
+				(errmsg("removing failover object with ID:%d",failoverID)));
+		g_cluster.wdCurrentFailovers = list_delete_ptr(g_cluster.wdCurrentFailovers,failoverObj);
+		return true;
 	}
+	return false;
+}
 
-	if (write_ipc_command_with_result_data(IPCCommand, res_type, NULL, 0))
+static WDFailoverObject* get_failover_object(POOL_REQUEST_KIND reqKind, int nodesCount, int *nodeList)
+{
+	ListCell *lc;
+	foreach(lc, g_cluster.wdCurrentFailovers)
 	{
-		/*
-		 * This is the complete lifecycle of command.
-		 * we are done with it
-		 */
-		
-		return IPC_CMD_COMPLETE;
+		WDFailoverObject* failoverObj = lfirst(lc);
+		if (failoverObj)
+		{
+			if (failoverObj->reqKind == reqKind && failoverObj->nodesCount == nodesCount)
+			{
+				bool equal = true;
+				int i;
+				for (i=0; i< nodesCount; i++)
+				{
+					/*we expect both list have same order ???*/
+					if(nodeList[i] != failoverObj->nodeList[i])
+					{
+						equal = false;
+						break;
+					}
+				}
+				if (equal)
+					return failoverObj;
+			}
+		}
 	}
-	return IPC_CMD_ERROR;
+	return NULL;
 }
 
-static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDIPCCommandData *IPCCommand)
+static void process_remote_failover_command_on_coordinator(WatchdogNode* wdNode, WDPacketData* pkt)
 {
-	char res_type = WD_IPC_CMD_RESULT_BAD;
-	/*
-	 * if cluster or myself is not in stable state
-	 * just return cluster in transaction
-	 */
-	ereport(LOG,
-			(errmsg("processing master node data request from IPC socket")));
-
-	IPCCommand->type = WD_GET_MASTER_DATA_REQUEST;
-	if (get_local_node_state() == WD_STANDBY)
+	if (get_local_node_state() != WD_COORDINATOR)
 	{
-		/* I am a standby node, Just forward the request to coordinator */
+		/* only lock holder can resign itself */
+		reply_with_minimal_message(wdNode, WD_ERROR_MESSAGE, pkt);
+	}
+	else
+	{
+		IPC_CMD_PREOCESS_RES res;
+		WDCommandData* ipcCommand = create_command_object(pkt->len);
+		ipcCommand->sourcePacket.type = pkt->type;
+		ipcCommand->sourcePacket.len = pkt->len;
+		ipcCommand->sourcePacket.command_id = pkt->command_id;
 
-		WDPacketData wdPacket;
-		init_wd_packet(&wdPacket);
-		set_message_type(&wdPacket, WD_GET_MASTER_DATA_REQUEST);
-		set_next_commandID_in_message(&wdPacket);
-		set_message_data(&wdPacket, IPCCommand->data_buf , IPCCommand->data_len);
-		/* save the command ID */
-		IPCCommand->internal_command_id = wdPacket.command_id;
-		if (send_message(g_cluster.masterNode, &wdPacket) <= 0)
+		if (pkt->len > 0)
+			memcpy(ipcCommand->sourcePacket.data, pkt->data, pkt->len);
+
+		ipcCommand->commandSource = COMMAND_SOURCE_REMOTE;
+		ipcCommand->sourceWdNode = wdNode;
+		gettimeofday(&ipcCommand->commandTime, NULL);
+
+		res = process_failover_command_on_coordinator(ipcCommand);
+		if (res == IPC_CMD_PROCESSING)
 		{
-			ereport(LOG,
-				(errmsg("failed to process master node data request from IPC socket"),
-					 errdetail("failed to forward the request to master watchdog node \"%s\"",g_cluster.masterNode->nodeName)));
-			/* we have failed to send to any node, return lock failed  */
-			res_type = WD_IPC_CMD_RESULT_BAD;
+			MemoryContext oldCxt = MemoryContextSwitchTo(TopMemoryContext);
+			g_cluster.ipc_commands = lappend(g_cluster.ipc_commands,ipcCommand);
+			MemoryContextSwitchTo(oldCxt);
 		}
 		else
 		{
-			/*
-			 * we need to wait for the result
-			 */
-			ereport(LOG,
-				(errmsg("data request from IPC socket is forwarded to master watchdog node \"%s\"",g_cluster.masterNode->nodeName),
-					 errdetail("waiting for the reply from master node...")));
-
-			return IPC_CMD_PROCESSING;
+			cleanUpIPCCommand(ipcCommand);
 		}
 	}
-	else if (get_local_node_state() == WD_COORDINATOR)
+}
+
+static IPC_CMD_PREOCESS_RES process_IPC_failover_command_on_coordinator(WDCommandData* ipcCommand)
+{
+	if (get_local_node_state() != WD_COORDINATOR)
+		return IPC_CMD_ERROR; /* should never hapen*/
+	return process_failover_command_on_coordinator(ipcCommand);
+}
+
+
+static bool reply_to_failove_command(WDCommandData* ipcCommand, WDFailoverCMDResults cmdResult, unsigned int failoverID)
+{
+	bool ret = false;
+	JsonNode* jNode = jw_create_with_object(true);
+	jw_put_int(jNode, WD_FAILOVER_RESULT_KEY, cmdResult);
+	jw_put_int(jNode, WD_FAILOVER_ID_KEY, failoverID);
+	/* create the packet */
+	jw_end_element(jNode);
+	jw_finish_document(jNode);
+
+	if (ipcCommand->commandSource == COMMAND_SOURCE_IPC)
 	{
-		/* This node is itself a master node, So send the empty result with OK tag */
-		res_type = WD_IPC_CMD_RESULT_OK;
+		ret = write_ipc_command_with_result_data(ipcCommand, WD_IPC_CMD_RESULT_OK,
+												 jw_get_json_string(jNode), jw_get_json_length(jNode) +1);
 	}
-	else /* we are not in any stable state at the moment */
+	else if (ipcCommand->commandSource == COMMAND_SOURCE_REMOTE)
 	{
-		res_type = WD_IPC_CMD_CLUSTER_IN_TRAN;
-	}
+		reply_with_message(ipcCommand->sourceWdNode, WD_CMD_REPLY_IN_DATA,
+						   jw_get_json_string(jNode), jw_get_json_length(jNode) + 1,
+						   &ipcCommand->sourcePacket);
 
-	if (write_ipc_command_with_result_data(IPCCommand, res_type, NULL, 0))
-	{
-		/*
-		 * This is the complete lifecycle of command.
-		 * we are done with it
-		 */
-		return IPC_CMD_COMPLETE;
 	}
-	return IPC_CMD_ERROR;
 
+	jw_destroy(jNode);
+	return ret;
 }
 
-static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDIPCCommandData *IPCCommand)
+/*
+ * The Function forwards the failover command to all standby nodes.
+ */
+static IPC_CMD_PREOCESS_RES process_failover_command_on_coordinator(WDCommandData* ipcCommand)
 {
-	char res_type = WD_IPC_CMD_RESULT_BAD;
+	char* func_name;
+	int node_count = 0;
+	int *node_id_list = NULL;
+	bool ret = false;
+	WDFailoverObject* failoverObj;
+	POOL_REQUEST_KIND reqKind;
+
+	if (get_local_node_state() != WD_COORDINATOR)
+		return IPC_CMD_ERROR; /* should never happen*/
+
 	/*
-	 * if cluster or myself is not in stable state
-	 * just return cluster in transaction
+	 * The coordinator node
+	 * Forward this command to all standby nodes.
+	 * Ask the caller to proceed with failover
+	 * but first check if this failover is already requested
+	 * by some other node.
 	 */
-	ereport(LOG,
-		(errmsg("processing sync request from IPC socket")));
 
-	IPCCommand->type = WD_FAILOVER_CMD_SYNC_REQUEST;
-	if (get_local_node_state() == WD_STANDBY)
+	ret = parse_wd_node_function_json(ipcCommand->sourcePacket.data, ipcCommand->sourcePacket.len,
+									  &func_name, &node_id_list, &node_count);
+	if (ret == false)
 	{
-		/* I am a standby node, Just forward the request to coordinator */
-		
-		WDPacketData wdPacket;
-		init_wd_packet(&wdPacket);
-		set_message_type(&wdPacket, WD_FAILOVER_CMD_SYNC_REQUEST);
-		set_next_commandID_in_message(&wdPacket);
-		set_message_data(&wdPacket, IPCCommand->data_buf , IPCCommand->data_len);
-		/* save the command ID */
-		IPCCommand->internal_command_id = wdPacket.command_id;
-		if (send_message(g_cluster.masterNode, &wdPacket) <= 0)
-		{
-			ereport(LOG,
-				(errmsg("failed to process sync request from IPC socket"),
-					 errdetail("failed to forward the request to master watchdog node \"%s\"",g_cluster.masterNode->nodeName)));
-			/* we have failed to send to any node, return lock failed  */
-			res_type = WD_IPC_CMD_RESULT_BAD;
-		}
-		else
-		{
-			/*
-			 * we need to wait for the result
-			 */
-			ereport(LOG,
-				(errmsg("sync request from IPC socket is forwarded to master watchdog node \"%s\"",g_cluster.masterNode->nodeName),
-					 errdetail("waiting for the reply from master node...")));
+		ereport(LOG,(
+			errmsg("failed to process failover command."),
+					errdetail("unable to parse the command data.")));
+		reply_to_failove_command(ipcCommand, FAILOVER_RES_INVALID_FUNCTION, 0);
+		return IPC_CMD_COMPLETE;
+	}
 
-			return IPC_CMD_PROCESSING;
-		}
+	if (strcasecmp(WD_FUNCTION_FAILBACK_REQUEST, func_name) == 0)
+		reqKind = NODE_UP_REQUEST;
+	else if (strcasecmp(WD_FUNCTION_DEGENERATE_REQUEST, func_name) == 0)
+		reqKind = NODE_DOWN_REQUEST;
+	else if (strcasecmp(WD_FUNCTION_PROMOTE_REQUEST, func_name) == 0)
+		reqKind = PROMOTE_NODE_REQUEST;
+	else
+	{
+		reply_to_failove_command(ipcCommand, FAILOVER_RES_INVALID_FUNCTION, 0);
+		return IPC_CMD_COMPLETE;
 	}
-	else if (get_local_node_state() == WD_COORDINATOR)
+	if (get_cluster_node_count() == 0)
 	{
 		/*
-		 * If I am coordinator, Just process the request locally
+		 * Since I am the only node in the cluster so nothing
+		 * we need to do here
 		 */
-		process_failover_command_sync_requests(g_cluster.localNode, NULL, IPCCommand);
+		ereport(LOG,(
+			errmsg("Pgpool-II has a go ahead for failover command."),
+				errdetail("I am the only node in the watchog cluster.")));
+		reply_to_failove_command(ipcCommand, FAILOVER_RES_PROCEED, 0);
 		return IPC_CMD_COMPLETE;
 	}
-	else /* we are not in any stable state at the moment */
-	{
-		res_type = WD_IPC_CMD_CLUSTER_IN_TRAN;
-	}
 
-	if (write_ipc_command_with_result_data(IPCCommand, res_type, NULL, 0))
+	if (ipcCommand->commandSource == COMMAND_SOURCE_REMOTE  && Req_info->switching)
 	{
 		/*
-		 * This is the complete lifecycle of command.
-		 * we are done with it
+		 * check if the failover is allowed before doing anything
 		 */
+		ereport(LOG,
+			(errmsg("sending watchdog response"),
+				 errdetail("failover request from other pgpool is canceled because of switching")));
+		reply_to_failove_command(ipcCommand, FAILOVER_RES_NOT_ALLOWED, 0);
 		return IPC_CMD_COMPLETE;
 	}
-	return IPC_CMD_ERROR;
-}
 
-static int node_has_requested_for_interlocking(WatchdogNode* wdNode, WDPacketData* pkt)
-{
-	/* only coordinator(master) node can process this request */
-	if (get_local_node_state() == WD_COORDINATOR)
+	/*
+	 * check if the same failover is already issued to the main
+	 * process
+	 */
+	failoverObj = get_failover_object(reqKind, node_count, node_id_list);
+	if (failoverObj)
 	{
-		/* check if we already have no lockholder node */
-		if (g_cluster.lockHolderNode == NULL || g_cluster.lockHolderNode == wdNode)
-		{
-			if (wdNode == g_cluster.localNode)
-			{
-				g_cluster.lockHolderNode = wdNode;
-				/* TODO inform all cluster about the new lock holder */
-				return true;
-			}
-			/* reply the node with success message */
-			else if (reply_with_minimal_message(wdNode, WD_ACCEPT_MESSAGE, pkt))
-			{
-				g_cluster.lockHolderNode = wdNode;
-				/* TODO inform all cluster about the new lock holder */
-				return true;
-			}
+		ereport(LOG,
+			(errmsg("ignoring the failover[%s] request, because same failover request is already in progress",func_name)));
+
+		/* Same failover is already in progress */
+		reply_to_failove_command(ipcCommand, FAILOVER_RES_ALREADY_ISSUED, 0);
+		return IPC_CMD_COMPLETE;
+	}
+	else
+	{
+		MemoryContext oldCxt;
+		ereport(DEBUG2,
+				(errmsg("no similar failover in progress")));
+		/*
+		 * okay now ask all nodes to start failover
+		 */
+		wd_packet_shallow_copy(&ipcCommand->sourcePacket, &ipcCommand->commandPacket);
+		ipcCommand->commandPacket.type = WD_REMOTE_FAILOVER_REQUEST;
+		set_next_commandID_in_message(&ipcCommand->commandPacket);
+
+		oldCxt = MemoryContextSwitchTo(TopMemoryContext);
+		/* No similar failover is in progress */
+		failoverObj = palloc0(sizeof(WDFailoverObject));
+		failoverObj->reqKind = reqKind;
+		failoverObj->nodesCount = node_count;
+		if (node_count > 0)
+		{
+			failoverObj->nodeList = palloc(sizeof(int) * node_count);
+			memcpy(failoverObj->nodeList, node_id_list, sizeof(int) * node_count);
+		}
+		failoverObj->failoverID = ipcCommand->commandPacket.command_id; /* use command id as failover id */
+		gettimeofday(&failoverObj->startTime, NULL);
+		failoverObj->wdRequestingNode = g_cluster.localNode;
+		g_cluster.wdCurrentFailovers = lappend(g_cluster.wdCurrentFailovers,failoverObj);
+
+		MemoryContextSwitchTo(oldCxt);
+		/* We may also need to send the Accept message here for remote node */
+
+		ipcCommand->sendToNode = NULL; /* command needs to be sent to all nodes */
+
+		/* see if there is any node we want to send to */
+		send_command_packet_to_remote_nodes(ipcCommand, false);
+
+		/* For a moment just think it is successfully sent to all nodes.*/
+		if (ipcCommand->commandSource == COMMAND_SOURCE_IPC)
+		{
+			reply_to_failove_command(ipcCommand, FAILOVER_RES_PROCEED, failoverObj->failoverID);
+			return IPC_CMD_COMPLETE;
 		}
 		else
 		{
-			reply_with_minimal_message(wdNode, WD_REJECT_MESSAGE, pkt);
+			process_wd_command_function(ipcCommand->sourceWdNode, &ipcCommand->sourcePacket,
+										func_name, node_count, node_id_list, failoverObj->failoverID);
+			if (get_cluster_node_count() == 1)
+			{
+				/* Since its just 2 nodes cluster, and the only other
+				 * node is the one that actually issued the failover
+				 * so the command actually completes here
+				 */
+				return IPC_CMD_COMPLETE;
+			}
 		}
 	}
-	else
-	{
-		reply_with_minimal_message(wdNode, WD_ERROR_MESSAGE, pkt);
-	}
-	return false;
+
+	return IPC_CMD_PROCESSING;
 }
 
-/*
- * process_failover_command_sync_requests()
- * the function is the main processor of all interlocking related requests.
- * it parses the request json and executes the requested intelocking command
- */
-static void process_failover_command_sync_requests(WatchdogNode* wdNode, WDPacketData* pkt, WDIPCCommandData* ipcCommand)
+static IPC_CMD_PREOCESS_RES process_IPC_failover_command(WDCommandData* ipcCommand)
 {
-	
-	WDFailoverCMDResults res = FAILOVER_RES_TRANSITION;
-	JsonNode* jNode = NULL;
-	int failoverLockID = -1;
-
-	/* only coordinator(master) node can process this request */
 	if (get_local_node_state() == WD_COORDINATOR)
 	{
-		char* json_data = NULL;
-		int data_len = 0;
-		json_value *root = NULL;
-		char* syncRequestType = NULL;
-		
-		/* We need to identify failover command type and sync function */
-		if (pkt)
+		return process_IPC_failover_command_on_coordinator(ipcCommand);
+	}
+	else if (get_local_node_state() == WD_STANDBY)
+	{
+		/* I am a standby node, Just forward the request to coordinator */
+
+		wd_packet_shallow_copy(&ipcCommand->sourcePacket, &ipcCommand->commandPacket);
+		set_next_commandID_in_message(&ipcCommand->commandPacket);
+
+		ipcCommand->sendToNode = g_cluster.masterNode; /* command needs to be sent to all nodes */
+		if (send_command_packet_to_remote_nodes(ipcCommand, true) <= 0)
 		{
-			json_data = pkt->data;
-			data_len = pkt->len;
+			ereport(LOG,
+				(errmsg("failed to process failover request from IPC socket"),
+					 errdetail("failed to forward the request to master watchdog node \"%s\"",g_cluster.masterNode->nodeName)));
+			/* we have failed to send to any node, return lock failed  */
+			return IPC_CMD_ERROR;
 		}
 		else
 		{
-			json_data = ipcCommand->data_buf;
-			data_len = ipcCommand->data_len;
+			/*
+			 * we need to wait for the result
+			 */
+			ereport(LOG,
+				(errmsg("failover request from IPC socket is forwarded to master watchdog node \"%s\"",g_cluster.masterNode->nodeName),
+					 errdetail("waiting for the reply from master node...")));
+			return IPC_CMD_PROCESSING;
 		}
+	}
+	/* we are not in stable state at the moment */
+	return IPC_CMD_ERROR;
+}
 
-		if (data_len > 0 && json_data)
+static IPC_CMD_PREOCESS_RES process_IPC_online_recovery(WDCommandData* ipcCommand)
+{
+	if (get_local_node_state() == WD_STANDBY ||
+		get_local_node_state() == WD_COORDINATOR)
+	{
+		/* save the hassel if I am the only alive node */
+		if (get_cluster_node_count() == 0)
+			return IPC_CMD_OK;
+		
+		wd_packet_shallow_copy(&ipcCommand->sourcePacket, &ipcCommand->commandPacket);
+		set_next_commandID_in_message(&ipcCommand->commandPacket);
+		
+		ipcCommand->sendToNode = NULL; /* command needs to be sent to all nodes */
+		if (send_command_packet_to_remote_nodes(ipcCommand, true) <= 0)
 		{
-			root = json_parse(json_data,data_len);
-			if (root && root->type == json_object)
-			{
-				syncRequestType = json_get_string_value_for_key(root, "SyncRequestType");
-				json_get_int_value_for_key(root, "FailoverLockID", &failoverLockID);
-			}
-			else
-			{
-				ereport(LOG,
-						(errmsg("unable to parse json data of interlocking command")));
-			}
+			ereport(LOG,
+					(errmsg("failed to process failover request from IPC socket"),
+					 errdetail("failed to forward the request to remote nodes")));
+			/* we have failed to send to any node, return lock failed  */
+			return IPC_CMD_ERROR;
 		}
-		if (syncRequestType)
+		return IPC_CMD_PROCESSING;
+	}
+	/* we are not in any stable state at the moment */
+	return IPC_CMD_TRY_AGAIN;
+}
+
+static IPC_CMD_PREOCESS_RES process_IPC_data_request_from_master(WDCommandData *ipcCommand)
+{
+	/*
+	 * if cluster or myself is not in stable state
+	 * just return cluster in transaction
+	 */
+	ereport(LOG,
+			(errmsg("processing master node data request from IPC socket")));
+
+	if (get_local_node_state() == WD_STANDBY)
+	{
+		/*
+		 * set the command id in the IPC packet before forwaring
+		 * it on the watchdog socket
+		 */
+		wd_packet_shallow_copy(&ipcCommand->sourcePacket, &ipcCommand->commandPacket);
+		set_next_commandID_in_message(&ipcCommand->commandPacket);
+
+		ipcCommand->sendToNode = g_cluster.masterNode;
+		if (send_command_packet_to_remote_nodes(ipcCommand, true) <= 0)
 		{
+			ereport(LOG,
+				(errmsg("failed to process master node data request from IPC socket"),
+					 errdetail("failed to forward the request to master watchdog node \"%s\"",g_cluster.masterNode->nodeName)));
+			/* we have failed to send to any node, return lock failed  */
+			return IPC_CMD_ERROR;
+		}
+		else
+		{
+			/*
+			 * we need to wait for the result
+			 */
+			ereport(LOG,
+				(errmsg("data request from IPC socket is forwarded to master watchdog node \"%s\"",g_cluster.masterNode->nodeName),
+					 errdetail("waiting for the reply from master node...")));
 
-			if (strcasecmp(WD_REQ_FAILOVER_START, syncRequestType) == 0)
-				res = node_is_asking_for_failover_start(wdNode, pkt);
+			return IPC_CMD_PROCESSING;
+		}
+	}
+	else if (get_local_node_state() == WD_COORDINATOR)
+	{
+		/* This node is itself a master node, So send the empty result with OK tag */
+		return IPC_CMD_OK;
+	}
+	
+	/* we are not in any stable state at the moment */
 
-			else if (strcasecmp(WD_REQ_FAILOVER_END, syncRequestType) == 0)
-				res = node_is_asking_for_failover_end(wdNode, pkt);
+	return IPC_CMD_TRY_AGAIN;
+}
 
-			else if (strcasecmp(WD_REQ_FAILOVER_RELEASE_LOCK, syncRequestType) == 0)
-				res = node_is_asking_for_failover_lock_release(wdNode, pkt, failoverLockID);
+static IPC_CMD_PREOCESS_RES process_IPC_failover_cmd_synchronise(WDCommandData *ipcCommand)
+{
+	/*
+	 * if cluster or myself is not in stable state
+	 * just return cluster in transaction
+	 */
+	ereport(LOG,
+		(errmsg("processing sync request from IPC socket")));
 
-			else if (strcasecmp(WD_REQ_FAILOVER_LOCK_STATUS, syncRequestType) == 0)
-				res = node_is_asking_for_failover_lock_status(wdNode, pkt, failoverLockID);
+	if (get_local_node_state() == WD_STANDBY)
+	{
+		/* I am a standby node, Just forward the request to coordinator */
+		wd_packet_shallow_copy(&ipcCommand->sourcePacket, &ipcCommand->commandPacket);
+		set_next_commandID_in_message(&ipcCommand->commandPacket);
 
-			else
-				res = FAILOVER_RES_ERROR;
+		ipcCommand->sendToNode = g_cluster.masterNode;
+		if (send_command_packet_to_remote_nodes(ipcCommand, true) <= 0)
+		{
+			ereport(LOG,
+				(errmsg("failed to process sync request from IPC socket"),
+					 errdetail("failed to forward the request to master watchdog node \"%s\"",g_cluster.masterNode->nodeName)));
+			/* we have failed to send to any node, return lock failed  */
+			return IPC_CMD_ERROR;
 		}
 		else
 		{
+			/*
+			 * wait for the result
+			 */
 			ereport(LOG,
-					(errmsg("invalid json data"),
-					 errdetail("unable to find interlocking command type")));
-			res = FAILOVER_RES_ERROR;
-		}
+				(errmsg("sync request from IPC socket is forwarded to master watchdog node \"%s\"",g_cluster.masterNode->nodeName),
+					 errdetail("waiting for the reply from master node...")));
 
-		if (root)
-		{
-			json_value_free(root);
+			return IPC_CMD_PROCESSING;
 		}
 	}
-	else
+	else if (get_local_node_state() == WD_COORDINATOR)
 	{
-		/* I am not the coordinator node. So just return an error */
-		res = FAILOVER_RES_ERROR;
+		/*
+		 * If I am coordinator, Just process the request locally
+		 */
+		return process_failover_locking_requests_on_cordinator(ipcCommand);
 	}
 
-	if (res != FAILOVER_RES_ERROR)
-	{
-		/* create the json result */
-		jNode = jw_create_with_object(true);
-		/* add the node count */
-		jw_put_int(jNode, "FailoverLockID", failoverLockID);
-		jw_put_int(jNode, "InterlockingResult", res);
-		/* create the packet */
-		jw_end_element(jNode);
-		jw_finish_document(jNode);
-	}
+	/* we are not in any stable state at the moment */
+	return IPC_CMD_TRY_AGAIN;
+}
 
-	if (wdNode != g_cluster.localNode)
+static int node_has_requested_for_interlocking(WatchdogNode* wdNode, WDPacketData* pkt)
+{
+	/* only coordinator(master) node can process this request */
+	if (get_local_node_state() == WD_COORDINATOR)
 	{
-		if (jNode == NULL)
+		/* check if we already have no lockholder node */
+		if (g_cluster.lockHolderNode == NULL || g_cluster.lockHolderNode == wdNode)
 		{
-			reply_with_minimal_message(wdNode, WD_ERROR_MESSAGE, pkt);
+			if (wdNode == g_cluster.localNode)
+			{
+				g_cluster.lockHolderNode = wdNode;
+				/* TODO inform all cluster about the new lock holder */
+				return true;
+			}
+			/* reply the node with success message */
+			else if (reply_with_minimal_message(wdNode, WD_ACCEPT_MESSAGE, pkt))
+			{
+				g_cluster.lockHolderNode = wdNode;
+				/* TODO inform all cluster about the new lock holder */
+				return true;
+			}
 		}
 		else
 		{
-			reply_with_message(wdNode, WD_DATA_MESSAGE, jw_get_json_string(jNode), jw_get_json_length(jNode) + 1, pkt);
+			reply_with_minimal_message(wdNode, WD_REJECT_MESSAGE, pkt);
 		}
 	}
 	else
 	{
-		/* reply on IPC Socket */
-		bool ret;
-		if (jNode != NULL)
+		reply_with_minimal_message(wdNode, WD_ERROR_MESSAGE, pkt);
+	}
+	return false;
+}
+
+static void process_remote_failover_locking_request(WatchdogNode* wdNode, WDPacketData* pkt)
+{
+	if (get_local_node_state() != WD_COORDINATOR)
+	{
+		/* only lock holder can resign itself */
+		reply_with_minimal_message(wdNode, WD_ERROR_MESSAGE, pkt);
+	}
+	else
+	{
+		IPC_CMD_PREOCESS_RES res;
+		WDCommandData* ipcCommand = create_command_object(pkt->len);
+		ipcCommand->sourcePacket.type = pkt->type;
+		ipcCommand->sourcePacket.len = pkt->len;
+		ipcCommand->sourcePacket.command_id = pkt->command_id;
+		if (pkt->len > 0)
+			memcpy(ipcCommand->sourcePacket.data, pkt->data, pkt->len);
+
+		ipcCommand->commandSource = COMMAND_SOURCE_REMOTE;
+		ipcCommand->sourceWdNode = wdNode;
+		gettimeofday(&ipcCommand->commandTime, NULL);
+
+		res = process_failover_locking_requests_on_cordinator(ipcCommand);
+		if (res == IPC_CMD_PROCESSING)
 		{
-			ret = write_ipc_command_with_result_data(ipcCommand, WD_IPC_CMD_RESULT_OK,
-											   jw_get_json_string(jNode), jw_get_json_length(jNode) +1);
-			jw_destroy(jNode);
+			MemoryContext oldCxt = MemoryContextSwitchTo(TopMemoryContext);
+			g_cluster.ipc_commands = lappend(g_cluster.ipc_commands,ipcCommand);
+			MemoryContextSwitchTo(oldCxt);
 		}
 		else
 		{
-			ret =write_ipc_command_with_result_data(ipcCommand, WD_IPC_CMD_RESULT_BAD, NULL, 0);
+			cleanUpIPCCommand(ipcCommand);
 		}
+	}
+}
+
+
+/*
+ * process_failover_locking_requests_on_cordinator()
+ * the function is the main processor of all interlocking related requests.
+ * it parses the request json and executes the requested intelocking command
+ */
+static IPC_CMD_PREOCESS_RES process_failover_locking_requests_on_cordinator(WDCommandData* ipcCommand)
+{
+	
+	WDFailoverCMDResults res = FAILOVER_RES_TRANSITION;
+	json_value* root;
+	int failoverLockID = -1;
+	unsigned int failoverID = 0;
+	char *syncRequestType;
+	WatchdogNode* wdNode;
+
+	if (get_local_node_state() != WD_COORDINATOR)
+		return IPC_CMD_ERROR;
 
-		if (ret == false)
+	if (ipcCommand->sourcePacket.data == NULL || ipcCommand->sourcePacket.len <= 0)
+	{
+		ereport(LOG,
+				(errmsg("failed to process locking request"),
+				 errdetail("invalid command packet")));
+		reply_to_failove_command(ipcCommand, FAILOVER_RES_INVALID_FUNCTION, failoverID);
+		return IPC_CMD_COMPLETE;
+	}
+	/* parse the json*/
+	root = json_parse(ipcCommand->sourcePacket.data,ipcCommand->sourcePacket.len);
+	if (root && root->type == json_object)
+	{
+		syncRequestType = json_get_string_value_for_key(root, "SyncRequestType");
+		json_get_int_value_for_key(root, "FailoverLockID", &failoverLockID);
+		json_get_int_value_for_key(root, "WDFailoverID", (int*)&failoverID);
+		if (syncRequestType == false)
 		{
 			ereport(LOG,
-					(errmsg("failed to write results for failover sync request to IPC socket")));
+				(errmsg("failed to process locking request"),
+					 errdetail("invalid command packet")));
+			reply_to_failove_command(ipcCommand, FAILOVER_RES_INVALID_FUNCTION, failoverID);
+			return IPC_CMD_COMPLETE;
 		}
 	}
+	else
+	{
+		ereport(LOG,
+			(errmsg("failed to process locking request"),
+				 errdetail("invalid command packet")));
+		reply_to_failove_command(ipcCommand, FAILOVER_RES_INVALID_FUNCTION, failoverID);
+		return IPC_CMD_COMPLETE;
+	}
+	
+	if (ipcCommand->commandSource == COMMAND_SOURCE_IPC)
+		wdNode = g_cluster.localNode;
+	else
+		wdNode = ipcCommand->sourceWdNode;
+
+	if (strcasecmp(WD_REQ_FAILOVER_START, syncRequestType) == 0)
+		res = node_is_asking_for_failover_start(wdNode, &ipcCommand->sourcePacket, failoverID);
+	
+	else if (strcasecmp(WD_REQ_FAILOVER_END, syncRequestType) == 0)
+		res = node_is_asking_for_failover_end(wdNode, &ipcCommand->sourcePacket, failoverID);
+	
+	else if (strcasecmp(WD_REQ_FAILOVER_RELEASE_LOCK, syncRequestType) == 0)
+		res = node_is_asking_for_failover_lock_release(wdNode, &ipcCommand->sourcePacket, failoverLockID, failoverID);
+	
+	else if (strcasecmp(WD_REQ_FAILOVER_LOCK_STATUS, syncRequestType) == 0)
+		res = node_is_asking_for_failover_lock_status(wdNode, &ipcCommand->sourcePacket, failoverLockID, failoverID);
+
+	else
+	{
+		ereport(LOG,
+			(errmsg("failed to process locking request"),
+				 errdetail("invalid command packet")));
+		res = FAILOVER_RES_INVALID_FUNCTION;
+	}
+	reply_to_failove_command(ipcCommand, res, failoverID);
+
+	if (root)
+		json_value_free(root);
+
+	return IPC_CMD_COMPLETE;
 }
 
 /*
@@ -2099,23 +2469,25 @@ static void process_failover_command_sync_requests(WatchdogNode* wdNode, WDPacke
  * Only coordinator/master node can execute the interlocking requests.
  */
 static WDFailoverCMDResults
-node_is_asking_for_failover_start(WatchdogNode* wdNode, WDPacketData* pkt)
+node_is_asking_for_failover_start(WatchdogNode* wdNode, WDPacketData* pkt, unsigned int failoverID)
 {
 	WDFailoverCMDResults res = FAILOVER_RES_TRANSITION;
 
 	ereport(LOG,
-			(errmsg("%s pgpool-II node \"%s\" is requesting to become a lock holder",
+			(errmsg("%s pgpool-II node \"%s\" is requesting to become a lock holder for failover ID: %d",
 					(g_cluster.localNode == wdNode)? "local":"remote",
-					wdNode->nodeName)));
+					wdNode->nodeName, failoverID)));
 
 	/* only coordinator(master) node can process this request */
 	if (get_local_node_state() == WD_COORDINATOR)
 	{
+		/* only the coordinator node can become a lock holder */
+		if (g_cluster.masterNode == wdNode)
 		/* check if we have no node in interlocking or requesting node is itself
 		 * a lock holder node
 		 */
-		if (g_cluster.interlockingNode.lockHolderNode == NULL ||
-			g_cluster.interlockingNode.lockHolderNode == wdNode)
+//		if (g_cluster.interlockingNode.lockHolderNode == NULL ||
+//			g_cluster.interlockingNode.lockHolderNode == wdNode)
 		{
 			int i = 0;
 			/* lock all command locks */
@@ -2135,7 +2507,14 @@ node_is_asking_for_failover_start(WatchdogNode* wdNode, WDPacketData* pkt)
 		{
 			/* some other node is holding the lock */
 			res = FAILOVER_RES_I_AM_NOT_LOCK_HOLDER;
-			ereport(LOG,
+			if (g_cluster.interlockingNode.lockHolderNode == NULL)
+				ereport(LOG,
+						(errmsg("request to become a lock holder is denied to %s pgpool-II node \"%s\"",
+								(g_cluster.localNode == wdNode)? "local":"remote",
+								wdNode->nodeName),
+						 errdetail("only master/coordinator can become a lock holder")));
+			else
+				ereport(LOG,
 					(errmsg("lock holder request denied to %s pgpool-II node \"%s\"",
 							(g_cluster.localNode == wdNode)? "local":"remote",
 							wdNode->nodeName),
@@ -2165,14 +2544,14 @@ node_is_asking_for_failover_start(WatchdogNode* wdNode, WDPacketData* pkt)
  * Only coordinator/master node can execute the interlocking requests.
  */
 static WDFailoverCMDResults
-node_is_asking_for_failover_end(WatchdogNode* wdNode, WDPacketData* pkt)
+node_is_asking_for_failover_end(WatchdogNode* wdNode, WDPacketData* pkt, unsigned int failoverID)
 {
 	WDFailoverCMDResults res = FAILOVER_RES_TRANSITION;
 
 	ereport(LOG,
-			(errmsg("%s pgpool-II node \"%s\" is requesting to resign from a lock holder",
+			(errmsg("%s pgpool-II node \"%s\" is requesting to resign from a lock holder for failover ID %d",
 					(g_cluster.localNode == wdNode)? "local":"remote",
-					wdNode->nodeName)));
+					wdNode->nodeName, failoverID)));
 
 	if (get_local_node_state() == WD_COORDINATOR)
 	{
@@ -2193,6 +2572,10 @@ node_is_asking_for_failover_end(WatchdogNode* wdNode, WDPacketData* pkt)
 					(errmsg("%s pgpool-II node \"%s\" has resigned from the lock holder",
 							(g_cluster.localNode == wdNode)? "local":"remote",
 							wdNode->nodeName)));
+			/* This marks the end of failover. Remove the
+			 * associated failover object
+			 */
+			remove_failover_object_by_id(failoverID);
 		}
 		else /* some other node is holding the lock */
 		{
@@ -2225,15 +2608,16 @@ node_is_asking_for_failover_end(WatchdogNode* wdNode, WDPacketData* pkt)
  * Only coordinator/master node can execute the interlocking requests.
  */
 static WDFailoverCMDResults
-node_is_asking_for_failover_lock_release(WatchdogNode* wdNode, WDPacketData* pkt, WDFailoverLock failoverLock)
+node_is_asking_for_failover_lock_release(WatchdogNode* wdNode, WDPacketData* pkt, WDFailoverLock failoverLock, unsigned int failoverID)
 {
 	WDFailoverCMDResults res = FAILOVER_RES_TRANSITION;
 
 	ereport(LOG,
-			(errmsg("%s pgpool-II node \"%s\" is requesting to release [%s] lock",
+			(errmsg("%s pgpool-II node \"%s\" is requesting to release [%s] lock for failover ID %d",
 					(g_cluster.localNode == wdNode)? "local":"remote",
 					wdNode->nodeName,
-					wd_failover_lock_name[failoverLock])));
+					wd_failover_lock_name[failoverLock],
+					failoverID)));
 
 	if (get_local_node_state() == WD_COORDINATOR)
 	{
@@ -2247,10 +2631,11 @@ node_is_asking_for_failover_lock_release(WatchdogNode* wdNode, WDPacketData* pkt
 				res = FAILOVER_RES_SUCCESS;
 
 				ereport(LOG,
-						(errmsg("%s pgpool-II node \"%s\" has released the [%s] lock",
+						(errmsg("%s pgpool-II node \"%s\" has released the [%s] lock for failover ID %d",
 								(g_cluster.localNode == wdNode)? "local":"remote",
 								wdNode->nodeName,
-								wd_failover_lock_name[failoverLock])));
+								wd_failover_lock_name[failoverLock],
+								failoverID)));
 			}
 			else
 			{
@@ -2287,19 +2672,20 @@ node_is_asking_for_failover_lock_release(WatchdogNode* wdNode, WDPacketData* pkt
  * Only coordinator/master node can execute the interlocking requests.
  */
 static WDFailoverCMDResults
-node_is_asking_for_failover_lock_status(WatchdogNode* wdNode, WDPacketData* pkt, WDFailoverLock failoverLock)
+node_is_asking_for_failover_lock_status(WatchdogNode* wdNode, WDPacketData* pkt, WDFailoverLock failoverLock, unsigned int failoverID)
 {
 	WDFailoverCMDResults res = FAILOVER_RES_TRANSITION;
 
 	ereport(LOG,
-			(errmsg("%s pgpool-II node \"%s\" is checking the status of [%s] lock",
+			(errmsg("%s pgpool-II node \"%s\" is checking the status of [%s] lock for failover ID %d",
 					(g_cluster.localNode == wdNode)? "local":"remote",
 					wdNode->nodeName,
-					wd_failover_lock_name[failoverLock])));
+					wd_failover_lock_name[failoverLock],
+					failoverID)));
 
 	if (get_local_node_state() == WD_COORDINATOR)
 	{
-		/* check if the node requesting to start the command is the lock holder */
+		/* check if the lock holder exists */
 		if (g_cluster.interlockingNode.lockHolderNode)
 		{
 			/* make sure the request is of a valid lock */
@@ -2327,14 +2713,37 @@ node_is_asking_for_failover_lock_status(WatchdogNode* wdNode, WDPacketData* pkt,
 		}
 		else
 		{
+			/* There is one special case, Since only the coordinator/master
+			 * can become a lock holder, and in case when some standby node asks for
+			 * the status of lock before the master node has even started to failover
+			 * The normal flow will return that no lock holder exist. That make the
+			 * standby node to think if master node is already finished with the
+			 * failover.
+			 */
+			if (get_failover_object_by_id(failoverID))
+			{
+				ereport(LOG,
+						(errmsg("[%s] lock status check request from %s pgpool-II node \"%s\" for failover ID %d",
+								wd_failover_lock_name[failoverLock],
+								(g_cluster.localNode == wdNode)? "local":"remote",
+								wdNode->nodeName,
+								failoverID),
+						 errdetail("but failover is not yet started by master node")));
+				res = FAILOVER_RES_NO_LOCKHOLDER_BUT_WAIT;
+				
+			}
+			else
+			{
 			/* no lock holder exists */
-			ereport(LOG,
-					(errmsg("[%s] lock status check request denied to %s pgpool-II node \"%s\"",
-							wd_failover_lock_name[failoverLock],
-							(g_cluster.localNode == wdNode)? "local":"remote",
-							wdNode->nodeName),
-					 errdetail("no lock holder exists")));
-			res = FAILOVER_RES_NO_LOCKHOLDER;
+				ereport(LOG,
+						(errmsg("[%s] lock status check request from %s pgpool-II node \"%s\" for failover ID %d",
+								wd_failover_lock_name[failoverLock],
+								(g_cluster.localNode == wdNode)? "local":"remote",
+								wdNode->nodeName,
+								failoverID),
+						 errdetail("no lock holder exists")));
+				res = FAILOVER_RES_NO_LOCKHOLDER;
+			}
 		}
 	}
 	else
@@ -2582,6 +2991,7 @@ static bool is_socket_connection_connected(SocketConnection* conn)
 	return (conn->sock > 0 && conn->sock_state == WD_SOCK_CONNECTED);
 }
 
+
 static bool is_node_reachable(WatchdogNode* wdNode)
 {
 	if (is_socket_connection_connected(&wdNode->client_socket))
@@ -2591,6 +3001,20 @@ static bool is_node_reachable(WatchdogNode* wdNode)
 	return false;
 }
 
+static bool is_node_active(WatchdogNode* wdNode)
+{
+	if (wdNode->state == WD_DEAD || wdNode->state == WD_LOST || wdNode->state == WD_SHUTDOWN)
+		return false;
+	return true;
+}
+
+static bool is_node_active_and_reachable(WatchdogNode* wdNode)
+{
+	if (is_node_active(wdNode))
+		return is_node_reachable(wdNode);
+	return false;
+}
+
 static int accept_incomming_connections(fd_set* rmask, int pending_fds_count)
 {
 	int processed_fds = 0;
@@ -2721,7 +3145,7 @@ static int update_successful_outgoing_cons(fd_set* wmask, int pending_fds_count)
 	return count;
 }
 
-static bool write_packet_to_socket(int sock, WDPacketData* pkt)
+static bool write_packet_to_socket(int sock, WDPacketData* pkt, bool ipcPacket)
 {
 	int ret = 0;
 	int command_id, len;
@@ -2750,17 +3174,20 @@ static bool write_packet_to_socket(int sock, WDPacketData* pkt)
 				 errdetail("%s",strerror(errno))));
 		return false;
 	}
-	/* COMMAND */
-	command_id = htonl(pkt->command_id);
-	if (write(sock, &command_id, 4) < 4)
+	if (ipcPacket == false)
 	{
-		ereport(LOG,
-			(errmsg("failed to send command id, Socket:%d Type:[%s], Command_ID:%d, data Length:%d",sock,pkt_type?pkt_type->name:"NULL", pkt->command_id,pkt->len),
-				 errdetail("%s",strerror(errno))));
+		/* IPC packets does not have command ID field*/
+		command_id = htonl(pkt->command_id);
+		if (write(sock, &command_id, 4) < 4)
+		{
+			ereport(LOG,
+				(errmsg("failed to send command id, Socket:%d Type:[%s], Command_ID:%d, data Length:%d",sock,pkt_type?pkt_type->name:"NULL", pkt->command_id,pkt->len),
+					 errdetail("%s",strerror(errno))));
 
-		return false;
+			return false;
+		}
 	}
-	/* LENGTH */
+	/* data length */
 	len = htonl(pkt->len);
 	if (write(sock, &len, 4) < 4)
 	{
@@ -2789,6 +3216,14 @@ static bool write_packet_to_socket(int sock, WDPacketData* pkt)
 	return true;
 }
 
+static void wd_packet_shallow_copy(WDPacketData* srcPkt, WDPacketData* dstPkt)
+{
+	dstPkt->command_id = srcPkt->command_id;
+	dstPkt->data = srcPkt->data;
+	dstPkt->len = srcPkt->len;
+	dstPkt->type = srcPkt->type;
+}
+
 static void init_wd_packet(WDPacketData* pkt)
 {
 	pkt->len = 0;
@@ -2957,30 +3392,35 @@ static WDPacketData* get_minimum_message(char type, WDPacketData* replyFor)
 }
 
 
-static WDIPCCommandData* get_wd_IPC_command_from_reply(WDPacketData* pkt)
+static WDCommandData* get_wd_IPC_command_from_reply(WDPacketData* pkt)
 {
 	ListCell *lc;
 	foreach(lc, g_cluster.ipc_commands)
 	{
-		WDIPCCommandData* ipcCommand = lfirst(lc);
+		WDCommandData* ipcCommand = lfirst(lc);
 		if (ipcCommand)
 		{
-			if (ipcCommand->internal_command_id == pkt->command_id)
+			if (ipcCommand->commandSource != COMMAND_SOURCE_IPC)
+				continue;
+			if (ipcCommand->commandPacket.command_id == pkt->command_id)
 				return ipcCommand;
 		}
 	}
 	return NULL;
 }
 
-static WDIPCCommandData* get_wd_IPC_command_from_socket(int sock)
+static WDCommandData* get_wd_IPC_command_from_socket(int sock)
 {
 	ListCell *lc;
 	foreach(lc, g_cluster.ipc_commands)
 	{
-		WDIPCCommandData* ipcCommand = lfirst(lc);
+		WDCommandData* ipcCommand = lfirst(lc);
 		if (ipcCommand)
 		{
-			if (ipcCommand->issueing_sock == sock)
+			if (ipcCommand->commandSource != COMMAND_SOURCE_IPC)
+				continue;
+
+			if (ipcCommand->sourceIPCSocket == sock)
 				return ipcCommand;
 		}
 	}
@@ -2988,17 +3428,18 @@ static WDIPCCommandData* get_wd_IPC_command_from_socket(int sock)
 }
 
 
-static void cleanUpIPCCommand(WDIPCCommandData* ipcCommand)
+static void cleanUpIPCCommand(WDCommandData* ipcCommand)
 {
 	/*
 	 * close the socket associated with ipcCommand
 	 * and remove it from ipcSocket list
 	 */
-	if (ipcCommand->issueing_sock > 0)
+	if (ipcCommand->commandSource == COMMAND_SOURCE_IPC &&
+		ipcCommand->sourceIPCSocket > 0)
 	{
-		close(ipcCommand->issueing_sock);
-		g_cluster.ipc_command_socks = list_delete_int(g_cluster.ipc_command_socks,ipcCommand->issueing_sock);
-		ipcCommand->issueing_sock = -1;
+		close(ipcCommand->sourceIPCSocket);
+		g_cluster.ipc_command_socks = list_delete_int(g_cluster.ipc_command_socks,ipcCommand->sourceIPCSocket);
+		ipcCommand->sourceIPCSocket = -1;
 	}
 	/* Now remove the ipcCommand instance from the command list */
 	g_cluster.ipc_commands = list_delete_ptr(g_cluster.ipc_commands,ipcCommand);
@@ -3184,7 +3625,7 @@ static int standard_packet_processor(WatchdogNode* wdNode, WDPacketData* pkt)
 		case WD_IAM_COORDINATOR_MESSAGE:
 		{
 			/*
-			 * if the message is received from coordinator reply with infor,
+			 * if the message is received from coordinator reply with info,
 			 * otherwise reject
 			 */
 			if (g_cluster.masterNode != NULL && wdNode != g_cluster.masterNode)
@@ -3218,7 +3659,7 @@ static bool send_message_to_connection(SocketConnection* conn, WDPacketData *pkt
 {
 	if (conn->sock > 0 && conn->sock_state == WD_SOCK_CONNECTED)
 	{
-		if (write_packet_to_socket(conn->sock, pkt) == true)
+		if (write_packet_to_socket(conn->sock, pkt, false) == true)
 			return true;
 		ereport(DEBUG1,
 				(errmsg("sending packet failed, closing connection")));
@@ -3230,13 +3671,34 @@ static bool send_message_to_connection(SocketConnection* conn, WDPacketData *pkt
 
 static bool send_message_to_node(WatchdogNode* wdNode, WDPacketData *pkt)
 {
-	if (send_message_to_connection(&wdNode->client_socket,pkt) == true)
-		return true;
-	if (send_message_to_connection(&wdNode->server_socket,pkt) == true)
-		return true;
-	ereport(DEBUG1,
-			(errmsg("sending packet to node \"%s\" failed, closing connection", wdNode->nodeName)));
-	return false;
+	bool ret;
+	
+	ret = send_message_to_connection(&wdNode->client_socket,pkt);
+	
+	if (ret == false)
+	{
+		ret = send_message_to_connection(&wdNode->server_socket,pkt);
+	}
+	if (ret)
+	{
+		/* we only update the last sent time if reply for packet is expected */
+		switch (pkt->type) {
+			case WD_REMOTE_FAILOVER_REQUEST:
+			case WD_FAILOVER_CMD_SYNC_REQUEST:
+			case WD_IPC_FAILOVER_COMMAND:
+				if (wdNode->last_sent_time.tv_sec <= 0)
+					gettimeofday(&wdNode->last_sent_time, NULL);
+				break;
+			default:
+    break;
+		}
+	}
+	else
+	{
+		ereport(DEBUG1,
+				(errmsg("sending packet %c to node \"%s\" failed", pkt->type,wdNode->nodeName)));
+	}
+	return ret;
 }
 
 /*
@@ -3254,16 +3716,177 @@ static int send_message(WatchdogNode* wdNode, WDPacketData *pkt)
 			return 1;
 		return 0;
 	}
-	/* NULL means send to all nodes */
+	/* NULL means send to all reachable nodes */
 	for (i=0; i< g_cluster.remoteNodeCount; i++)
 	{
 		wdNode = &(g_cluster.remoteNodes[i]);
-		if (send_message_to_node(wdNode,pkt))
+		if (is_node_reachable(wdNode) && send_message_to_node(wdNode,pkt))
 			count++;
 	}
 	return count;
 }
 
+static IPC_CMD_PREOCESS_RES wd_command_processor_for_node_lost_event(WDCommandData* ipcCommand, WatchdogNode* wdLostNode)
+{
+	if (ipcCommand->sendToNode)
+	{
+		/* The command was sent to one node only */
+		if (ipcCommand->sendToNode == wdLostNode)
+		{
+			/* Fail this command, Since the only
+			 * node it was sent to is lost
+			 */
+			ipcCommand->commandStatus = COMMAND_FINISHED_SEND_FAILED;
+			wd_command_is_complete(ipcCommand);
+			return IPC_CMD_ERROR;
+		}
+		else
+		{
+			/* Dont worry this command is fine for now */
+			return IPC_CMD_PROCESSING;
+		}
+	}
+	else
+	{
+		/* search the node that is lost */
+		int i;
+		for (i=0; i< g_cluster.remoteNodeCount; i++)
+		{
+			WDCommandNodeResult* nodeResult = &ipcCommand->nodeResults[i];
+			if (nodeResult->wdNode == wdLostNode)
+			{
+				if (nodeResult->cmdState == COMMAND_STATE_SENT)
+				{
+					ereport(LOG,
+							(errmsg("remote node \"%s\" lost while ipc command was in progress ",wdLostNode->nodeName)));
+					/* since the node is lost and will be removed from the cluster
+					 * So remove decrement the sent count of command and see what is
+					 * the situation after that
+					 */
+					nodeResult->cmdState = COMMAND_STATE_DO_NOT_SEND;
+					ipcCommand->commandSendToCount--;
+					if (ipcCommand->commandSendToCount <= ipcCommand->commandReplyFromCount)
+					{
+						/* If we have already received the results from all alive nodes
+						 * finish the command
+						 */
+						ipcCommand->commandStatus = COMMAND_FINISHED_ALL_REPLIED;
+						wd_command_is_complete(ipcCommand);
+						return IPC_CMD_COMPLETE;
+					}
+				}
+				break;
+			}
+		}
+	}
+	return IPC_CMD_PROCESSING;
+}
+
+static void wd_command_is_complete(WDCommandData* ipcCommand)
+{
+	if (ipcCommand->commandCompleteFunc)
+	{
+		ipcCommand->commandCompleteFunc(ipcCommand);
+		return;
+	}
+	/* There is not special function for this command
+	 * use the standard reply
+	 */
+	if (ipcCommand->commandSource == COMMAND_SOURCE_IPC)
+	{
+		char res_type;
+		switch (ipcCommand->commandStatus) {
+			case COMMAND_FINISHED_ALL_REPLIED:
+				res_type = WD_IPC_CMD_RESULT_OK;
+				break;
+			case COMMAND_FINISHED_TIMEOUT:
+				res_type = WD_IPC_CMD_TIMEOUT;
+				break;
+			case COMMAND_FINISHED_NODE_REJECTED:
+			case COMMAND_FINISHED_SEND_FAILED:
+				res_type = WD_IPC_CMD_RESULT_BAD;
+				break;
+			default:
+				res_type= WD_IPC_CMD_RESULT_OK;
+				break;
+		}
+		write_ipc_command_with_result_data(ipcCommand, res_type, NULL, 0);
+	}
+	else if (ipcCommand->commandSource == COMMAND_SOURCE_REMOTE)
+	{
+		char res_type;
+
+		if (ipcCommand->commandStatus == COMMAND_FINISHED_ALL_REPLIED)
+			res_type = WD_ACCEPT_MESSAGE;
+		else
+			res_type = WD_REJECT_MESSAGE;
+
+		reply_with_minimal_message(ipcCommand->sourceWdNode, res_type, &ipcCommand->commandPacket);
+	}
+}
+
+
+static void node_lost_while_ipc_command(WatchdogNode* wdNode)
+{
+	List* ipcCommands_to_del = NIL;
+	ListCell *lc;
+	foreach(lc, g_cluster.ipc_commands)
+	{
+		WDCommandData* ipcCommand = lfirst(lc);
+		IPC_CMD_PREOCESS_RES res = wd_command_processor_for_node_lost_event(ipcCommand, wdNode);
+		if (res != IPC_CMD_PROCESSING)
+		{
+			ipcCommands_to_del = lappend(ipcCommands_to_del,ipcCommand);
+		}
+	}
+	/* delete completed commands */
+	foreach(lc, ipcCommands_to_del)
+	{
+		WDCommandData* ipcCommand = lfirst(lc);
+		cleanUpIPCCommand(ipcCommand);
+	}
+}
+
+
+/*
+ * The function walks through all command and resends
+ * the failed maessage again if it can.
+ */
+static void service_ipc_commands(void)
+{
+	ListCell *lc;
+	foreach(lc, g_cluster.ipc_commands)
+	{
+		WDCommandData* ipcCommand = lfirst(lc);
+
+		if (ipcCommand && ipcCommand->commandSendToErrorCount)
+		{
+			int i;
+			for (i=0; i< g_cluster.remoteNodeCount; i++)
+			{
+				WDCommandNodeResult* nodeResult = &ipcCommand->nodeResults[i];
+				if (nodeResult->cmdState == COMMAND_STATE_SEND_ERROR)
+				{
+					if (is_node_active_and_reachable(nodeResult->wdNode))
+					{
+						ereport(LOG,
+								(errmsg("remote node \"%s\" is reachable again, resending the command packet ",nodeResult->wdNode->nodeName)));
+
+						if (send_message_to_node(nodeResult->wdNode, &ipcCommand->commandPacket) == true)
+						{
+							nodeResult->cmdState = COMMAND_STATE_SENT;
+							ipcCommand->commandSendToErrorCount--;
+							ipcCommand->commandSendToCount++;
+							if (ipcCommand->commandSendToErrorCount == 0)
+								break;
+						}
+					}
+				}
+			}
+		}
+	}
+}
+
 static void service_internal_command(void)
 {
 	int i;
@@ -3275,15 +3898,54 @@ static void service_internal_command(void)
 		WDCommandNodeResult* nodeResult = &g_cluster.currentCommand.nodeResults[i];
 		if (nodeResult->cmdState == COMMAND_STATE_SEND_ERROR)
 		{
-			if (is_node_reachable(nodeResult->wdNode))
+			if (is_node_active_and_reachable(nodeResult->wdNode))
+			{
+				if (send_message_to_node(nodeResult->wdNode, &g_cluster.currentCommand.commandPacket) == true)
+				{
+					nodeResult->cmdState = COMMAND_STATE_SENT;
+					g_cluster.currentCommand.commandSendToCount++;
+				}
+			}
+		}
+	}
+}
+
+/* remove the unreachable nodes from cluster */
+static void service_unreachable_nodes(void)
+{
+	int i;
+	struct timeval currTime;
+	gettimeofday(&currTime,NULL);
+	
+	for (i = 0; i< g_cluster.remoteNodeCount; i++)
+	{
+		WatchdogNode* wdNode = &(g_cluster.remoteNodes[i]);
+
+		if (is_node_active(wdNode) == false)
+			continue;
+
+		if (is_node_reachable(wdNode) || wdNode->client_socket.sock_state == WD_SOCK_WAITING_FOR_CONNECT)
+		{
+			/* check if we are waiting for reply from this node */
+			if (wdNode->last_sent_time.tv_sec > 0)
 			{
-				if (send_message_to_node(nodeResult->wdNode, &g_cluster.currentCommand.packet) == true)
+				if (WD_TIME_DIFF_SEC(currTime,wdNode->last_sent_time) >=  MAX_SECS_WAIT_FOR_REPLY_FROM_NODE)
 				{
-					nodeResult->cmdState = COMMAND_STATE_SENT;
-					g_cluster.currentCommand.commandSendToCount++;
+					ereport(LOG,
+							(errmsg("remote node \"%s\" is not replying..",wdNode->nodeName),
+							 errdetail("marking the node as lost")));
+					/* mark the node as lost */
+					watchdog_state_machine(WD_EVENT_REMOTE_NODE_LOST, wdNode, NULL);
 				}
 			}
 		}
+		else
+		{
+			ereport(LOG,
+					(errmsg("remote node \"%s\" is not reachable",wdNode->nodeName),
+					 errdetail("marking the node as lost")));
+			watchdog_state_machine(WD_EVENT_REMOTE_NODE_LOST, wdNode, NULL);
+		}
 	}
 }
 
@@ -3292,7 +3954,7 @@ static bool watchdog_internal_command_packet_processor(WatchdogNode* wdNode, WDP
 	int i;
 	WDCommandNodeResult* nodeResult = NULL;
 	/* verify the packet is reply for our command */
-	if (pkt->command_id != g_cluster.currentCommand.packet.command_id)
+	if (pkt->command_id != g_cluster.currentCommand.commandPacket.command_id)
 		return false;
 	if (g_cluster.currentCommand.commandStatus != COMMAND_IN_PROGRESS)
 		return false;
@@ -3332,7 +3994,6 @@ static bool watchdog_internal_command_packet_processor(WatchdogNode* wdNode, WDP
 
 	if (g_cluster.currentCommand.commandReplyFromCount >= g_cluster.currentCommand.commandSendToCount)
 	{
-		g_cluster.currentCommand.commandFinished = true;
 		if (pkt->type == WD_REJECT_MESSAGE || pkt->type == WD_ERROR_MESSAGE)
 			g_cluster.currentCommand.commandStatus = COMMAND_FINISHED_NODE_REJECTED;
 		else
@@ -3342,7 +4003,6 @@ static bool watchdog_internal_command_packet_processor(WatchdogNode* wdNode, WDP
 	else if (pkt->type == WD_REJECT_MESSAGE || pkt->type == WD_ERROR_MESSAGE)
 	{
 		/* Error or reject message by any node imidiately finishes the command */
-		g_cluster.currentCommand.commandFinished = true;
 		g_cluster.currentCommand.commandStatus = COMMAND_FINISHED_NODE_REJECTED;
 		watchdog_state_machine(WD_EVENT_COMMAND_FINISHED, wdNode, pkt);
 	}
@@ -3354,68 +4014,27 @@ static bool watchdog_internal_command_packet_processor(WatchdogNode* wdNode, WDP
 static void check_for_current_command_timeout(void)
 {
 	struct timeval currTime;
-	if (g_cluster.currentCommand.commandStatus != COMMAND_IN_PROGRESS ||
-		g_cluster.currentCommand.commandFinished != 0)
+	if (g_cluster.currentCommand.commandStatus != COMMAND_IN_PROGRESS)
 		return;
 
 	gettimeofday(&currTime,NULL);
 	if (WD_TIME_DIFF_SEC(currTime,g_cluster.currentCommand.commandTime) >=  g_cluster.currentCommand.commandTimeoutSecs)
 	{
-		g_cluster.currentCommand.commandFinished = true;
 		g_cluster.currentCommand.commandStatus = COMMAND_FINISHED_TIMEOUT;
 		watchdog_state_machine(WD_EVENT_COMMAND_FINISHED, NULL, NULL);
 	}
 }
 
-static char get_current_command_resultant_message_type(void)
-{
-	char res = WD_ACCEPT_MESSAGE;
-	int i;
-	if (g_cluster.currentCommand.commandFinished == 0)
-		return WD_NO_MESSAGE;
-	if (g_cluster.currentCommand.sendToNode == NULL)
-	{
-		/* The command was for all nodes */
-		for (i = 0; i< g_cluster.remoteNodeCount; i++)
-		{
-			WDCommandNodeResult* nodeRes = &g_cluster.currentCommand.nodeResults[i];
-			if (nodeRes->cmdState != COMMAND_STATE_REPLIED)
-				continue;
-			if (nodeRes->result_type != WD_ACCEPT_MESSAGE && nodeRes->result_type != WD_INFO_MESSAGE)
-			{
-				/* failed */
-				if (res != WD_ERROR_MESSAGE)
-					res = nodeRes->result_type;
-			}
-		}
-	}
-	else
-	{
-		if (g_cluster.currentCommand.commandSendToCount == 0) /* We failed to send to any node */
-			return WD_ERROR_MESSAGE;
-		if (g_cluster.currentCommand.commandReplyFromCount == 0) /* We got no reply */
-			return WD_ERROR_MESSAGE;
-		for (i = 0; i< g_cluster.remoteNodeCount; i++)
-		{
-			WDCommandNodeResult* nodeRes = &g_cluster.currentCommand.nodeResults[i];
-			if (nodeRes->wdNode == g_cluster.currentCommand.sendToNode)
-			{
-				if (nodeRes->cmdState != COMMAND_STATE_REPLIED)
-					return WD_ERROR_MESSAGE;
-				return nodeRes->result_type;
-			}
-		}
-		return WD_ERROR_MESSAGE;
-	}
-	return res;
-}
 
 static void clear_current_command(void)
 {
 	g_cluster.currentCommand.commandStatus = COMMAND_EMPTY;
-	g_cluster.currentCommand.packet.type = WD_NO_MESSAGE;
-	if (g_cluster.currentCommand.packet.data)
-		pfree(g_cluster.currentCommand.packet.data);
+	g_cluster.currentCommand.sourcePacket.type = WD_NO_MESSAGE;
+	g_cluster.currentCommand.commandPacket.type = WD_NO_MESSAGE;
+	if (g_cluster.currentCommand.sourcePacket.data)
+		pfree(g_cluster.currentCommand.sourcePacket.data);
+	g_cluster.currentCommand.sourcePacket.data = NULL;
+	g_cluster.currentCommand.commandPacket.data = NULL;
 }
 
 /*
@@ -3430,10 +4049,10 @@ static int issue_watchdog_internal_command(WatchdogNode* wdNode, WDPacketData *p
 	gettimeofday(&g_cluster.currentCommand.commandTime, NULL);
 
 	g_cluster.currentCommand.commandTimeoutSecs = timeout_sec;
-	g_cluster.currentCommand.packet.type = pkt->type;
-	g_cluster.currentCommand.packet.command_id = pkt->command_id;
-	g_cluster.currentCommand.packet.len = 0;
-	g_cluster.currentCommand.packet.data = NULL;
+	g_cluster.currentCommand.commandPacket.type = pkt->type;
+	g_cluster.currentCommand.commandPacket.command_id = pkt->command_id;
+	g_cluster.currentCommand.commandPacket.len = 0;
+	g_cluster.currentCommand.commandPacket.data = NULL;
 
 	g_cluster.currentCommand.sendToNode = wdNode;
 	g_cluster.currentCommand.commandSendToCount = 0;
@@ -3446,7 +4065,7 @@ static int issue_watchdog_internal_command(WatchdogNode* wdNode, WDPacketData *p
 		{
 			WDCommandNodeResult* nodeResult = &g_cluster.currentCommand.nodeResults[i];
 			clear_command_node_result(nodeResult);
-			if (nodeResult->wdNode->state == WD_DEAD ||  nodeResult->wdNode->state == WD_SHUTDOWN)
+			if (is_node_active(nodeResult->wdNode) == false)
 			{
 				ereport(DEBUG2,
 						(errmsg("not sending watchdog internal command packet to DEAD %s",nodeResult->wdNode->nodeName)));
@@ -3502,11 +4121,10 @@ static int issue_watchdog_internal_command(WatchdogNode* wdNode, WDPacketData *p
 	}
 	if (save_message && pkt->len > 0)
 	{
-		g_cluster.currentCommand.packet.data = MemoryContextAlloc(TopMemoryContext,pkt->len);
-		memcpy(g_cluster.currentCommand.packet.data,pkt->data,pkt->len);
-		g_cluster.currentCommand.packet.len = pkt->len;
+		g_cluster.currentCommand.commandPacket.data = MemoryContextAlloc(TopMemoryContext,pkt->len);
+		memcpy(g_cluster.currentCommand.commandPacket.data,pkt->data,pkt->len);
+		g_cluster.currentCommand.commandPacket.len = pkt->len;
 	}
-	g_cluster.currentCommand.commandFinished = false;
 	return g_cluster.currentCommand.commandSendToCount;
 }
 
@@ -3535,7 +4153,7 @@ static bool service_lost_connections(void)
 	for (i = 0; i< g_cluster.remoteNodeCount; i++)
 	{
 		WatchdogNode* wdNode = &(g_cluster.remoteNodes[i]);
-		if (wdNode->state == WD_SHUTDOWN)
+		if (wdNode->state == WD_SHUTDOWN || wdNode->state == WD_DEAD)
 			continue;
 
 		if (is_socket_connection_connected(&wdNode->client_socket) == false)
@@ -3553,14 +4171,12 @@ static bool service_lost_connections(void)
 					watchdog_state_machine(WD_EVENT_NEW_OUTBOUND_CONNECTION, wdNode, NULL);
 					ret = true;
 				}
-
 			}
 		}
 	}
 	return ret;
 }
 
-
 /*
  * The function only considers the node state.
  * All node states count towards the cluster participating nodes
@@ -3627,6 +4243,14 @@ static int send_cluster_command(WatchdogNode* wdNode, char type, int timeout_sec
 	WDPacketData *pkt = get_message_of_type(type);
 	if (pkt)
 	{
+		/* If the old command is in progress and we are about to issue a new command
+		 * finish the old command with timeout
+		 */
+		if (is_cluster_command_in_progress())
+		{
+			g_cluster.currentCommand.commandStatus = COMMAND_FINISHED_TIMEOUT;
+			watchdog_state_machine(WD_EVENT_COMMAND_FINISHED, NULL, NULL);
+		}
 		ret = issue_watchdog_internal_command(wdNode, pkt, timeout_sec);
 		free_packet(pkt);
 	}
@@ -3663,44 +4287,55 @@ static inline WD_STATES get_local_node_state(void)
 	return g_cluster.localNode->state;
 }
 
-
 /*
  * returns true if no message is swollowed by the
  * processor and no further action is required
  */
 static bool wd_commands_packet_processor(WD_EVENTS event, WatchdogNode* wdNode, WDPacketData* pkt)
 {
-	WDIPCCommandData* ipcCommand;
+	WDCommandData* ipcCommand;
 	
 	if (event != WD_EVENT_PACKET_RCV)
 		return false;
 	if (pkt == NULL)
 		return false;
-	
+
 	if (pkt->type == WD_FAILOVER_CMD_SYNC_REQUEST)
 	{
-		process_failover_command_sync_requests(wdNode, pkt, NULL);
+		process_remote_failover_locking_request(wdNode, pkt);
 		return true;
 	}
-	
-	if (pkt->type == WD_REPLICATE_VARIABLE_REQUEST)
+
+	if (pkt->type == WD_IPC_FAILOVER_COMMAND)
 	{
-		process_pgpool_replicate_command(wdNode, pkt);
+		process_remote_failover_command_on_coordinator(wdNode, pkt);
 		return true;
 	}
-	
+
+	if (pkt->type == WD_REMOTE_FAILOVER_REQUEST)
+	{
+		process_pgpool_remote_failover_command(wdNode, pkt);
+		return true;
+	}
+
+	if (pkt->type == WD_IPC_ONLINE_RECOVERY_COMMAND)
+	{
+		process_remote_online_recovery_command(wdNode, pkt);
+		return true;
+	}
+
 	if (pkt->type == WD_INTERLOCKING_REQUEST)
 	{
 		node_has_requested_for_interlocking(wdNode, pkt);
 		return true;
 	}
-	
+
 	if (pkt->type == WD_INTERUNLOCKING_REQUEST)
 	{
 		node_has_resigned_from_interlocking(wdNode, pkt);
 		return true;
 	}
-	
+
 	if (pkt->type == WD_DATA_MESSAGE)
 	{
 		ipcCommand = get_wd_IPC_command_from_reply(pkt);
@@ -3715,21 +4350,47 @@ static bool wd_commands_packet_processor(WD_EVENTS event, WatchdogNode* wdNode,
 		}
 		return false;
 	}
-	
-	
-	if (pkt->type == WD_ACCEPT_MESSAGE ||
-		pkt->type == WD_REJECT_MESSAGE ||
-		pkt->type == WD_ERROR_MESSAGE)
+
+	if (pkt->type == WD_CMD_REPLY_IN_DATA)
 	{
 		ipcCommand = get_wd_IPC_command_from_reply(pkt);
 		if (ipcCommand == NULL)
 		{
 			return false;
 		}
+		/* Just forward the data to IPC socket and finsh the command */
+		if (write_ipc_command_with_result_data(ipcCommand, WD_IPC_CMD_RESULT_OK, pkt->data, pkt->len) == false)
+			ereport(LOG,
+					(errmsg("failed to forward data message to IPC command socket")));
+		/*
+		 * ok we are done, delete this command
+		 */
+		cleanUpIPCCommand(ipcCommand);
 		
-		if (ipcCommand->type == WD_INTERLOCKING_REQUEST ||
-			ipcCommand->type == WD_INTERUNLOCKING_REQUEST ||
-			ipcCommand->type == WD_FAILOVER_CMD_SYNC_REQUEST)
+		return true; /* do not process this packet further */
+
+	}
+	
+	else if (pkt->type == WD_ACCEPT_MESSAGE ||
+			 pkt->type == WD_REJECT_MESSAGE ||
+			 pkt->type == WD_ERROR_MESSAGE)
+	{
+		ipcCommand = get_wd_IPC_command_from_reply(pkt);
+
+		if (ipcCommand == NULL)
+			return false;
+
+		if (ipcCommand->commandPacket.type == WD_IPC_FAILOVER_COMMAND)
+		{
+			if (pkt->type == WD_ACCEPT_MESSAGE)
+				reply_to_failove_command(ipcCommand, FAILOVER_RES_PROCEED, 0);
+			else
+				reply_to_failove_command(ipcCommand, FAILOVER_RES_MASTER_REJECTED, 0);
+			return true;
+		}
+		else if (ipcCommand->commandPacket.type == WD_INTERLOCKING_REQUEST ||
+			ipcCommand->commandPacket.type == WD_INTERUNLOCKING_REQUEST ||
+			ipcCommand->commandPacket.type == WD_FAILOVER_CMD_SYNC_REQUEST)
 		{
 			/*
 			 * we are expecting only one reply for this
@@ -3754,8 +4415,11 @@ static bool wd_commands_packet_processor(WD_EVENTS event, WatchdogNode* wdNode,
 			return true; /* do not process this packet further */
 		}
 		
-		if (ipcCommand->type == WD_REPLICATE_VARIABLE_REQUEST)
+		else if (ipcCommand->commandPacket.type == WD_REMOTE_FAILOVER_REQUEST ||
+				 ipcCommand->commandPacket.type == WD_IPC_ONLINE_RECOVERY_COMMAND)
+		{
 			return reply_is_received_for_pgpool_replicate_command(wdNode, pkt, ipcCommand);
+		}
 	}
 	
 	return false;
@@ -3830,10 +4494,15 @@ static bool any_interface_available(void)
 static int watchdog_state_machine(WD_EVENTS event, WatchdogNode* wdNode, WDPacketData* pkt)
 {
 	ereport(DEBUG1,
-			(errmsg("STATE MACHINE INVOKED WITH EVENT = %s Current State = %s",wd_event_name[event], wd_state_names[get_local_node_state()])));
+			(errmsg("STATE MACHINE INVOKED WITH EVENT = %s Current State = %s",
+					wd_event_name[event], wd_state_names[get_local_node_state()])));
 	
 	if (event == WD_EVENT_REMOTE_NODE_LOST)
 	{
+		/* close all socket connections to the node */
+		close_socket_connection(&wdNode->client_socket);
+		close_socket_connection(&wdNode->server_socket);
+
 		if (wdNode->state == WD_SHUTDOWN)
 		{
 			ereport(LOG,
@@ -3851,6 +4520,12 @@ static int watchdog_state_machine(WD_EVENTS event, WatchdogNode* wdNode, WDPacke
 					(errmsg("watchdog cluster has lost the coordinator node")));
 			g_cluster.masterNode = NULL;
 		}
+
+		/* clear the wait timer on the node */
+		wdNode->last_sent_time.tv_sec = 0;
+		wdNode->last_sent_time.tv_usec = 0;
+		node_lost_while_ipc_command(wdNode);
+
 	}
 	else if (event == WD_EVENT_PACKET_RCV)
 	{
@@ -3863,11 +4538,12 @@ static int watchdog_state_machine(WD_EVENTS event, WatchdogNode* wdNode, WDPacke
 			standard_packet_processor(wdNode, pkt);
 		}
 
-		if (pkt->type == WD_INFORM_I_AM_GOING_DOWN)		/* TODO do it better way */
+		if (pkt->type == WD_INFORM_I_AM_GOING_DOWN)
 		{
 			wdNode->state = WD_SHUTDOWN;
 			return watchdog_state_machine(WD_EVENT_REMOTE_NODE_LOST, wdNode, NULL);
 		}
+
 		if (watchdog_internal_command_packet_processor(wdNode,pkt) == true)
 		{
 			return 0;
@@ -3999,7 +4675,7 @@ static int watchdog_state_machine_loading(WD_EVENTS event, WatchdogNode* wdNode,
 				}
 			}
 			free_packet(addPkt);
-			set_timeout(4);
+			set_timeout(MAX_SECS_WAIT_FOR_REPLY_FROM_NODE);
 		}
 			break;
 
@@ -4086,8 +4762,9 @@ static int watchdog_state_machine_joining(WD_EVENTS event, WatchdogNode* wdNode,
 	{
 		case WD_EVENT_WD_STATE_CHANGED:
 			g_cluster.masterNode = NULL;
-			send_cluster_command(NULL, WD_REQ_INFO_MESSAGE, 5);
-			set_timeout(5);
+			try_connecting_with_all_unreachable_nodes();
+			send_cluster_command(NULL, WD_REQ_INFO_MESSAGE, 4);
+			set_timeout(MAX_SECS_WAIT_FOR_REPLY_FROM_NODE);
 			break;
 
 		case WD_EVENT_TIMEOUT:
@@ -4096,7 +4773,7 @@ static int watchdog_state_machine_joining(WD_EVENTS event, WatchdogNode* wdNode,
 
 		case WD_EVENT_COMMAND_FINISHED:
 		{
-			if (g_cluster.currentCommand.packet.type == WD_REQ_INFO_MESSAGE)
+			if (g_cluster.currentCommand.commandPacket.type == WD_REQ_INFO_MESSAGE)
 				set_state(WD_INITIALIZING);
 		}
 			break;
@@ -4109,7 +4786,7 @@ static int watchdog_state_machine_joining(WD_EVENTS event, WatchdogNode* wdNode,
 					if (wdNode->state == WD_ADD_MESSAGE_SENT)
 						ereport(FATAL,
 							(return_code(POOL_EXIT_FATAL),
-							 errmsg("Add to watchdog cluster request is rejected by node \"%s:%d\"",wdNode->hostname,wdNode->wd_port),
+							 errmsg("add to watchdog cluster request is rejected by node \"%s:%d\"",wdNode->hostname,wdNode->wd_port),
 								 errhint("check the watchdog configurations.")));
 					break;
 
@@ -4237,14 +4914,14 @@ static int watchdog_state_machine_standForCord(WD_EVENTS event, WatchdogNode* wd
 	switch (event)
 	{
 		case WD_EVENT_WD_STATE_CHANGED:
-			send_cluster_command(NULL, WD_STAND_FOR_COORDINATOR_MESSAGE, 5);
+			send_cluster_command(NULL, WD_STAND_FOR_COORDINATOR_MESSAGE, 4);
 			/* wait for 5 seconds if someone rejects us*/
-			set_timeout(5);
+			set_timeout(MAX_SECS_WAIT_FOR_REPLY_FROM_NODE);
 			break;
 
 		case WD_EVENT_COMMAND_FINISHED:
 		{
-			if (g_cluster.currentCommand.packet.type == WD_STAND_FOR_COORDINATOR_MESSAGE)
+			if (g_cluster.currentCommand.commandPacket.type == WD_STAND_FOR_COORDINATOR_MESSAGE)
 			{
 				if (g_cluster.currentCommand.commandStatus == COMMAND_FINISHED_ALL_REPLIED ||
 					g_cluster.currentCommand.commandStatus == COMMAND_FINISHED_TIMEOUT)
@@ -4358,8 +5035,8 @@ static int watchdog_state_machine_coordinator(WD_EVENTS event, WatchdogNode* wdN
 		case WD_EVENT_WD_STATE_CHANGED:
 		{
 			int i;
-			send_cluster_command(NULL, WD_DECLARE_COORDINATOR_MESSAGE, 5);
-			set_timeout(10);
+			send_cluster_command(NULL, WD_DECLARE_COORDINATOR_MESSAGE, 4);
+			set_timeout(MAX_SECS_WAIT_FOR_REPLY_FROM_NODE);
 			ereport(LOG,
 					(errmsg("I am announcing my self as master/coordinator watchdog node")));
 
@@ -4375,7 +5052,7 @@ static int watchdog_state_machine_coordinator(WD_EVENTS event, WatchdogNode* wdN
 
 		case WD_EVENT_COMMAND_FINISHED:
 		{
-			if (g_cluster.currentCommand.packet.type == WD_DECLARE_COORDINATOR_MESSAGE)
+			if (g_cluster.currentCommand.commandPacket.type == WD_DECLARE_COORDINATOR_MESSAGE)
 			{
 				if (g_cluster.currentCommand.commandStatus == COMMAND_FINISHED_ALL_REPLIED ||
 					g_cluster.currentCommand.commandStatus == COMMAND_FINISHED_TIMEOUT)
@@ -4427,7 +5104,7 @@ static int watchdog_state_machine_coordinator(WD_EVENTS event, WatchdogNode* wdN
 				}
 			}
 			
-			else if (g_cluster.currentCommand.packet.type == WD_IAM_COORDINATOR_MESSAGE)
+			else if (g_cluster.currentCommand.commandPacket.type == WD_IAM_COORDINATOR_MESSAGE)
 			{
 				if (g_cluster.currentCommand.commandStatus == COMMAND_FINISHED_ALL_REPLIED)
 				{
@@ -4510,8 +5187,15 @@ static int watchdog_state_machine_coordinator(WD_EVENTS event, WatchdogNode* wdN
 			break;
 
 		case WD_EVENT_TIMEOUT:
-			send_cluster_command(NULL, WD_IAM_COORDINATOR_MESSAGE, BEACON_MESSAGE_INTERVAL_SECONDS);
-			set_timeout(BEACON_MESSAGE_INTERVAL_SECONDS);
+			if (is_cluster_command_in_progress())
+			{
+				set_timeout(MAX_SECS_WAIT_FOR_REPLY_FROM_NODE);
+			}
+			else
+			{
+				send_cluster_command(NULL, WD_IAM_COORDINATOR_MESSAGE, 5);
+				set_timeout(BEACON_MESSAGE_INTERVAL_SECONDS);
+			}
 			break;
 
 		case WD_EVENT_REMOTE_NODE_LOST:
@@ -4785,7 +5469,7 @@ static int watchdog_state_machine_voting(WD_EVENTS event, WatchdogNode* wdNode,
 	{
 		case WD_EVENT_WD_STATE_CHANGED:
 			clear_current_command();
-			set_timeout(6);
+			set_timeout(MAX_SECS_WAIT_FOR_REPLY_FROM_NODE);
 			break;
 			
 		case WD_EVENT_TIMEOUT:
@@ -4859,7 +5543,7 @@ static int watchdog_state_machine_standby(WD_EVENTS event, WatchdogNode* wdNode,
 
 		case WD_EVENT_COMMAND_FINISHED:
 		{
-			if (g_cluster.currentCommand.packet.type == WD_JOIN_COORDINATOR_MESSAGE)
+			if (g_cluster.currentCommand.commandPacket.type == WD_JOIN_COORDINATOR_MESSAGE)
 			{
 				if (g_cluster.currentCommand.commandStatus == COMMAND_FINISHED_ALL_REPLIED ||
 					g_cluster.currentCommand.commandStatus == COMMAND_FINISHED_TIMEOUT)
@@ -5082,7 +5766,7 @@ static int set_state(WD_STATES newState)
 }
 
 
-static void allocate_resultNodes_in_IPCCommand(WDIPCCommandData* ipcCommand)
+static void allocate_resultNodes_in_IPCCommand(WDCommandData* ipcCommand)
 {
 	MemoryContext oldCxt;
 	int i;
@@ -5099,156 +5783,136 @@ static void allocate_resultNodes_in_IPCCommand(WDIPCCommandData* ipcCommand)
 	MemoryContextSwitchTo(oldCxt);
 }
 
-static IPC_CMD_PREOCESS_RES execute_replicate_command(WDIPCCommandData* ipcCommand)
-{
-	int i;
-	IPC_CMD_PREOCESS_RES res;
-	
-	WDPacketData wdPacket;
-	init_wd_packet(&wdPacket);
-	set_message_type(&wdPacket, WD_REPLICATE_VARIABLE_REQUEST);
-	set_next_commandID_in_message(&wdPacket);
-	set_message_data(&wdPacket,ipcCommand->data_buf, ipcCommand->data_len);
-	
-	allocate_resultNodes_in_IPCCommand(ipcCommand);
-	ipcCommand->sendTo_count = 0;
-	ipcCommand->reply_from_count = 0;
-	ipcCommand->internal_command_id = wdPacket.command_id;
-	ipcCommand->type = wdPacket.type;
-	
-	for (i=0; i< g_cluster.remoteNodeCount; i++)
-	{
-		WDCommandNodeResult* nodeResult = &ipcCommand->nodeResults[i];
-		if (send_message_to_node(nodeResult->wdNode, &wdPacket) == true)
-		{
-			nodeResult->cmdState = COMMAND_STATE_SENT;
-			ipcCommand->sendTo_count++;
-		}
-		else
-			nodeResult->cmdState = COMMAND_STATE_SEND_ERROR;
-	}
-	/*
-	 * The current quorum status of standby node can be out of sync,
-	 * so update it before making any decision on command success
-	 */
-	if (get_local_node_state() == WD_STANDBY)
-		update_quorum_status();
-
-	if (ipcCommand->sendTo_count == 0)
-	{
-		/* We are not able to send the message to any node.
-		 * But this does not straight away means we are failed.
-		 * There are two scenarios.
-		 *
-		 * 1- The current cluster setting requires only single node
-		 * to complete the quorum.
-		 *
-		 * 2- Currrently the cluster does not holds the quorum and
-		 * I am the only node alive
-		 *
-		 * in these both of these above cases the command will be marked as successful
-		 * even if we are not able to send to any node
-		 */
-		if (get_mimimum_nodes_required_for_quorum() == 0)
-			res = IPC_CMD_COMPLETE;
-		/*
-		 * If quorum is not present at the moment, Sending to all connected nodes
-		 * is enough to mark it as success
-		 */
-		else if (g_cluster.quorum_status < 0 && get_cluster_node_count() == 0)
-			res = IPC_CMD_COMPLETE;
-		else
-			res = IPC_CMD_ERROR;
-	}
-	else if (ipcCommand->sendTo_count < get_mimimum_nodes_required_for_quorum() )
-	{
-		if (g_cluster.quorum_status < 0 && get_cluster_node_count() == ipcCommand->sendTo_count)
-			res = IPC_CMD_PROCESSING;
-		else
-			res = IPC_CMD_ERROR;
-	}
-	else
-	{
-		res = IPC_CMD_PROCESSING;
-	}
-	return res;
-}
-
 
-static bool process_pgpool_replicate_command(WatchdogNode* wdNode, WDPacketData* pkt)
+static void process_pgpool_remote_failover_command(WatchdogNode* wdNode, WDPacketData* pkt)
 {
 	char* func_name;
 	int node_count = 0;
 	int *node_id_list = NULL;
-	bool ret = false;
 
 	if (pkt->data == NULL || pkt->len == 0)
 	{
 		ereport(LOG,
-			(errmsg("watchdog is unable to process pgpool replicate command"),
+			(errmsg("watchdog is unable to process pgpool failover command"),
 				 errdetail("command packet contains no data")));
-		return false;
+		reply_with_minimal_message(wdNode, WD_ERROR_MESSAGE, pkt);
+		return;
 	}
 
+	if (wdNode != g_cluster.masterNode)
+	{
+		ereport(LOG,
+			(errmsg("watchdog is unable to process pgpool failover command received from \"%s\"",wdNode->nodeName),
+				 errdetail("only master/coordinator (\"%s\") node can send the replicate commands",g_cluster.masterNode->nodeName)));
+		reply_with_minimal_message(wdNode, WD_ERROR_MESSAGE, pkt);
+		return;
+	}
 	if (parse_wd_node_function_json(pkt->data, pkt->len, &func_name, &node_id_list, &node_count))
-		ret = process_wd_command_function(wdNode, pkt, func_name, node_count, node_id_list);
+	{
+		process_wd_command_function(wdNode, pkt, func_name, node_count, node_id_list, pkt->command_id);
+	}
 	else
+	{
+		ereport(LOG,
+			(errmsg("watchdog is unable to process pgpool failover command"),
+				 errdetail("command packet contains invalid data")));
 		reply_with_minimal_message(wdNode, WD_ERROR_MESSAGE, pkt);
+	}
 
 	if (func_name)
 		pfree(func_name);
 	if (node_id_list)
 		pfree(node_id_list);
-
-	return ret;
 }
 
-static bool process_wd_command_function(WatchdogNode* wdNode, WDPacketData* pkt, char* func_name, int node_count, int* node_id_list)
+static void process_remote_online_recovery_command(WatchdogNode* wdNode, WDPacketData* pkt)
 {
-	if (strcasecmp(WD_FUNCTION_START_RECOVERY, func_name) == 0)
+	char* func_name;
+	int node_count = 0;
+	int *node_id_list = NULL;
+	
+	if (pkt->data == NULL || pkt->len == 0)
 	{
-		if (*InRecovery != RECOVERY_INIT)
-		{
-			reply_with_minimal_message(wdNode, WD_REJECT_MESSAGE, pkt);
-		}
-		else
+		ereport(LOG,
+				(errmsg("watchdog is unable to process pgpool online recovery command"),
+				 errdetail("command packet contains no data")));
+		reply_with_minimal_message(wdNode, WD_ERROR_MESSAGE, pkt);
+		return;
+	}
+
+	ereport(LOG,
+		(errmsg("watchdog received online recovery request from \"%s\"",wdNode->nodeName)));
+
+	if (parse_wd_node_function_json(pkt->data, pkt->len, &func_name, &node_id_list, &node_count))
+	{
+		if (strcasecmp(WD_FUNCTION_START_RECOVERY, func_name) == 0)
 		{
-			*InRecovery = RECOVERY_ONLINE;
-			if (Req_info->conn_counter == 0)
-			{
-				reply_with_minimal_message(wdNode, WD_ACCEPT_MESSAGE, pkt);
-			}
-			else if(pool_config->recovery_timeout <= 0)
+			if (*InRecovery != RECOVERY_INIT)
 			{
 				reply_with_minimal_message(wdNode, WD_REJECT_MESSAGE, pkt);
 			}
 			else
 			{
-				WDFunctionCommandData* wd_func_command;
-				MemoryContext oldCxt = MemoryContextSwitchTo(TopMemoryContext);
-				
-				wd_func_command = palloc(sizeof(WDFunctionCommandData));
-				wd_func_command->commandType = pkt->type;
-				wd_func_command->commandID = pkt->command_id;
-				wd_func_command->funcName = MemoryContextStrdup(TopMemoryContext,func_name);
-				wd_func_command->wdNode = wdNode;
-
-				/* Add this command for timer tick */
-				add_wd_command_for_timer_events(pool_config->recovery_timeout, true, wd_func_command);
-
-				MemoryContextSwitchTo(oldCxt);
-				
+				*InRecovery = RECOVERY_ONLINE;
+				if (Req_info->conn_counter == 0)
+				{
+					reply_with_minimal_message(wdNode, WD_ACCEPT_MESSAGE, pkt);
+				}
+				else if(pool_config->recovery_timeout <= 0)
+				{
+					reply_with_minimal_message(wdNode, WD_REJECT_MESSAGE, pkt);
+				}
+				else
+				{
+					WDFunctionCommandData* wd_func_command;
+					MemoryContext oldCxt = MemoryContextSwitchTo(TopMemoryContext);
+					
+					wd_func_command = palloc(sizeof(WDFunctionCommandData));
+					wd_func_command->commandType = pkt->type;
+					wd_func_command->commandID = pkt->command_id;
+					wd_func_command->funcName = MemoryContextStrdup(TopMemoryContext,func_name);
+					wd_func_command->wdNode = wdNode;
+					
+					/* Add this command for timer tick */
+					add_wd_command_for_timer_events(pool_config->recovery_timeout, true, wd_func_command);
+					
+					MemoryContextSwitchTo(oldCxt);
+					
+				}
 			}
 		}
+		else if (strcasecmp(WD_FUNCTION_END_RECOVERY, func_name) == 0)
+		{
+			*InRecovery = RECOVERY_INIT;
+			reply_with_minimal_message(wdNode, WD_ACCEPT_MESSAGE, pkt);
+			kill(getppid(), SIGUSR2);
+		}
+		else
+		{
+			ereport(LOG,
+				(errmsg("watchdog failed to process online recovery request"),
+					 errdetail("invalid command [%s] in online recovery request from \"%s\"",func_name,wdNode->nodeName)));
+			reply_with_minimal_message(wdNode, WD_ERROR_MESSAGE, pkt);
+		}
 	}
-	else if (strcasecmp(WD_FUNCTION_END_RECOVERY, func_name) == 0)
+	else
 	{
-		*InRecovery = RECOVERY_INIT;
-		reply_with_minimal_message(wdNode, WD_ACCEPT_MESSAGE, pkt);
-		kill(getppid(), SIGUSR2);
+		ereport(LOG,
+			(errmsg("watchdog failed to process online recovery request"),
+				 errdetail("invalid data in online recovery request from \"%s\"",wdNode->nodeName)));
+		reply_with_minimal_message(wdNode, WD_ERROR_MESSAGE, pkt);
 	}
 	
-	else if (strcasecmp(WD_FUNCTION_FAILBACK_REQUEST, func_name) == 0)
+	if (func_name)
+		pfree(func_name);
+	if (node_id_list)
+		pfree(node_id_list);
+}
+
+static void process_wd_command_function(WatchdogNode* wdNode, WDPacketData* pkt, char* func_name,
+										int node_count, int* node_id_list, unsigned int failover_id)
+{
+	if (strcasecmp(WD_FUNCTION_FAILBACK_REQUEST, func_name) == 0)
 	{
 		if (Req_info->switching)
 		{
@@ -5260,8 +5924,7 @@ static bool process_wd_command_function(WatchdogNode* wdNode, WDPacketData* pkt,
 		else
 		{
 			reply_with_minimal_message(wdNode, WD_ACCEPT_MESSAGE, pkt);
-			wd_set_node_mask_for_failback_req(node_id_list, node_count);
-			send_failback_request(node_id_list[0],false);
+			send_failback_request(node_id_list[0],false, failover_id);
 		}
 	}
 	
@@ -5277,11 +5940,10 @@ static bool process_wd_command_function(WatchdogNode* wdNode, WDPacketData* pkt,
 		else
 		{
 			reply_with_minimal_message(wdNode, WD_ACCEPT_MESSAGE, pkt);
-			wd_set_node_mask_for_degenerate_req(node_id_list, node_count);
-			degenerate_backend_set(node_id_list, node_count, false);
+			degenerate_backend_set(node_id_list, node_count, false, failover_id);
 		}
 	}
-	
+
 	else if (strcasecmp(WD_FUNCTION_PROMOTE_REQUEST, func_name) == 0)
 	{
 		if (Req_info->switching)
@@ -5294,8 +5956,7 @@ static bool process_wd_command_function(WatchdogNode* wdNode, WDPacketData* pkt,
 		else
 		{
 			reply_with_minimal_message(wdNode, WD_ACCEPT_MESSAGE, pkt);
-			wd_set_node_mask_for_promote_req(node_id_list, node_count);
-			promote_backend(node_id_list[0]);
+			promote_backend(node_id_list[0], failover_id);
 		}
 	}
 	else
@@ -5303,11 +5964,10 @@ static bool process_wd_command_function(WatchdogNode* wdNode, WDPacketData* pkt,
 		/* This is not supported function */
 		reply_with_minimal_message(wdNode, WD_ERROR_MESSAGE, pkt);
 	}
-	return true;
 }
 
 
-static bool reply_is_received_for_pgpool_replicate_command(WatchdogNode* wdNode, WDPacketData* pkt, WDIPCCommandData* ipcCommand)
+static bool reply_is_received_for_pgpool_replicate_command(WatchdogNode* wdNode, WDPacketData* pkt, WDCommandData* ipcCommand)
 {
 	int i;
 	WDCommandNodeResult* nodeResult = NULL;
@@ -5328,36 +5988,29 @@ static bool reply_is_received_for_pgpool_replicate_command(WatchdogNode* wdNode,
 				(errmsg("unable to find result node for pgpool-II replicate command packet received from watchdog node \"%s\"",wdNode->nodeName)));
 		return true;
 	}
+
 	nodeResult->result_type = pkt->type;
 	nodeResult->cmdState = COMMAND_STATE_REPLIED;
-	ipcCommand->reply_from_count++;
+	ipcCommand->commandReplyFromCount++;
 	ereport(DEBUG2,
 		(errmsg("watchdog node \"%s\" has replied for pgpool-II replicate command packet",wdNode->nodeName),
-			 errdetail("command was sent to %d nodes and %d nodes have replied to it",ipcCommand->sendTo_count,ipcCommand->reply_from_count)));
+			 errdetail("command was sent to %d nodes and %d nodes have replied to it",ipcCommand->commandSendToCount,ipcCommand->commandReplyFromCount)));
 
-	if (ipcCommand->reply_from_count >= ipcCommand->sendTo_count)
+	if (pkt->type != WD_ACCEPT_MESSAGE)
+	{
+		/* reject message from any node finishes the command */
+		ipcCommand->commandStatus = COMMAND_FINISHED_NODE_REJECTED;
+		wd_command_is_complete(ipcCommand);
+		cleanUpIPCCommand(ipcCommand);
+	}
+	else if (ipcCommand->commandReplyFromCount >= ipcCommand->commandSendToCount)
 	{
 		/*
 		 * we have received results from all nodes
 		 * analyze the result
 		 */
-
-		char res_type = WD_IPC_CMD_RESULT_OK;
-		
-		for (i=0; i< g_cluster.remoteNodeCount; i++)
-		{
-			nodeResult = &ipcCommand->nodeResults[i];
-			if (nodeResult->cmdState == COMMAND_STATE_REPLIED &&
-				nodeResult->result_type != WD_ACCEPT_MESSAGE)
-			{
-				res_type = WD_IPC_CMD_RESULT_BAD;
-				break;
-			}
-		}
-		if (write_ipc_command_with_result_data(ipcCommand, res_type, NULL, 0) == false)
-			ereport(LOG,
-				(errmsg("failed to forward message to IPC command socket")));
-
+		ipcCommand->commandStatus = COMMAND_FINISHED_ALL_REPLIED;
+		wd_command_is_complete(ipcCommand);
 		cleanUpIPCCommand(ipcCommand);
 	}
 
@@ -5370,7 +6023,7 @@ static bool reply_is_received_for_pgpool_replicate_command(WatchdogNode* wdNode,
  */
 static bool process_wd_command_timer_event(bool timer_expired, WDFunctionCommandData* wd_func_command)
 {
-	if (wd_func_command->commandType == WD_REPLICATE_VARIABLE_REQUEST)
+	if (wd_func_command->commandType == WD_IPC_ONLINE_RECOVERY_COMMAND)
 	{
 		if (wd_func_command->funcName && strcasecmp("START_RECOVERY", wd_func_command->funcName) == 0)
 		{
@@ -5379,7 +6032,6 @@ static bool process_wd_command_timer_event(bool timer_expired, WDFunctionCommand
 				WDPacketData emptyPkt;
 				emptyPkt.command_id = wd_func_command->commandID;
 				reply_with_minimal_message(wd_func_command->wdNode, WD_ACCEPT_MESSAGE, &emptyPkt);
-				/* TODO delete command object */
 				return true;
 			}
 			else if (timer_expired)
@@ -5673,7 +6325,7 @@ static bool check_IPC_client_authentication(json_value *rootObj, bool internal_c
  * this one also informs the calling client about the failure
  */
 
-static bool check_and_report_IPC_authentication(WDIPCCommandData* ipcCommand)
+static bool check_and_report_IPC_authentication(WDCommandData* ipcCommand)
 {
 	json_value *root = NULL;
 	bool internal_client_only = false;
@@ -5683,7 +6335,7 @@ static bool check_and_report_IPC_authentication(WDIPCCommandData* ipcCommand)
 		return false;	/* should never happen*/
 
 	/* first identify the command type */
-	switch(ipcCommand->type)
+	switch(ipcCommand->sourcePacket.type)
 	{
 		case WD_NODE_STATUS_CHANGE_COMMAND:
 		case WD_REGISTER_FOR_NOTIFICATION:
@@ -5691,7 +6343,8 @@ static bool check_and_report_IPC_authentication(WDIPCCommandData* ipcCommand)
 			internal_client_only = false;
 			break;
 
-		case WD_FUNCTION_COMMAND:
+		case WD_IPC_FAILOVER_COMMAND:
+		case WD_IPC_ONLINE_RECOVERY_COMMAND:
 		case WD_FAILOVER_CMD_SYNC_REQUEST:
 		case WD_GET_MASTER_DATA_REQUEST:
 			/* only allowed internaly.*/
@@ -5710,15 +6363,18 @@ static bool check_and_report_IPC_authentication(WDIPCCommandData* ipcCommand)
 		return true;
 	}
 
-	if (ipcCommand->data_len <= 0 || ipcCommand->data_buf == NULL)
+	if (ipcCommand->sourcePacket.len <= 0 || ipcCommand->sourcePacket.data == NULL)
 	{
 		ereport(LOG,
 			(errmsg("authentication failed"),
 				 errdetail("IPC command contains no data")));
+		ipcCommand->errorMessage = MemoryContextStrdup(ipcCommand->memoryContext,
+													   "authentication failed: invalid data");
+
 		return false;
 	}
 
-	root = json_parse(ipcCommand->data_buf,ipcCommand->data_len);
+	root = json_parse(ipcCommand->sourcePacket.data,ipcCommand->sourcePacket.len);
 	/* The root node must be object */
 	if (root == NULL || root->type != json_object)
 	{
@@ -5726,6 +6382,10 @@ static bool check_and_report_IPC_authentication(WDIPCCommandData* ipcCommand)
 		ereport(LOG,
 			(errmsg("authentication failed"),
 				 errdetail("IPC command contains an invalid data")));
+
+		ipcCommand->errorMessage = MemoryContextStrdup(ipcCommand->memoryContext,
+													   "authentication failed: invalid data");
+
 		return false;
 	}
 
@@ -5734,19 +6394,11 @@ static bool check_and_report_IPC_authentication(WDIPCCommandData* ipcCommand)
 
 	if (ret == false)
 	{
-		char* error_json;
 		ereport(WARNING,
 			(errmsg("authentication failed"),
 				 errdetail("invalid IPC key")));
-
-		error_json = get_wd_simple_error_message_json("IPC client authentication failed");
-		if (write_ipc_command_with_result_data(ipcCommand, WD_IPC_CMD_RESULT_BAD,
-											   error_json, strlen(error_json) +1))
-		{
-			ereport(LOG,
-					(errmsg("failed to forward error message to IPC socket")));
-		}
-		pfree(error_json);
+		ipcCommand->errorMessage = MemoryContextStrdup(ipcCommand->memoryContext,
+													   "authentication failed: invalid KEY");
 	}
 	return ret;
 }
@@ -5785,3 +6437,67 @@ static void print_received_packet_info(WDPacketData* pkt,WatchdogNode* wdNode)
 					   pkt_type?pkt_type->name:"UNKNOWN",
 					   wd_state_names[get_local_node_state()])));
 }
+
+static int send_command_packet_to_remote_nodes(WDCommandData* ipcCommand, bool source_included)
+{
+	int i;
+	ipcCommand->commandSendToCount = 0;
+	ipcCommand->commandReplyFromCount = 0;
+	ipcCommand->commandSendToErrorCount = 0;
+	allocate_resultNodes_in_IPCCommand(ipcCommand);
+	ereport(DEBUG2,
+			(errmsg("sending the %c type message to \"%s\"",
+					ipcCommand->commandPacket.type,
+					ipcCommand->sendToNode?ipcCommand->sendToNode->nodeName:"ALL NODES")));
+	for (i=0; i< g_cluster.remoteNodeCount; i++)
+	{
+		WDCommandNodeResult* nodeResult = &ipcCommand->nodeResults[i];
+		if (ipcCommand->sendToNode != NULL && ipcCommand->sendToNode != nodeResult->wdNode)
+		{
+			/* The command is intended for specific node and this is not the one */
+			nodeResult->cmdState = COMMAND_STATE_DO_NOT_SEND;
+		}
+		else if (source_included == false && ipcCommand->sourceWdNode == nodeResult->wdNode &&
+				 ipcCommand->commandSource == COMMAND_SOURCE_REMOTE)
+		{
+			ereport(DEBUG1,
+					(errmsg("not sending the %c type message to command originator node \"%s\"",
+							ipcCommand->commandPacket.type,nodeResult->wdNode->nodeName)));
+			/*
+			 * The message is not supposed to be sent to the watchdog
+			 * node that started this command
+			 */
+			nodeResult->cmdState = COMMAND_STATE_DO_NOT_SEND;
+		}
+		else if (is_node_active(nodeResult->wdNode) == false)
+		{
+			nodeResult->cmdState = COMMAND_STATE_DO_NOT_SEND;
+		}
+		else if (is_node_reachable(nodeResult->wdNode) == false)
+		{
+			nodeResult->cmdState = COMMAND_STATE_SEND_ERROR;
+			ipcCommand->commandSendToErrorCount++;
+		}
+		else if (send_message_to_node(nodeResult->wdNode, &ipcCommand->commandPacket) == true)
+		{
+			ereport(DEBUG2,
+					(errmsg("%c type message written to socket for node \"%s\"",
+							ipcCommand->commandPacket.type,nodeResult->wdNode->nodeName)));
+
+			nodeResult->cmdState = COMMAND_STATE_SENT;
+			ipcCommand->commandSendToCount++;
+		}
+		else
+		{
+			nodeResult->cmdState = COMMAND_STATE_SEND_ERROR;
+			ipcCommand->commandSendToErrorCount++;
+		}
+	}
+	return ipcCommand->commandSendToCount;
+}
+
+static bool is_cluster_command_in_progress(void)
+{
+	return g_cluster.currentCommand.commandStatus == COMMAND_IN_PROGRESS;
+}
+
diff --git a/src/watchdog/wd_commands.c b/src/watchdog/wd_commands.c
index 0133267..9eea88b 100644
--- a/src/watchdog/wd_commands.c
+++ b/src/watchdog/wd_commands.c
@@ -49,29 +49,22 @@
 #include "watchdog/wd_ipc_commands.h"
 #include "watchdog/wd_ipc_defines.h"
 
-#define WD_DEFAULT_IPC_COMMAND_TIMEOUT	5 /* default number of seconds to wait for IPC command results*/
+#define WD_DEFAULT_IPC_COMMAND_TIMEOUT	8 /* default number of seconds to wait for IPC command results*/
 #define WD_INTERLOCK_WAIT_MSEC		500
 #define WD_INTERLOCK_TIMEOUT_SEC	10
 #define WD_INTERLOCK_WAIT_COUNT ((int) ((WD_INTERLOCK_TIMEOUT_SEC * 1000)/WD_INTERLOCK_WAIT_MSEC))
 
-#define FAILBACK_REQUEST_NODE_MASK		0x01
-#define DEGENERATE_REQUEST_NODE_MASK	0x02
-#define PROMOTE_REQUEST_NODE_MASK		0x04
-
 static void sleep_in_waiting(void);
 static void FreeCmdResult(WDIPCCmdResult* res);
 
-static WDFailoverCMDResults wd_issue_failover_lock_command(char* syncReqType, enum WDFailoverLocks lockID);
-static char* get_wd_failover_cmd_type_json(char* reqType, enum WDFailoverLocks lockID);
-static WDFailoverCMDResults wd_send_failover_sync_command(char* syncReqType, enum WDFailoverLocks lockID);
-
-static int wd_set_node_mask (unsigned char req_mask, int *node_id_set, int count);
-static int wd_chk_node_mask (unsigned char req_mask, int *node_id_set, int count);
+static WDFailoverCMDResults wd_issue_failover_lock_command(char* syncReqType, enum WDFailoverLocks lockID, unsigned int wd_failover_id);
+static char* get_wd_failover_cmd_type_json(char* reqType, enum WDFailoverLocks lockID, unsigned int wd_failover_id);
+static WDFailoverCMDResults wd_send_failover_sync_command(char* syncReqType, enum WDFailoverLocks lockID, unsigned int wd_failover_id);
 
 static int open_wd_command_sock(bool throw_error);
+static WDFailoverCMDResults wd_get_failover_result_from_data(WDIPCCmdResult *result, unsigned int *wd_failover_id);
 
 /* shared memory variables */
-unsigned char *WD_Node_List = NULL;
 char *watchdog_ipc_address = NULL;
 unsigned int *ipc_shared_key = NULL;   /* key lives in shared memory
 										* used to identify the ipc internal
@@ -80,18 +73,6 @@ unsigned int *ipc_shared_key = NULL;   /* key lives in shared memory
 
 void wd_ipc_initialize_data(void)
 {
-	/* allocate node list */
-	if (WD_Node_List == NULL)
-	{
-		WD_Node_List = pool_shared_memory_create(sizeof(unsigned char) * MAX_NUM_BACKENDS);
-		memset(WD_Node_List, 0, sizeof(unsigned char) * MAX_NUM_BACKENDS);
-		ereport(DEBUG1,
-				(errmsg("WD_Node_List: sizeof(unsigned char) (%zu) * MAX_NUM_BACKENDS (%d) = %zu bytes requested for shared memory",
-						sizeof(unsigned char),
-						MAX_WATCHDOG_NUM,
-						sizeof(unsigned char) * MAX_NUM_BACKENDS)));
-	}
-
 	if (watchdog_ipc_address == NULL)
 	{
 		char wd_ipc_sock_addr[255];
@@ -203,7 +184,7 @@ issue_command_to_watchdog(char type, int timeout_sec, char* data, int data_len,
 				if (errno == EAGAIN || errno == EINTR)
 					continue;
 				ereport(WARNING,
-					(errmsg("error reading from IPC command socket"),
+					(errmsg("error reading from IPC command socket for ipc command %c",type),
 						 errdetail("select system call failed with error \"%s\"",strerror(errno))));
 				close(sock);
 				return NULL;
@@ -214,7 +195,7 @@ issue_command_to_watchdog(char type, int timeout_sec, char* data, int data_len,
 				if (socket_read(sock, &res_type, 1 ,0) <=0)
 				{
 					ereport(WARNING,
-						(errmsg("error reading from IPC command socket"),
+						(errmsg("error reading from IPC command socket for ipc command %c",type),
 							 errdetail("read from socket failed with error \"%s\"",strerror(errno))));
 					close(sock);
 					return result;
@@ -223,7 +204,7 @@ issue_command_to_watchdog(char type, int timeout_sec, char* data, int data_len,
 				if (socket_read(sock, &res_length, sizeof(int), 0) <= 0)
 				{
 					ereport(WARNING,
-						(errmsg("error reading from IPC command socket"),
+						(errmsg("error reading from IPC command socket for ipc command %c",type),
 							 errdetail("read from socket failed with error \"%s\"",strerror(errno))));
 					close(sock);
 					return result;
@@ -242,7 +223,7 @@ issue_command_to_watchdog(char type, int timeout_sec, char* data, int data_len,
 						pfree(result->data);
 						pfree(result);
 						ereport(DEBUG1,
-							(errmsg("error reading from IPC command socket"),
+							(errmsg("error reading from IPC command socket for ipc command %c",type),
 								 errdetail("read from socket failed with error \"%s\"",strerror(errno))));
 						close(sock);
 						return NULL;
@@ -338,7 +319,7 @@ wd_start_recovery(void)
 	char* func = get_wd_node_function_json(WD_FUNCTION_START_RECOVERY, NULL,0,
 										   shared_key?*shared_key:0,pool_config->wd_authkey);
 
-	WDIPCCmdResult *result = issue_command_to_watchdog(WD_FUNCTION_COMMAND,
+	WDIPCCmdResult *result = issue_command_to_watchdog(WD_IPC_ONLINE_RECOVERY_COMMAND,
 													   pool_config->recovery_timeout,
 													   func, strlen(func), true);
 	pfree(func);
@@ -385,7 +366,7 @@ wd_end_recovery(void)
 										   shared_key?*shared_key:0,pool_config->wd_authkey);
 
 	
-	WDIPCCmdResult *result = issue_command_to_watchdog(WD_FUNCTION_COMMAND,
+	WDIPCCmdResult *result = issue_command_to_watchdog(WD_IPC_ONLINE_RECOVERY_COMMAND,
 													   WD_DEFAULT_IPC_COMMAND_TIMEOUT,
 													   func, strlen(func), true);
 	pfree(func);
@@ -424,60 +405,28 @@ wd_end_recovery(void)
 }
 
 
-WdCommandResult
-wd_send_failback_request(int node_id)
+WDFailoverCMDResults
+wd_send_failback_request(int node_id, unsigned int *wd_failover_id)
 {
 	int n = node_id;
-	char type;
 	char* func;
 	unsigned int *shared_key = get_ipc_shared_key();
+	WDFailoverCMDResults res;
 
-	/* if failback packet is received already, do nothing */
-	if (wd_chk_node_mask_for_failback_req(&n,1))
-		return COMMAND_OK;
-	
 	func = get_wd_node_function_json(WD_FUNCTION_FAILBACK_REQUEST,&n, 1,
 									 shared_key?*shared_key:0,pool_config->wd_authkey);
 
-	WDIPCCmdResult *result = issue_command_to_watchdog(WD_FUNCTION_COMMAND,
+	WDIPCCmdResult *result = issue_command_to_watchdog(WD_IPC_FAILOVER_COMMAND,
 													   WD_DEFAULT_IPC_COMMAND_TIMEOUT,
 													   func, strlen(func), true);
 	pfree(func);
-	
-	if (result == NULL)
-	{
-		ereport(WARNING,
-			(errmsg("watchdog failed to send failback command"),
-				 errdetail("issue command to watchdog returned NULL")));
-		return COMMAND_FAILED;
-	}
-	
-	type = result->type;
-	FreeCmdResult(result);
-	if (type == WD_IPC_CMD_CLUSTER_IN_TRAN)
-	{
-		ereport(WARNING,
-			(errmsg("watchdog failed to send failback command"),
-				 errdetail("watchdog cluster is not in stable state"),
-					errhint("try again when the cluster is fully initialized")));
-		return CLUSTER_IN_TRANSATIONING;
-	}
-	else if (type == WD_IPC_CMD_TIMEOUT)
-	{
-		ereport(WARNING,
-			(errmsg("watchdog failed to send failback command"),
-				 errdetail("ipc command timeout")));
-		return COMMAND_TIMEOUT;
-	}
-	else if (type == WD_IPC_CMD_RESULT_OK)
-	{
-		return COMMAND_OK;
-	}
 
-	return COMMAND_FAILED;
+	res = wd_get_failover_result_from_data(result, wd_failover_id);
+	FreeCmdResult(result);
+	return res;
 }
 
-static char* get_wd_failover_cmd_type_json(char* reqType, enum WDFailoverLocks lockID)
+static char* get_wd_failover_cmd_type_json(char* reqType, enum WDFailoverLocks lockID, unsigned int wd_failover_id)
 {
 	char* json_str;
 	JsonNode* jNode = jw_create_with_object(true);
@@ -489,6 +438,7 @@ static char* get_wd_failover_cmd_type_json(char* reqType, enum WDFailoverLocks l
 
 	jw_put_string(jNode, "SyncRequestType", reqType);
 	jw_put_int(jNode, "FailoverLockID", lockID);
+	jw_put_int(jNode, "WDFailoverID", wd_failover_id);
 	jw_finish_document(jNode);
 	json_str = pstrdup(jw_get_json_string(jNode));
 	jw_destroy(jNode);
@@ -496,12 +446,12 @@ static char* get_wd_failover_cmd_type_json(char* reqType, enum WDFailoverLocks l
 }
 
 static WDFailoverCMDResults
-wd_send_failover_sync_command(char* syncReqType, enum WDFailoverLocks lockID)
+wd_send_failover_sync_command(char* syncReqType, enum WDFailoverLocks lockID, unsigned int wd_failover_id)
 {
-	int interlockingResult = FAILOVER_RES_ERROR;
-	json_value *root;
-	
-	char* json_data = get_wd_failover_cmd_type_json(syncReqType, lockID);
+	WDFailoverCMDResults res;
+	unsigned int failover_id;
+
+	char* json_data = get_wd_failover_cmd_type_json(syncReqType, lockID, wd_failover_id);
 
 	WDIPCCmdResult *result = issue_command_to_watchdog(WD_FAILOVER_CMD_SYNC_REQUEST
 													   ,pool_config->recovery_timeout,
@@ -509,159 +459,105 @@ wd_send_failover_sync_command(char* syncReqType, enum WDFailoverLocks lockID)
 
 	pfree(json_data);
 
-	if (result == NULL || result->length <= 0)
+	res = wd_get_failover_result_from_data(result, &failover_id);
+
+	FreeCmdResult(result);
+	return res;
+}
+
+static WDFailoverCMDResults wd_get_failover_result_from_data(WDIPCCmdResult *result, unsigned int *wd_failover_id)
+{
+	if (result == NULL)
+		return FAILOVER_RES_ERROR;
+
+	if (result == NULL)
 	{
 		ereport(WARNING,
-			(errmsg("watchdog failed to send failover command"),
+			(errmsg("failover command on watchdog failed"),
 				 errdetail("issue command to watchdog returned NULL")));
 		return FAILOVER_RES_ERROR;
 	}
-	else if (result->type == WD_IPC_CMD_TIMEOUT)
+	
+	if (result->type == WD_IPC_CMD_CLUSTER_IN_TRAN)
 	{
 		ereport(WARNING,
-			(errmsg("watchdog failed to send failover command"),
-				 errdetail("ipc command timeout")));
-		FreeCmdResult(result);
-		return FAILOVER_RES_ERROR;
+				(errmsg("failover command on watchdog failed"),
+				 errdetail("watchdog cluster is not in stable state"),
+					errhint("try again when the cluster is fully initialized")));
+		return FAILOVER_RES_TRANSITION;
 	}
-	if (result->length <= 0)
+	else if (result->type == WD_IPC_CMD_TIMEOUT)
 	{
 		ereport(WARNING,
-			(errmsg("watchdog failed to send failover command"),
-				 errdetail("issue command to watchdog returned no data")));
-		FreeCmdResult(result);
-		return FAILOVER_RES_ERROR;
+				(errmsg("failover command on watchdog failed"),
+				 errdetail("ipc command timeout")));
+		return FAILOVER_RES_TIMEOUT;
 	}
-
-	root = json_parse(result->data,result->length);
-	/* The root node must be object */
-	if (root == NULL || root->type != json_object)
+	else if (result->type == WD_IPC_CMD_RESULT_OK)
 	{
-		ereport(NOTICE,
-				(errmsg("unable to parse json data from replicate command")));
-		FreeCmdResult(result);
-		return FAILOVER_RES_ERROR;
-	}
+		WDFailoverCMDResults res = FAILOVER_RES_ERROR;
+		json_value *root;
 
-	if (root && json_get_int_value_for_key(root, "InterlockingResult", &interlockingResult))
-	{
-		json_value_free(root);
-		FreeCmdResult(result);
-		return FAILOVER_RES_ERROR;
+		root = json_parse(result->data,result->length);
+		/* The root node must be object */
+		if (root == NULL || root->type != json_object)
+		{
+			ereport(NOTICE,
+					(errmsg("unable to parse json data from failover command result")));
+			return res;
+		}
+		if (root && json_get_int_value_for_key(root, WD_FAILOVER_RESULT_KEY, (int*)&res))
+		{
+			json_value_free(root);
+			return FAILOVER_RES_ERROR;
+		}
+		if (root && json_get_int_value_for_key(root, WD_FAILOVER_ID_KEY, (int*)wd_failover_id))
+		{
+			json_value_free(root);
+			return FAILOVER_RES_ERROR;
+		}
+		return res;
 	}
-
-	json_value_free(root);
-	FreeCmdResult(result);
-	
-	if (interlockingResult < 0 || interlockingResult > FAILOVER_RES_NO_LOCKHOLDER)
-		return FAILOVER_RES_ERROR;
-
-	return interlockingResult;
+	return FAILOVER_RES_ERROR;
 }
 
-WdCommandResult
-wd_degenerate_backend_set(int *node_id_set, int count)
+WDFailoverCMDResults
+wd_degenerate_backend_set(int *node_id_set, int count, unsigned int *wd_failover_id)
 {
-	char type;
+	WDFailoverCMDResults res;
 	char* func;
 	unsigned int *shared_key = get_ipc_shared_key();
-
-	/* if failback packet is received already, do nothing */
-	if (wd_chk_node_mask_for_degenerate_req(node_id_set,count))
-		return COMMAND_OK;
 	
 	func = get_wd_node_function_json(WD_FUNCTION_DEGENERATE_REQUEST,node_id_set, count,
 									 shared_key?*shared_key:0,pool_config->wd_authkey);
 
-	WDIPCCmdResult *result = issue_command_to_watchdog(WD_FUNCTION_COMMAND ,
+	WDIPCCmdResult *result = issue_command_to_watchdog(WD_IPC_FAILOVER_COMMAND ,
 													   WD_DEFAULT_IPC_COMMAND_TIMEOUT,
 													   func, strlen(func), true);
 	pfree(func);
-	
-	if (result == NULL)
-	{
-		ereport(WARNING,
-			(errmsg("degenerate backend set command failed"),
-				 errdetail("issue command to watchdog returned NULL")));
-		return COMMAND_FAILED;
-	}
-	
-	type = result->type;
+	res = wd_get_failover_result_from_data(result, wd_failover_id);
 	FreeCmdResult(result);
-	if (type == WD_IPC_CMD_CLUSTER_IN_TRAN)
-	{
-		ereport(WARNING,
-				(errmsg("degenerate backend set command failed"),
-				 errdetail("watchdog cluster is not in stable state"),
-					errhint("try again when the cluster is fully initialized")));
-		return CLUSTER_IN_TRANSATIONING;
-	}
-	else if (type == WD_IPC_CMD_TIMEOUT)
-	{
-		ereport(WARNING,
-				(errmsg("degenerate backend set command failed"),
-				 errdetail("ipc command timeout")));
-		return COMMAND_TIMEOUT;
-	}
-	else if (type == WD_IPC_CMD_RESULT_OK)
-	{
-		return COMMAND_OK;
-	}
-	
-	return COMMAND_FAILED;
+	return res;
 }
 
-WdCommandResult
-wd_promote_backend(int node_id)
+WDFailoverCMDResults
+wd_promote_backend(int node_id, unsigned int *wd_failover_id)
 {
+	WDFailoverCMDResults res;
 	int n = node_id;
-	char type;
 	char* func;
 	WDIPCCmdResult *result;
 	unsigned int *shared_key = get_ipc_shared_key();
-
-	/* if promote packet is received already, do nothing */
-	if (wd_chk_node_mask_for_promote_req(&n,1))
-		return COMMAND_OK;
 	
 	func = get_wd_node_function_json(WD_FUNCTION_PROMOTE_REQUEST,&n, 1,
 									 shared_key?*shared_key:0,pool_config->wd_authkey);
-	result = issue_command_to_watchdog(WD_FUNCTION_COMMAND ,
+	result = issue_command_to_watchdog(WD_IPC_FAILOVER_COMMAND,
 									   WD_DEFAULT_IPC_COMMAND_TIMEOUT,
 									   func, strlen(func), true);
 	pfree(func);
-
-	if (result == NULL)
-	{
-		ereport(WARNING,
-			(errmsg("promote backend node command failed"),
-				 errdetail("issue command to watchdog returned NULL")));
-		return COMMAND_FAILED;
-	}
-	
-	type = result->type;
+	res = wd_get_failover_result_from_data(result, wd_failover_id);
 	FreeCmdResult(result);
-	if (type == WD_IPC_CMD_CLUSTER_IN_TRAN)
-	{
-		ereport(WARNING,
-			(errmsg("promote backend node command failed"),
-				 errdetail("watchdog cluster is not in stable state"),
-					errhint("try again when the cluster is fully initialized")));
-		return CLUSTER_IN_TRANSATIONING;
-	}
-	else if (type == WD_IPC_CMD_TIMEOUT)
-	{
-		ereport(WARNING,
-			(errmsg("promote backend node command failed"),
-				 errdetail("ipc command timeout")));
-		return COMMAND_TIMEOUT;
-	}
-	else if (type == WD_IPC_CMD_RESULT_OK)
-	{
-		return COMMAND_OK;
-	}
-	
-	return COMMAND_FAILED;
+	return res;
 }
 
 /*
@@ -760,42 +656,42 @@ open_wd_command_sock(bool throw_error)
 	return sock;
 }
 
-WDFailoverCMDResults wd_start_failover_interlocking(void)
+WDFailoverCMDResults wd_start_failover_interlocking(unsigned int wd_failover_id)
 {
 	if (pool_config->use_watchdog)
-		return wd_issue_failover_lock_command(WD_REQ_FAILOVER_START, 0);
+		return wd_issue_failover_lock_command(WD_REQ_FAILOVER_START, 0, wd_failover_id);
 	return FAILOVER_RES_I_AM_LOCK_HOLDER;
 }
 
-WDFailoverCMDResults wd_end_failover_interlocking(void)
+WDFailoverCMDResults wd_end_failover_interlocking(unsigned int wd_failover_id)
 {
 	if (pool_config->use_watchdog)
-		return wd_issue_failover_lock_command(WD_REQ_FAILOVER_END, 0);
+		return wd_issue_failover_lock_command(WD_REQ_FAILOVER_END, 0, wd_failover_id);
 	return FAILOVER_RES_SUCCESS;
 }
 
-WDFailoverCMDResults wd_failover_lock_release(enum WDFailoverLocks lock)
+WDFailoverCMDResults wd_failover_lock_release(enum WDFailoverLocks lock, unsigned int wd_failover_id)
 {
 	if (pool_config->use_watchdog)
-		return wd_issue_failover_lock_command(WD_REQ_FAILOVER_RELEASE_LOCK, lock);
+		return wd_issue_failover_lock_command(WD_REQ_FAILOVER_RELEASE_LOCK, lock, wd_failover_id);
 	return FAILOVER_RES_SUCCESS;
 }
 
-WDFailoverCMDResults wd_failover_lock_status(enum WDFailoverLocks lock)
+WDFailoverCMDResults wd_failover_lock_status(enum WDFailoverLocks lock, unsigned int wd_failover_id)
 {
 	if (pool_config->use_watchdog)
-		return wd_issue_failover_lock_command(WD_REQ_FAILOVER_LOCK_STATUS, lock);
+		return wd_issue_failover_lock_command(WD_REQ_FAILOVER_LOCK_STATUS, lock, wd_failover_id);
 	return FAILOVER_RES_UNLOCKED;
 }
 
-void wd_wait_until_command_complete_or_timeout(enum WDFailoverLocks lock)
+void wd_wait_until_command_complete_or_timeout(enum WDFailoverLocks lock, unsigned int wd_failover_id)
 {
 	WDFailoverCMDResults res = FAILOVER_RES_TRANSITION;
 	int	count = WD_INTERLOCK_WAIT_COUNT;
 
 	while (pool_config->use_watchdog)
 	{
-		res = wd_failover_lock_status(lock);
+		res = wd_failover_lock_status(lock, wd_failover_id);
 		if (res == FAILOVER_RES_UNLOCKED ||
 			res == FAILOVER_RES_NO_LOCKHOLDER)
 		{
@@ -818,16 +714,16 @@ void wd_wait_until_command_complete_or_timeout(enum WDFailoverLocks lock)
  * if watchdog is in transition state
  */
 
-static WDFailoverCMDResults wd_issue_failover_lock_command(char* syncReqType, enum WDFailoverLocks lockID)
+static WDFailoverCMDResults wd_issue_failover_lock_command(char* syncReqType, enum WDFailoverLocks lockID, unsigned int wd_failover_id)
 {
 	WDFailoverCMDResults res;
 	int x;
-	for (x=0; x < MAX_SEC_WAIT_FOR_CLUSTER_TRANSATION; x++)
+	for (x=0; x < MAX_SEC_WAIT_FOR_CLUSTER_TRANSATION/2; x++)
 	{
-		res = wd_send_failover_sync_command(syncReqType, lockID);
+		res = wd_send_failover_sync_command(syncReqType, lockID, wd_failover_id);
 		if (res != FAILOVER_RES_TRANSITION)
 			break;
-		sleep(1);
+		sleep(2);
 	}
 	return res;
 }
@@ -840,78 +736,12 @@ sleep_in_waiting(void)
 }
 
 
-/* check mask, and if maskted return 1 and clear it, otherwise return 0 */
-static int
-wd_chk_node_mask (unsigned char req_mask, int *node_id_set, int count)
-{
-	int rtn = 0;
-	int i;
-	int offset = 0;
-	for ( i = 0 ; i < count ; i ++)
-	{
-		offset = *(node_id_set+i);
-		if ((*(WD_Node_List + offset) & req_mask) != 0)
-		{
-			*(WD_Node_List + offset) ^= req_mask;
-			rtn = 1;
-		}
-	}
-	return rtn;
-}
-
-/* set mask */
-static int
-wd_set_node_mask (unsigned char req_mask, int *node_id_set, int count)
-{
-	int i;
-	int offset = 0;
-	for ( i = 0 ; i < count ; i ++)
-	{
-		offset = *(node_id_set+i);
-		*(WD_Node_List + offset) |= req_mask;
-	}
-	return 0;
-}
-
-
-int
-wd_set_node_mask_for_failback_req(int *node_id_set, int count)
-{
-	return wd_set_node_mask (FAILBACK_REQUEST_NODE_MASK, node_id_set, count);
-}
-
-int
-wd_set_node_mask_for_degenerate_req(int *node_id_set, int count)
-{
-	return wd_set_node_mask (DEGENERATE_REQUEST_NODE_MASK, node_id_set, count);
-}
-
-int
-wd_set_node_mask_for_promote_req(int *node_id_set, int count)
-{
-	return wd_set_node_mask (PROMOTE_REQUEST_NODE_MASK, node_id_set, count);
-}
-
-int
-wd_chk_node_mask_for_failback_req(int *node_id_set, int count)
-{
-	return wd_chk_node_mask (FAILBACK_REQUEST_NODE_MASK, node_id_set, count);
-}
-
-int
-wd_chk_node_mask_for_degenerate_req(int *node_id_set, int count)
-{
-	return wd_chk_node_mask (DEGENERATE_REQUEST_NODE_MASK, node_id_set, count);
-}
-
-int
-wd_chk_node_mask_for_promote_req(int *node_id_set, int count)
-{
-	return wd_chk_node_mask (PROMOTE_REQUEST_NODE_MASK, node_id_set, count);
-}
 
 static void FreeCmdResult(WDIPCCmdResult* res)
 {
+	if (res == NULL)
+		return;
+
 	if (res->data)
 		pfree(res->data);
 	pfree (res);
diff --git a/src/watchdog/wd_json_data.c b/src/watchdog/wd_json_data.c
index 693d2d1..572f8f0 100644
--- a/src/watchdog/wd_json_data.c
+++ b/src/watchdog/wd_json_data.c
@@ -617,7 +617,7 @@ bool parse_wd_node_function_json(char* json_data, int data_len, char** func_name
 		json_value_free(root);
 		ereport(LOG,
 			(errmsg("watchdog is unable to parse node function json"),
-				 errdetail("invalid json data \"%s\"",json_data)));
+				 errdetail("invalid json data \"%.*s\"",data_len,json_data)));
 		return false;
 	}
 	ptr = json_get_string_value_for_key(root, "Function");
@@ -679,12 +679,12 @@ bool parse_wd_node_function_json(char* json_data, int data_len, char** func_name
 	return true;
 }
 
-char* get_wd_simple_error_message_json(char* message)
+char* get_wd_simple_message_json(char* message)
 {
 	char* json_str;
 	JsonNode* jNode = jw_create_with_object(true);
 
-	jw_put_string(jNode, "ERROR", message);
+	jw_put_string(jNode, "MESSAGE", message);
 	jw_finish_document(jNode);
 	json_str = pstrdup(jw_get_json_string(jNode));
 	jw_destroy(jNode);
wd_rewamp_failover.diff (159,721 bytes)

Muhammad Usama

2016-11-01 00:43

developer   ~0001143

Hi

First of all sorry for the late reply. Basically the issue was a little more deep rooted and need a design change so it took a me a long time to fix it.
Can you please try out the latest attached patch(wd_rewamp_failover.diff) to see if it behaves as expected.
I am not finished with testing it as yet and patch might also have some extra debug infos but want to share the early version with you to make sure we can get it out as early as possible.

Best regards

supp_k

2016-11-01 20:01

reporter   ~0001145

Hi Muhammad,

we have environment that consists of 3 pgpool nodes.

Skim verification on CentOS 6.8 x86_64 reveals that the patch doesn't solve the problem. The failover action is performed by one node occasionally.

We tested several cases including full down of a pgpool node (poweroff). In case one node disappeares it works but when there are all three pgpool nodes active and one postgres backend downs then it doesn't work.

Muhammad Usama

2016-11-01 22:29

developer   ~0001146

Hi, Thanks for getting back on this.

I am also testing with three nodes cluster using the centos on openstack but not able to reproduce the mentioned problem. Do you mean when one of the bcakend PostgreSQL servers goes down, then pgpool-II failover does not happen? Or does the failover actually happens, but not on all pgpool-II nodes?
Also can you please share the pgpool.confs and logs of all nodes for the failing scenario?

supp_k

2016-11-02 17:39

reporter   ~0001150

Hi Muhammad,

we have 3 servers:
1) Pgpool
2) Pgpool Master + Postgres Master A
3) Pgpool + Postgres B

Emulate cases:
1) Poweroff server 0000002 or 0000003.
     Result: New Pgpool master elected; Failover performed OK
3) Kill Postgres A or B.
     Result: Failover not performed.

Muhammad Usama

2016-11-02 19:36

developer   ~0001151

I have also tested the same scenario, but somehow it is working on my side. Can you please share the pgpool-II log and configuration files.

Thanks and regards!

supp_k

2016-11-10 21:16

reporter   ~0001162

If I kill master postgres process then I see the failover works. But the pgpool cluster doesnt provide any answers to SQL queries despite the fact the failover is complete.

Please see the attached log files.

server_1 (91,125 bytes)
Nov 10 14:28:29 srv-2268613 rsyslogd-2177: imuxsock lost 1350 messages from pid 16279 due to rate-limiting
Nov 10 14:28:31 srv-2268613 pgpool[16279]: [954655-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16279]: [954655-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16279]: [954656-1] LOG:  received degenerate backend request for node_id: 0 from pid [16279]
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [837-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16285]: [3713-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16285]: [3713-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16285]: [3714-1] LOG:  received degenerate backend request for node_id: 0 from pid [16285]
Nov 10 14:28:31 srv-2268613 pgpool[16286]: [3730-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16289]: [7659-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16281]: [4059-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16282]: [3880-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16287]: [16162-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16283]: [6200-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16280]: [3737-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16286]: [3730-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16289]: [7659-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16281]: [4059-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16282]: [3880-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16287]: [16162-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16283]: [6200-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16280]: [3737-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16286]: [3731-1] LOG:  received degenerate backend request for node_id: 0 from pid [16286]
Nov 10 14:28:31 srv-2268613 pgpool[16289]: [7660-1] LOG:  received degenerate backend request for node_id: 0 from pid [16289]
Nov 10 14:28:31 srv-2268613 pgpool[16281]: [4060-1] LOG:  received degenerate backend request for node_id: 0 from pid [16281]
Nov 10 14:28:31 srv-2268613 pgpool[16282]: [3881-1] LOG:  received degenerate backend request for node_id: 0 from pid [16282]
Nov 10 14:28:31 srv-2268613 pgpool[16287]: [16163-1] LOG:  received degenerate backend request for node_id: 0 from pid [16287]
Nov 10 14:28:31 srv-2268613 pgpool[16283]: [6201-1] LOG:  received degenerate backend request for node_id: 0 from pid [16283]
Nov 10 14:28:31 srv-2268613 pgpool[16280]: [3738-1] LOG:  received degenerate backend request for node_id: 0 from pid [16280]
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [845-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [856-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [862-1] LOG:  remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" is requesting to become a lock holder for failover ID: 37
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [863-1] LOG:  request to become a lock holder is denied to remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432"
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [863-2] DETAIL:  only master/coordinator can become a lock holder
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [870-1] LOG:  remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" is requesting to become a lock holder for failover ID: 37
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [871-1] LOG:  request to become a lock holder is denied to remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432"
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [871-2] DETAIL:  only master/coordinator can become a lock holder
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [873-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [875-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16285]: [3715-1] LOG:  degenerate backend request for 1 node(s) from pid [16285] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [876-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [878-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16286]: [3732-1] LOG:  degenerate backend request for 1 node(s) from pid [16286] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [879-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [881-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16289]: [7661-1] LOG:  degenerate backend request for 1 node(s) from pid [16289] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [882-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [884-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16281]: [4061-1] LOG:  degenerate backend request for 1 node(s) from pid [16281] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [885-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [887-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16282]: [3882-1] LOG:  degenerate backend request for 1 node(s) from pid [16282] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [888-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [890-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16287]: [16164-1] LOG:  degenerate backend request for 1 node(s) from pid [16287] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [891-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[16283]: [6202-1] LOG:  degenerate backend request for 1 node(s) from pid [16283] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [898-1] LOG:  remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" is checking the status of [FAILOVER] lock for failover ID 37
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [899-1] LOG:  [FAILOVER] lock status check request from remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" for failover ID 37
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [899-2] DETAIL:  but failover is not yet started by master node
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [901-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[16280]: [3739-1] LOG:  degenerate backend request for 1 node(s) from pid [16280] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [903-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [904-1] LOG:  processing sync request from IPC socket
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [905-1] LOG:  local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432" is requesting to become a lock holder for failover ID: 37
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [906-1] LOG:  local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432" is the lock holder
Nov 10 14:28:31 srv-2268613 pgpool[15715]: [2542-1] LOG:  starting degeneration. shutdown host a.db.node(15432)
Nov 10 14:28:31 srv-2268613 pgpool[15715]: [2544-1] LOG:  Restart all children
Nov 10 14:28:31 srv-2268613 pgpool[16278]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16279]: [954658-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16280]: [3740-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16281]: [4062-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16282]: [3883-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16283]: [6203-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16284]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16285]: [3716-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16286]: [3733-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16287]: [16165-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16288]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16289]: [7662-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16290]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16291]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16292]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16294]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 rsyslogd-2177: imuxsock lost 670 messages from pid 16295 due to rate-limiting
Nov 10 14:28:31 srv-2268613 pgpool[16295]: [2132-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16293]: [1551-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 rsyslogd-2177: imuxsock lost 1258 messages from pid 16296 due to rate-limiting
Nov 10 14:28:31 srv-2268613 pgpool[16296]: [2534-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16297]: [1550-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16298]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16299]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16300]: [1550-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16301]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16303]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16304]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16305]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16306]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16307]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16308]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16309]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16310]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16311]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16312]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16313]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16314]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16315]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16316]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16317]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16318]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16319]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16320]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16321]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16322]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16323]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16324]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16325]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16326]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16327]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16328]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16329]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16330]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16331]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16332]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16333]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16334]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16335]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16336]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16337]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16338]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16339]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16340]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16341]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16342]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16343]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16344]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16345]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16346]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16347]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16348]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16349]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16350]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16351]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16352]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16353]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16354]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 rsyslogd-2177: imuxsock begins to drop messages from pid 15715 due to rate-limiting
Nov 10 14:28:31 srv-2268613 pgpool[16364]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16355]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16365]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16356]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16357]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16358]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16360]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16361]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16362]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16363]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16359]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16366]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16367]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16368]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16369]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16370]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16371]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16372]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16373]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16374]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16375]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16376]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16377]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16379]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16378]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16380]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16381]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16382]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16383]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16384]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16385]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16386]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16387]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16388]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16389]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16390]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16391]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16392]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16393]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16394]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16395]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16398]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16397]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16396]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16399]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16400]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16401]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16403]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16402]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16404]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16405]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16406]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16407]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16408]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16409]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16410]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16411]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16412]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16413]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16414]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16415]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16416]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16417]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16418]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16419]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16420]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16421]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16422]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16423]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16424]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16425]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16426]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16427]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16428]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16429]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16430]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16431]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16432]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16433]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16434]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16435]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16436]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16437]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16438]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16439]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16440]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16441]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16442]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16443]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16444]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16445]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16446]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16447]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16448]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16449]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [913-1] LOG:  remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" is checking the status of [FAILOVER] lock for failover ID 37
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [914-1] LOG:  FAILOVER lock is currently LOCKED
Nov 10 14:28:31 srv-2268613 pgpool[16450]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [914-2] DETAIL:  request was from remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:31 srv-2268613 pgpool[16451]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16452]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16453]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16454]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16455]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16456]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16457]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16458]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16459]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16460]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16461]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16462]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16463]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16464]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16465]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16466]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16467]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16468]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16469]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16470]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16471]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16472]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16473]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16474]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16475]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16476]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16477]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16478]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16479]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16480]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16481]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16482]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16483]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16484]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16485]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16486]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16487]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16488]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16489]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16490]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16491]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16492]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16493]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16494]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16495]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16496]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16497]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16498]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16499]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16500]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16501]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16502]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16503]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16504]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16505]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16506]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16507]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16508]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16509]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16510]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16511]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16512]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16513]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16514]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16515]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16516]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16517]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 root: Failover event. Parameters:[0 a.db.node 15432 /var/lib/pgsql/9.5/data 1 b.db.node 0 0 15432 /var/lib/pgsql/9.5/data]
Nov 10 14:28:31 srv-2268613 root: Make sure the master 'b.db.node' is async
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [921-1] LOG:  remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" is checking the status of [FAILOVER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [922-1] LOG:  FAILOVER lock is currently LOCKED
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [922-2] DETAIL:  request was from remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [945-1] LOG:  new IPC connection received
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [946-1] LOG:  processing sync request from IPC socket
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [947-1] LOG:  local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432" is requesting to release [FAILOVER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [948-1] LOG:  local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432" has released the [FAILOVER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 rsyslogd-2177: imuxsock lost 318 messages from pid 15715 due to rate-limiting
Nov 10 14:28:32 srv-2268613 pgpool[15715]: [2786-1] LOG:  find_primary_node_repeatedly: waiting for finding a primary node
Nov 10 14:28:32 srv-2268613 pgpool[15715]: [2787-1] LOG:  find_primary_node: checking backend no 0
Nov 10 14:28:32 srv-2268613 pgpool[15715]: [2788-1] LOG:  find_primary_node: checking backend no 1
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [955-1] LOG:  remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" is checking the status of [FAILOVER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [956-1] LOG:  FAILOVER lock is currently FREE
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [956-2] DETAIL:  request was from remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [963-1] LOG:  remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" is checking the status of [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [964-1] LOG:  FOLLOW MASTER lock is currently LOCKED
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [964-2] DETAIL:  request was from remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [971-1] LOG:  remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" is checking the status of [FAILOVER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [972-1] LOG:  FAILOVER lock is currently FREE
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [972-2] DETAIL:  request was from remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [979-1] LOG:  remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" is checking the status of [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [980-1] LOG:  FOLLOW MASTER lock is currently LOCKED
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [980-2] DETAIL:  request was from remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [987-1] LOG:  remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" is checking the status of [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [988-1] LOG:  FOLLOW MASTER lock is currently LOCKED
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [988-2] DETAIL:  request was from remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [995-1] LOG:  remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" is checking the status of [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [996-1] LOG:  FOLLOW MASTER lock is currently LOCKED
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [996-2] DETAIL:  request was from remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:33 srv-2268613 pgpool[15715]: [2816-1] LOG:  find_primary_node: checking backend no 0
Nov 10 14:28:33 srv-2268613 pgpool[15715]: [2817-1] LOG:  find_primary_node: checking backend no 1
Nov 10 14:28:33 srv-2268613 pgpool[15715]: [2839-1] LOG:  find_primary_node: primary node id is 1
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [998-1] LOG:  new IPC connection received
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [999-1] LOG:  processing sync request from IPC socket
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [1000-1] LOG:  local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432" is requesting to release [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [1001-1] LOG:  local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432" has released the [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:33 srv-2268613 pgpool[15715]: [2840-1] LOG:  failover: set new primary node: 1
Nov 10 14:28:33 srv-2268613 pgpool[15715]: [2841-1] LOG:  failover: set new master node: 1
Nov 10 14:28:33 srv-2268613 pgpool[16657]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16657]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16658]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16658]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16686]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16686]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16689]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16689]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16690]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16690]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16691]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16691]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16676]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16674]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16674]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16679]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16679]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16677]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16677]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16680]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16680]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16684]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16684]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16685]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16685]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16683]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16683]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16682]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16682]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16681]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16681]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16688]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16688]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16673]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16673]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16687]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16687]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16672]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16672]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16653]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16653]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16675]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16675]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16654]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16654]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16678]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16678]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16676]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16655]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16655]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16698]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16698]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16699]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16699]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16731]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16700]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16700]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16731]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16656]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16656]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16732]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16732]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16733]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16733]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16701]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16701]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16702]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16702]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16734]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16734]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16703]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16703]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16735]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16735]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16704]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16704]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16753]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16753]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16705]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16705]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16737]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16737]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16706]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16706]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16751]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16751]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16707]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16707]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16660]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16660]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16739]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16739]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16708]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16708]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16754]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16754]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16709]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16709]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16743]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16743]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16659]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16659]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16661]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16661]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16752]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16752]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16662]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16662]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16738]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16693]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16693]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16738]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16694]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16694]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16741]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16741]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16663]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16663]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16746]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16746]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16664]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16664]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16742]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16742]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16695]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16695]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16749]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16749]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16697]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16697]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16745]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16745]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16696]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16696]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16665]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16665]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16758]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16758]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16666]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16666]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16750]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16750]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16667]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16667]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16670]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16670]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16759]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16759]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16748]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16748]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16671]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16671]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16744]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16744]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16692]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16692]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16740]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16740]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16710]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16710]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16747]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16747]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16755]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16755]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16711]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16711]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16736]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16736]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16757]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16757]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16712]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16712]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16756]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16756]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16713]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16713]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16760]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16760]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16714]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16714]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16715]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16715]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16716]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16716]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16761]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16652]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16761]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16652]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16717]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16717]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16718]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16718]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16762]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16762]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16719]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16719]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16720]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16720]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16763]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16763]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16721]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16721]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16722]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16722]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16726]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16726]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16723]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16723]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16727]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16727]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16724]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16724]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16728]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16728]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16725]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16725]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16651]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16651]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16729]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16729]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16730]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16730]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16771]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16771]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [1008-1] LOG:  remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" is checking the status of [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:33 srv-2268613 rsyslogd-2177: imuxsock begins to drop messages from pid 15717 due to rate-limiting
Nov 10 14:28:33 srv-2268613 pgpool[16767]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16767]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16768]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16768]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16769]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16770]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16770]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16769]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16772]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16772]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16766]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16766]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16773]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16773]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16774]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16774]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16764]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16764]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16775]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16775]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16791]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16791]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16792]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16792]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16793]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16793]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16794]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16794]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16776]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16776]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16795]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16795]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16777]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16777]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16796]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16796]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16778]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16778]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16779]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16779]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16798]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16798]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16780]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16780]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16800]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16800]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16781]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16781]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16765]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16765]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16802]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16802]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16782]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16782]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16803]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16803]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16783]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16783]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16799]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16799]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16784]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16784]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16797]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16797]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16785]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16785]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16801]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16801]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16786]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16786]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16787]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16787]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16805]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16805]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16788]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16788]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16789]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16789]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16807]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16790]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16790]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16806]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16804]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16807]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16804]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16806]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16809]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16809]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16810]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16810]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16817]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16817]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16813]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16813]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16818]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16818]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16814]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16814]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16812]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16812]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16820]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16820]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16815]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16815]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16821]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16821]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16816]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16816]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16822]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16822]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16808]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16811]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16808]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16811]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16819]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16819]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16823]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16823]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16824]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16824]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16829]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16829]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16830]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16830]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16831]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16831]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16832]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16832]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16833]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16833]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16834]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16834]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16848]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16827]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16827]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16848]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16828]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16828]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16836]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16836]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16849]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16849]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16837]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16837]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16850]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16850]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16838]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16838]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16839]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16839]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16851]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16851]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16840]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16840]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16852]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16852]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16835]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16835]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16853]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16853]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16841]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16841]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16842]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16842]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16854]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16854]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16843]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16843]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16855]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16855]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16845]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16845]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16856]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16856]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16844]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16844]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16846]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16846]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16857]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16857]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16847]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16847]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16858]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16858]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16859]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16859]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16825]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16825]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16826]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16826]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16860]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16860]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16866]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16866]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16861]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16861]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16862]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16862]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16863]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16863]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16864]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16864]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16584]: [2390-1] LOG:  worker process received restart request
Nov 10 14:28:33 srv-2268613 pgpool[16865]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16865]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[15715]: [2842-1] LOG:  failover done. shutdown host a.db.node(15432)
Nov 10 14:28:33 srv-2268613 pgpool[16869]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16869]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16870]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16870]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16871]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16871]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16872]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16872]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16867]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16867]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16873]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16873]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16874]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16874]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16868]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16868]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16875]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16875]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16876]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16876]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16877]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16877]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16878]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16878]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16879]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16879]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16880]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16880]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16881]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16881]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16882]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16882]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16883]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16883]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16884]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16884]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16885]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16885]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16886]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16886]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16887]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16887]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16888]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16888]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16889]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16889]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16890]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16890]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:34 srv-2268613 pgpool[16583]: [2400-1] LOG:  restart request received in pcp child process
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2843-1] LOG:  PCP child 16583 exits with status 0 in failover()
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2844-1] LOG:  fork a new PCP child pid 16891 in failover()
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2847-1] LOG:  child process with pid: 16278 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2848-1] LOG:  child process with pid: 16278 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2850-1] LOG:  child process with pid: 16279 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2851-1] LOG:  child process with pid: 16279 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2853-1] LOG:  child process with pid: 16280 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2854-1] LOG:  child process with pid: 16280 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2856-1] LOG:  child process with pid: 16281 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2857-1] LOG:  child process with pid: 16281 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2859-1] LOG:  child process with pid: 16282 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2860-1] LOG:  child process with pid: 16282 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2862-1] LOG:  child process with pid: 16283 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2863-1] LOG:  child process with pid: 16283 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2865-1] LOG:  child process with pid: 16284 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2866-1] LOG:  child process with pid: 16284 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2868-1] LOG:  child process with pid: 16285 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2869-1] LOG:  child process with pid: 16285 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2871-1] LOG:  child process with pid: 16286 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2872-1] LOG:  child process with pid: 16286 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2874-1] LOG:  child process with pid: 16287 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2875-1] LOG:  child process with pid: 16287 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2877-1] LOG:  child process with pid: 16288 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2878-1] LOG:  child process with pid: 16288 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2880-1] LOG:  child process with pid: 16289 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2881-1] LOG:  child process with pid: 16289 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2883-1] LOG:  child process with pid: 16290 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2884-1] LOG:  child process with pid: 16290 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2886-1] LOG:  child process with pid: 16291 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2887-1] LOG:  child process with pid: 16291 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2889-1] LOG:  child process with pid: 16292 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2890-1] LOG:  child process with pid: 16292 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2892-1] LOG:  child process with pid: 16293 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2893-1] LOG:  child process with pid: 16293 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2895-1] LOG:  child process with pid: 16294 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2896-1] LOG:  child process with pid: 16294 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2898-1] LOG:  child process with pid: 16295 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2899-1] LOG:  child process with pid: 16295 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2901-1] LOG:  child process with pid: 16296 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2902-1] LOG:  child process with pid: 16296 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2904-1] LOG:  child process with pid: 16297 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2905-1] LOG:  child process with pid: 16297 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2907-1] LOG:  child process with pid: 16298 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2908-1] LOG:  child process with pid: 16298 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2910-1] LOG:  child process with pid: 16299 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2911-1] LOG:  child process with pid: 16299 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2913-1] LOG:  child process with pid: 16300 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2914-1] LOG:  child process with pid: 16300 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2916-1] LOG:  child process with pid: 16301 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2917-1] LOG:  child process with pid: 16301 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2919-1] LOG:  child process with pid: 16303 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2920-1] LOG:  child process with pid: 16303 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2922-1] LOG:  child process with pid: 16304 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2923-1] LOG:  child process with pid: 16304 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2925-1] LOG:  child process with pid: 16305 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2926-1] LOG:  child process with pid: 16305 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2928-1] LOG:  child process with pid: 16306 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2929-1] LOG:  child process with pid: 16306 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2931-1] LOG:  child process with pid: 16307 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2932-1] LOG:  child process with pid: 16307 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2934-1] LOG:  child process with pid: 16308 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2935-1] LOG:  child process with pid: 16308 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2937-1] LOG:  child process with pid: 16309 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2938-1] LOG:  child process with pid: 16309 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2940-1] LOG:  child process with pid: 16310 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2941-1] LOG:  child process with pid: 16310 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2943-1] LOG:  child process with pid: 16311 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2944-1] LOG:  child process with pid: 16311 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2946-1] LOG:  child process with pid: 16312 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2947-1] LOG:  child process with pid: 16312 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2949-1] LOG:  child process with pid: 16313 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2950-1] LOG:  child process with pid: 16313 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2952-1] LOG:  child process with pid: 16314 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2953-1] LOG:  child process with pid: 16314 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2955-1] LOG:  child process with pid: 16315 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2956-1] LOG:  child process with pid: 16315 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2958-1] LOG:  child process with pid: 16316 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2959-1] LOG:  child process with pid: 16316 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2961-1] LOG:  child process with pid: 16317 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2962-1] LOG:  child process with pid: 16317 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2964-1] LOG:  child process with pid: 16318 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2965-1] LOG:  child process with pid: 16318 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2967-1] LOG:  child process with pid: 16319 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2968-1] LOG:  child process with pid: 16319 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2970-1] LOG:  child process with pid: 16320 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2971-1] LOG:  child process with pid: 16320 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2973-1] LOG:  child process with pid: 16321 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2974-1] LOG:  child process with pid: 16321 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2976-1] LOG:  child process with pid: 16322 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2977-1] LOG:  child process with pid: 16322 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2979-1] LOG:  child process with pid: 16323 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2980-1] LOG:  child process with pid: 16323 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2982-1] LOG:  child process with pid: 16324 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2983-1] LOG:  child process with pid: 16324 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2985-1] LOG:  child process with pid: 16325 exits with status 0
Nov 10 14:28:34 srv-2268613 rsyslogd-2177: imuxsock begins to drop messages from pid 15715 due to rate-limiting
Nov 10 14:28:42 srv-2268613 rsyslogd-2177: imuxsock lost 19 messages from pid 15717 due to rate-limiting
^C
[root@srv-2268613 ~]# 

server_1 (91,125 bytes)

supp_k

2016-11-10 21:16

reporter  

server_1-2 (91,125 bytes)
Nov 10 14:28:29 srv-2268613 rsyslogd-2177: imuxsock lost 1350 messages from pid 16279 due to rate-limiting
Nov 10 14:28:31 srv-2268613 pgpool[16279]: [954655-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16279]: [954655-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16279]: [954656-1] LOG:  received degenerate backend request for node_id: 0 from pid [16279]
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [837-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16285]: [3713-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16285]: [3713-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16285]: [3714-1] LOG:  received degenerate backend request for node_id: 0 from pid [16285]
Nov 10 14:28:31 srv-2268613 pgpool[16286]: [3730-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16289]: [7659-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16281]: [4059-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16282]: [3880-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16287]: [16162-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16283]: [6200-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16280]: [3737-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16286]: [3730-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16289]: [7659-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16281]: [4059-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16282]: [3880-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16287]: [16162-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16283]: [6200-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16280]: [3737-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16286]: [3731-1] LOG:  received degenerate backend request for node_id: 0 from pid [16286]
Nov 10 14:28:31 srv-2268613 pgpool[16289]: [7660-1] LOG:  received degenerate backend request for node_id: 0 from pid [16289]
Nov 10 14:28:31 srv-2268613 pgpool[16281]: [4060-1] LOG:  received degenerate backend request for node_id: 0 from pid [16281]
Nov 10 14:28:31 srv-2268613 pgpool[16282]: [3881-1] LOG:  received degenerate backend request for node_id: 0 from pid [16282]
Nov 10 14:28:31 srv-2268613 pgpool[16287]: [16163-1] LOG:  received degenerate backend request for node_id: 0 from pid [16287]
Nov 10 14:28:31 srv-2268613 pgpool[16283]: [6201-1] LOG:  received degenerate backend request for node_id: 0 from pid [16283]
Nov 10 14:28:31 srv-2268613 pgpool[16280]: [3738-1] LOG:  received degenerate backend request for node_id: 0 from pid [16280]
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [845-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [856-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [862-1] LOG:  remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" is requesting to become a lock holder for failover ID: 37
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [863-1] LOG:  request to become a lock holder is denied to remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432"
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [863-2] DETAIL:  only master/coordinator can become a lock holder
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [870-1] LOG:  remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" is requesting to become a lock holder for failover ID: 37
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [871-1] LOG:  request to become a lock holder is denied to remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432"
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [871-2] DETAIL:  only master/coordinator can become a lock holder
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [873-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [875-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16285]: [3715-1] LOG:  degenerate backend request for 1 node(s) from pid [16285] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [876-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [878-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16286]: [3732-1] LOG:  degenerate backend request for 1 node(s) from pid [16286] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [879-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [881-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16289]: [7661-1] LOG:  degenerate backend request for 1 node(s) from pid [16289] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [882-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [884-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16281]: [4061-1] LOG:  degenerate backend request for 1 node(s) from pid [16281] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [885-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [887-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16282]: [3882-1] LOG:  degenerate backend request for 1 node(s) from pid [16282] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [888-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [890-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16287]: [16164-1] LOG:  degenerate backend request for 1 node(s) from pid [16287] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [891-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[16283]: [6202-1] LOG:  degenerate backend request for 1 node(s) from pid [16283] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [898-1] LOG:  remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" is checking the status of [FAILOVER] lock for failover ID 37
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [899-1] LOG:  [FAILOVER] lock status check request from remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" for failover ID 37
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [899-2] DETAIL:  but failover is not yet started by master node
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [901-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[16280]: [3739-1] LOG:  degenerate backend request for 1 node(s) from pid [16280] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [903-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [904-1] LOG:  processing sync request from IPC socket
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [905-1] LOG:  local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432" is requesting to become a lock holder for failover ID: 37
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [906-1] LOG:  local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432" is the lock holder
Nov 10 14:28:31 srv-2268613 pgpool[15715]: [2542-1] LOG:  starting degeneration. shutdown host a.db.node(15432)
Nov 10 14:28:31 srv-2268613 pgpool[15715]: [2544-1] LOG:  Restart all children
Nov 10 14:28:31 srv-2268613 pgpool[16278]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16279]: [954658-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16280]: [3740-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16281]: [4062-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16282]: [3883-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16283]: [6203-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16284]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16285]: [3716-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16286]: [3733-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16287]: [16165-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16288]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16289]: [7662-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16290]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16291]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16292]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16294]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 rsyslogd-2177: imuxsock lost 670 messages from pid 16295 due to rate-limiting
Nov 10 14:28:31 srv-2268613 pgpool[16295]: [2132-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16293]: [1551-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 rsyslogd-2177: imuxsock lost 1258 messages from pid 16296 due to rate-limiting
Nov 10 14:28:31 srv-2268613 pgpool[16296]: [2534-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16297]: [1550-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16298]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16299]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16300]: [1550-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16301]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16303]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16304]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16305]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16306]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16307]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16308]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16309]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16310]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16311]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16312]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16313]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16314]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16315]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16316]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16317]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16318]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16319]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16320]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16321]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16322]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16323]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16324]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16325]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16326]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16327]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16328]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16329]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16330]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16331]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16332]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16333]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16334]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16335]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16336]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16337]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16338]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16339]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16340]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16341]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16342]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16343]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16344]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16345]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16346]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16347]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16348]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16349]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16350]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16351]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16352]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16353]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16354]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 rsyslogd-2177: imuxsock begins to drop messages from pid 15715 due to rate-limiting
Nov 10 14:28:31 srv-2268613 pgpool[16364]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16355]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16365]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16356]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16357]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16358]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16360]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16361]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16362]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16363]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16359]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16366]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16367]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16368]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16369]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16370]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16371]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16372]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16373]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16374]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16375]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16376]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16377]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16379]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16378]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16380]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16381]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16382]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16383]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16384]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16385]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16386]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16387]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16388]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16389]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16390]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16391]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16392]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16393]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16394]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16395]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16398]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16397]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16396]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16399]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16400]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16401]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16403]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16402]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16404]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16405]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16406]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16407]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16408]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16409]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16410]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16411]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16412]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16413]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16414]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16415]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16416]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16417]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16418]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16419]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16420]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16421]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16422]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16423]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16424]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16425]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16426]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16427]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16428]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16429]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16430]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16431]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16432]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16433]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16434]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16435]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16436]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16437]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16438]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16439]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16440]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16441]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16442]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16443]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16444]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16445]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16446]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16447]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16448]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16449]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [913-1] LOG:  remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" is checking the status of [FAILOVER] lock for failover ID 37
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [914-1] LOG:  FAILOVER lock is currently LOCKED
Nov 10 14:28:31 srv-2268613 pgpool[16450]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [914-2] DETAIL:  request was from remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:31 srv-2268613 pgpool[16451]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16452]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16453]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16454]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16455]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16456]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16457]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16458]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16459]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16460]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16461]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16462]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16463]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16464]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16465]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16466]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16467]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16468]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16469]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16470]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16471]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16472]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16473]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16474]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16475]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16476]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16477]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16478]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16479]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16480]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16481]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16482]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16483]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16484]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16485]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16486]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16487]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16488]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16489]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16490]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16491]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16492]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16493]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16494]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16495]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16496]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16497]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16498]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16499]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16500]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16501]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16502]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16503]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16504]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16505]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16506]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16507]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16508]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16509]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16510]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16511]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16512]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16513]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16514]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16515]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16516]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16517]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 root: Failover event. Parameters:[0 a.db.node 15432 /var/lib/pgsql/9.5/data 1 b.db.node 0 0 15432 /var/lib/pgsql/9.5/data]
Nov 10 14:28:31 srv-2268613 root: Make sure the master 'b.db.node' is async
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [921-1] LOG:  remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" is checking the status of [FAILOVER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [922-1] LOG:  FAILOVER lock is currently LOCKED
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [922-2] DETAIL:  request was from remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [945-1] LOG:  new IPC connection received
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [946-1] LOG:  processing sync request from IPC socket
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [947-1] LOG:  local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432" is requesting to release [FAILOVER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [948-1] LOG:  local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432" has released the [FAILOVER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 rsyslogd-2177: imuxsock lost 318 messages from pid 15715 due to rate-limiting
Nov 10 14:28:32 srv-2268613 pgpool[15715]: [2786-1] LOG:  find_primary_node_repeatedly: waiting for finding a primary node
Nov 10 14:28:32 srv-2268613 pgpool[15715]: [2787-1] LOG:  find_primary_node: checking backend no 0
Nov 10 14:28:32 srv-2268613 pgpool[15715]: [2788-1] LOG:  find_primary_node: checking backend no 1
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [955-1] LOG:  remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" is checking the status of [FAILOVER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [956-1] LOG:  FAILOVER lock is currently FREE
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [956-2] DETAIL:  request was from remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [963-1] LOG:  remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" is checking the status of [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [964-1] LOG:  FOLLOW MASTER lock is currently LOCKED
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [964-2] DETAIL:  request was from remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [971-1] LOG:  remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" is checking the status of [FAILOVER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [972-1] LOG:  FAILOVER lock is currently FREE
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [972-2] DETAIL:  request was from remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [979-1] LOG:  remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" is checking the status of [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [980-1] LOG:  FOLLOW MASTER lock is currently LOCKED
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [980-2] DETAIL:  request was from remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [987-1] LOG:  remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" is checking the status of [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [988-1] LOG:  FOLLOW MASTER lock is currently LOCKED
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [988-2] DETAIL:  request was from remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [995-1] LOG:  remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" is checking the status of [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [996-1] LOG:  FOLLOW MASTER lock is currently LOCKED
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [996-2] DETAIL:  request was from remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:33 srv-2268613 pgpool[15715]: [2816-1] LOG:  find_primary_node: checking backend no 0
Nov 10 14:28:33 srv-2268613 pgpool[15715]: [2817-1] LOG:  find_primary_node: checking backend no 1
Nov 10 14:28:33 srv-2268613 pgpool[15715]: [2839-1] LOG:  find_primary_node: primary node id is 1
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [998-1] LOG:  new IPC connection received
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [999-1] LOG:  processing sync request from IPC socket
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [1000-1] LOG:  local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432" is requesting to release [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [1001-1] LOG:  local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432" has released the [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:33 srv-2268613 pgpool[15715]: [2840-1] LOG:  failover: set new primary node: 1
Nov 10 14:28:33 srv-2268613 pgpool[15715]: [2841-1] LOG:  failover: set new master node: 1
Nov 10 14:28:33 srv-2268613 pgpool[16657]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16657]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16658]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16658]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16686]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16686]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16689]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16689]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16690]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16690]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16691]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16691]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16676]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16674]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16674]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16679]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16679]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16677]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16677]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16680]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16680]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16684]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16684]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16685]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16685]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16683]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16683]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16682]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16682]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16681]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16681]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16688]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16688]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16673]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16673]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16687]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16687]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16672]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16672]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16653]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16653]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16675]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16675]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16654]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16654]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16678]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16678]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16676]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16655]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16655]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16698]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16698]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16699]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16699]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16731]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16700]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16700]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16731]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16656]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16656]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16732]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16732]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16733]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16733]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16701]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16701]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16702]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16702]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16734]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16734]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16703]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16703]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16735]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16735]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16704]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16704]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16753]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16753]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16705]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16705]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16737]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16737]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16706]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16706]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16751]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16751]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16707]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16707]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16660]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16660]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16739]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16739]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16708]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16708]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16754]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16754]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16709]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16709]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16743]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16743]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16659]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16659]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16661]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16661]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16752]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16752]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16662]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16662]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16738]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16693]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16693]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16738]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16694]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16694]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16741]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16741]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16663]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16663]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16746]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16746]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16664]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16664]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16742]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16742]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16695]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16695]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16749]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16749]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16697]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16697]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16745]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16745]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16696]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16696]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16665]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16665]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16758]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16758]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16666]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16666]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16750]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16750]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16667]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16667]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16670]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16670]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16759]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16759]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16748]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16748]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16671]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16671]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16744]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16744]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16692]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16692]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16740]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16740]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16710]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16710]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16747]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16747]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16755]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16755]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16711]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16711]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16736]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16736]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16757]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16757]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16712]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16712]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16756]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16756]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16713]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16713]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16760]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16760]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16714]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16714]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16715]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16715]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16716]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16716]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16761]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16652]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16761]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16652]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16717]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16717]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16718]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16718]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16762]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16762]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16719]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16719]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16720]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16720]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16763]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16763]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16721]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16721]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16722]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16722]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16726]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16726]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16723]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16723]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16727]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16727]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16724]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16724]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16728]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16728]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16725]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16725]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16651]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16651]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16729]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16729]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16730]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16730]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16771]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16771]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [1008-1] LOG:  remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" is checking the status of [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:33 srv-2268613 rsyslogd-2177: imuxsock begins to drop messages from pid 15717 due to rate-limiting
Nov 10 14:28:33 srv-2268613 pgpool[16767]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16767]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16768]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16768]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16769]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16770]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16770]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16769]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16772]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16772]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16766]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16766]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16773]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16773]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16774]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16774]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16764]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16764]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16775]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16775]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16791]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16791]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16792]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16792]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16793]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16793]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16794]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16794]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16776]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16776]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16795]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16795]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16777]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16777]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16796]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16796]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16778]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16778]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16779]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16779]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16798]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16798]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16780]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16780]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16800]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16800]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16781]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16781]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16765]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16765]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16802]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16802]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16782]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16782]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16803]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16803]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16783]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16783]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16799]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16799]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16784]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16784]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16797]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16797]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16785]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16785]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16801]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16801]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16786]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16786]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16787]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16787]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16805]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16805]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16788]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16788]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16789]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16789]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16807]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16790]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16790]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16806]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16804]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16807]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16804]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16806]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16809]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16809]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16810]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16810]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16817]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16817]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16813]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16813]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16818]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16818]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16814]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16814]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16812]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16812]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16820]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16820]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16815]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16815]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16821]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16821]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16816]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16816]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16822]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16822]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16808]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16811]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16808]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16811]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16819]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16819]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16823]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16823]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16824]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16824]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16829]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16829]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16830]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16830]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16831]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16831]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16832]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16832]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16833]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16833]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16834]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16834]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16848]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16827]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16827]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16848]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16828]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16828]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16836]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16836]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16849]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16849]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16837]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16837]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16850]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16850]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16838]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16838]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16839]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16839]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16851]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16851]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16840]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16840]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16852]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16852]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16835]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16835]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16853]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16853]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16841]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16841]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16842]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16842]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16854]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16854]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16843]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16843]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16855]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16855]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16845]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16845]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16856]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16856]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16844]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16844]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16846]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16846]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16857]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16857]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16847]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16847]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16858]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16858]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16859]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16859]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16825]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16825]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16826]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16826]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16860]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16860]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16866]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16866]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16861]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16861]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16862]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16862]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16863]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16863]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16864]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16864]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16584]: [2390-1] LOG:  worker process received restart request
Nov 10 14:28:33 srv-2268613 pgpool[16865]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16865]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[15715]: [2842-1] LOG:  failover done. shutdown host a.db.node(15432)
Nov 10 14:28:33 srv-2268613 pgpool[16869]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16869]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16870]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16870]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16871]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16871]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16872]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16872]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16867]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16867]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16873]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16873]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16874]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16874]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16868]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16868]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16875]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16875]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16876]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16876]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16877]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16877]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16878]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16878]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16879]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16879]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16880]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16880]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16881]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16881]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16882]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16882]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16883]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16883]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16884]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16884]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16885]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16885]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16886]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16886]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16887]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16887]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16888]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16888]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16889]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16889]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16890]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16890]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:34 srv-2268613 pgpool[16583]: [2400-1] LOG:  restart request received in pcp child process
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2843-1] LOG:  PCP child 16583 exits with status 0 in failover()
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2844-1] LOG:  fork a new PCP child pid 16891 in failover()
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2847-1] LOG:  child process with pid: 16278 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2848-1] LOG:  child process with pid: 16278 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2850-1] LOG:  child process with pid: 16279 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2851-1] LOG:  child process with pid: 16279 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2853-1] LOG:  child process with pid: 16280 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2854-1] LOG:  child process with pid: 16280 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2856-1] LOG:  child process with pid: 16281 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2857-1] LOG:  child process with pid: 16281 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2859-1] LOG:  child process with pid: 16282 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2860-1] LOG:  child process with pid: 16282 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2862-1] LOG:  child process with pid: 16283 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2863-1] LOG:  child process with pid: 16283 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2865-1] LOG:  child process with pid: 16284 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2866-1] LOG:  child process with pid: 16284 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2868-1] LOG:  child process with pid: 16285 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2869-1] LOG:  child process with pid: 16285 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2871-1] LOG:  child process with pid: 16286 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2872-1] LOG:  child process with pid: 16286 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2874-1] LOG:  child process with pid: 16287 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2875-1] LOG:  child process with pid: 16287 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2877-1] LOG:  child process with pid: 16288 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2878-1] LOG:  child process with pid: 16288 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2880-1] LOG:  child process with pid: 16289 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2881-1] LOG:  child process with pid: 16289 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2883-1] LOG:  child process with pid: 16290 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2884-1] LOG:  child process with pid: 16290 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2886-1] LOG:  child process with pid: 16291 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2887-1] LOG:  child process with pid: 16291 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2889-1] LOG:  child process with pid: 16292 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2890-1] LOG:  child process with pid: 16292 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2892-1] LOG:  child process with pid: 16293 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2893-1] LOG:  child process with pid: 16293 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2895-1] LOG:  child process with pid: 16294 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2896-1] LOG:  child process with pid: 16294 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2898-1] LOG:  child process with pid: 16295 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2899-1] LOG:  child process with pid: 16295 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2901-1] LOG:  child process with pid: 16296 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2902-1] LOG:  child process with pid: 16296 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2904-1] LOG:  child process with pid: 16297 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2905-1] LOG:  child process with pid: 16297 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2907-1] LOG:  child process with pid: 16298 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2908-1] LOG:  child process with pid: 16298 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2910-1] LOG:  child process with pid: 16299 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2911-1] LOG:  child process with pid: 16299 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2913-1] LOG:  child process with pid: 16300 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2914-1] LOG:  child process with pid: 16300 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2916-1] LOG:  child process with pid: 16301 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2917-1] LOG:  child process with pid: 16301 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2919-1] LOG:  child process with pid: 16303 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2920-1] LOG:  child process with pid: 16303 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2922-1] LOG:  child process with pid: 16304 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2923-1] LOG:  child process with pid: 16304 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2925-1] LOG:  child process with pid: 16305 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2926-1] LOG:  child process with pid: 16305 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2928-1] LOG:  child process with pid: 16306 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2929-1] LOG:  child process with pid: 16306 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2931-1] LOG:  child process with pid: 16307 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2932-1] LOG:  child process with pid: 16307 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2934-1] LOG:  child process with pid: 16308 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2935-1] LOG:  child process with pid: 16308 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2937-1] LOG:  child process with pid: 16309 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2938-1] LOG:  child process with pid: 16309 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2940-1] LOG:  child process with pid: 16310 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2941-1] LOG:  child process with pid: 16310 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2943-1] LOG:  child process with pid: 16311 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2944-1] LOG:  child process with pid: 16311 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2946-1] LOG:  child process with pid: 16312 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2947-1] LOG:  child process with pid: 16312 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2949-1] LOG:  child process with pid: 16313 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2950-1] LOG:  child process with pid: 16313 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2952-1] LOG:  child process with pid: 16314 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2953-1] LOG:  child process with pid: 16314 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2955-1] LOG:  child process with pid: 16315 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2956-1] LOG:  child process with pid: 16315 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2958-1] LOG:  child process with pid: 16316 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2959-1] LOG:  child process with pid: 16316 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2961-1] LOG:  child process with pid: 16317 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2962-1] LOG:  child process with pid: 16317 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2964-1] LOG:  child process with pid: 16318 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2965-1] LOG:  child process with pid: 16318 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2967-1] LOG:  child process with pid: 16319 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2968-1] LOG:  child process with pid: 16319 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2970-1] LOG:  child process with pid: 16320 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2971-1] LOG:  child process with pid: 16320 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2973-1] LOG:  child process with pid: 16321 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2974-1] LOG:  child process with pid: 16321 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2976-1] LOG:  child process with pid: 16322 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2977-1] LOG:  child process with pid: 16322 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2979-1] LOG:  child process with pid: 16323 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2980-1] LOG:  child process with pid: 16323 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2982-1] LOG:  child process with pid: 16324 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2983-1] LOG:  child process with pid: 16324 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2985-1] LOG:  child process with pid: 16325 exits with status 0
Nov 10 14:28:34 srv-2268613 rsyslogd-2177: imuxsock begins to drop messages from pid 15715 due to rate-limiting
Nov 10 14:28:42 srv-2268613 rsyslogd-2177: imuxsock lost 19 messages from pid 15717 due to rate-limiting
^C
[root@srv-2268613 ~]# 

server_1-2 (91,125 bytes)

supp_k

2016-11-10 21:16

reporter  

server_1-3 (91,125 bytes)
Nov 10 14:28:29 srv-2268613 rsyslogd-2177: imuxsock lost 1350 messages from pid 16279 due to rate-limiting
Nov 10 14:28:31 srv-2268613 pgpool[16279]: [954655-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16279]: [954655-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16279]: [954656-1] LOG:  received degenerate backend request for node_id: 0 from pid [16279]
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [837-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16285]: [3713-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16285]: [3713-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16285]: [3714-1] LOG:  received degenerate backend request for node_id: 0 from pid [16285]
Nov 10 14:28:31 srv-2268613 pgpool[16286]: [3730-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16289]: [7659-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16281]: [4059-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16282]: [3880-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16287]: [16162-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16283]: [6200-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16280]: [3737-1] LOG:  reading and processing packets
Nov 10 14:28:31 srv-2268613 pgpool[16286]: [3730-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16289]: [7659-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16281]: [4059-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16282]: [3880-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16287]: [16162-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16283]: [6200-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16280]: [3737-2] DETAIL:  postmaster on DB node 0 was shutdown by administrative command
Nov 10 14:28:31 srv-2268613 pgpool[16286]: [3731-1] LOG:  received degenerate backend request for node_id: 0 from pid [16286]
Nov 10 14:28:31 srv-2268613 pgpool[16289]: [7660-1] LOG:  received degenerate backend request for node_id: 0 from pid [16289]
Nov 10 14:28:31 srv-2268613 pgpool[16281]: [4060-1] LOG:  received degenerate backend request for node_id: 0 from pid [16281]
Nov 10 14:28:31 srv-2268613 pgpool[16282]: [3881-1] LOG:  received degenerate backend request for node_id: 0 from pid [16282]
Nov 10 14:28:31 srv-2268613 pgpool[16287]: [16163-1] LOG:  received degenerate backend request for node_id: 0 from pid [16287]
Nov 10 14:28:31 srv-2268613 pgpool[16283]: [6201-1] LOG:  received degenerate backend request for node_id: 0 from pid [16283]
Nov 10 14:28:31 srv-2268613 pgpool[16280]: [3738-1] LOG:  received degenerate backend request for node_id: 0 from pid [16280]
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [845-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [856-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [862-1] LOG:  remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" is requesting to become a lock holder for failover ID: 37
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [863-1] LOG:  request to become a lock holder is denied to remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432"
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [863-2] DETAIL:  only master/coordinator can become a lock holder
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [870-1] LOG:  remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" is requesting to become a lock holder for failover ID: 37
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [871-1] LOG:  request to become a lock holder is denied to remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432"
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [871-2] DETAIL:  only master/coordinator can become a lock holder
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [873-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [875-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16285]: [3715-1] LOG:  degenerate backend request for 1 node(s) from pid [16285] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [876-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [878-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16286]: [3732-1] LOG:  degenerate backend request for 1 node(s) from pid [16286] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [879-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [881-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16289]: [7661-1] LOG:  degenerate backend request for 1 node(s) from pid [16289] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [882-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [884-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16281]: [4061-1] LOG:  degenerate backend request for 1 node(s) from pid [16281] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [885-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [887-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16282]: [3882-1] LOG:  degenerate backend request for 1 node(s) from pid [16282] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [888-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [890-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[16287]: [16164-1] LOG:  degenerate backend request for 1 node(s) from pid [16287] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [891-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[16283]: [6202-1] LOG:  degenerate backend request for 1 node(s) from pid [16283] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [898-1] LOG:  remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" is checking the status of [FAILOVER] lock for failover ID 37
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [899-1] LOG:  [FAILOVER] lock status check request from remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" for failover ID 37
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [899-2] DETAIL:  but failover is not yet started by master node
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [901-1] LOG:  ignoring the failover[DEGENERATE_BACKEND_REQUEST] request, because same failover request is already in progress
Nov 10 14:28:31 srv-2268613 pgpool[16280]: [3739-1] LOG:  degenerate backend request for 1 node(s) from pid [16280] is canceled  by other pgpool
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [903-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [904-1] LOG:  processing sync request from IPC socket
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [905-1] LOG:  local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432" is requesting to become a lock holder for failover ID: 37
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [906-1] LOG:  local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432" is the lock holder
Nov 10 14:28:31 srv-2268613 pgpool[15715]: [2542-1] LOG:  starting degeneration. shutdown host a.db.node(15432)
Nov 10 14:28:31 srv-2268613 pgpool[15715]: [2544-1] LOG:  Restart all children
Nov 10 14:28:31 srv-2268613 pgpool[16278]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16279]: [954658-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16280]: [3740-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16281]: [4062-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16282]: [3883-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16283]: [6203-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16284]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16285]: [3716-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16286]: [3733-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16287]: [16165-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16288]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16289]: [7662-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16290]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16291]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16292]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16294]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 rsyslogd-2177: imuxsock lost 670 messages from pid 16295 due to rate-limiting
Nov 10 14:28:31 srv-2268613 pgpool[16295]: [2132-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16293]: [1551-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 rsyslogd-2177: imuxsock lost 1258 messages from pid 16296 due to rate-limiting
Nov 10 14:28:31 srv-2268613 pgpool[16296]: [2534-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16297]: [1550-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16298]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16299]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16300]: [1550-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16301]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16303]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16304]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16305]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16306]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16307]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16308]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16309]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16310]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16311]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16312]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16313]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16314]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16315]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16316]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16317]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16318]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16319]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16320]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16321]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16322]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16323]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16324]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16325]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16326]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16327]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16328]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16329]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16330]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16331]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16332]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16333]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16334]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16335]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16336]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16337]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16338]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16339]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16340]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16341]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16342]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16343]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16344]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16345]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16346]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16347]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16348]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16349]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16350]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16351]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16352]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16353]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16354]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 rsyslogd-2177: imuxsock begins to drop messages from pid 15715 due to rate-limiting
Nov 10 14:28:31 srv-2268613 pgpool[16364]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16355]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16365]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16356]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16357]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16358]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16360]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16361]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16362]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16363]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16359]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16366]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16367]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16368]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16369]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16370]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16371]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16372]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16373]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16374]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16375]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16376]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16377]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16379]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16378]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16380]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16381]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16382]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16383]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16384]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16385]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16386]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16387]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16388]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16389]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16390]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16391]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16392]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16393]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16394]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16395]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16398]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16397]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16396]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16399]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16400]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16401]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16403]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16402]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16404]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16405]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16406]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16407]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16408]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16409]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16410]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16411]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16412]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16413]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16414]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16415]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16416]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16417]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16418]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16419]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16420]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16421]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16422]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16423]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16424]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16425]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16426]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16427]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16428]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16429]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16430]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16431]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16432]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16433]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16434]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16435]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16436]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16437]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16438]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16439]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16440]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16441]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16442]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16443]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16444]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16445]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16446]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16447]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16448]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16449]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [913-1] LOG:  remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" is checking the status of [FAILOVER] lock for failover ID 37
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [914-1] LOG:  FAILOVER lock is currently LOCKED
Nov 10 14:28:31 srv-2268613 pgpool[16450]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[15717]: [914-2] DETAIL:  request was from remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:31 srv-2268613 pgpool[16451]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16452]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16453]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16454]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16455]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16456]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16457]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16458]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16459]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16460]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16461]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16462]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16463]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16464]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16465]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16466]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16467]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16468]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16469]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16470]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16471]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16472]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16473]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16474]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16475]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16476]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16477]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16478]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16479]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16480]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16481]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16482]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16483]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16484]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16485]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16486]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16487]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16488]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16489]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16490]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16491]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16492]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16493]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16494]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16495]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16496]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16497]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16498]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16499]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16500]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16501]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16502]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16503]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16504]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16505]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16506]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16507]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16508]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16509]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16510]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16511]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16512]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16513]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16514]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16515]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16516]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 pgpool[16517]: [1549-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268613 root: Failover event. Parameters:[0 a.db.node 15432 /var/lib/pgsql/9.5/data 1 b.db.node 0 0 15432 /var/lib/pgsql/9.5/data]
Nov 10 14:28:31 srv-2268613 root: Make sure the master 'b.db.node' is async
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [921-1] LOG:  remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" is checking the status of [FAILOVER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [922-1] LOG:  FAILOVER lock is currently LOCKED
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [922-2] DETAIL:  request was from remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [945-1] LOG:  new IPC connection received
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [946-1] LOG:  processing sync request from IPC socket
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [947-1] LOG:  local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432" is requesting to release [FAILOVER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [948-1] LOG:  local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432" has released the [FAILOVER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 rsyslogd-2177: imuxsock lost 318 messages from pid 15715 due to rate-limiting
Nov 10 14:28:32 srv-2268613 pgpool[15715]: [2786-1] LOG:  find_primary_node_repeatedly: waiting for finding a primary node
Nov 10 14:28:32 srv-2268613 pgpool[15715]: [2787-1] LOG:  find_primary_node: checking backend no 0
Nov 10 14:28:32 srv-2268613 pgpool[15715]: [2788-1] LOG:  find_primary_node: checking backend no 1
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [955-1] LOG:  remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" is checking the status of [FAILOVER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [956-1] LOG:  FAILOVER lock is currently FREE
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [956-2] DETAIL:  request was from remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [963-1] LOG:  remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" is checking the status of [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [964-1] LOG:  FOLLOW MASTER lock is currently LOCKED
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [964-2] DETAIL:  request was from remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [971-1] LOG:  remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" is checking the status of [FAILOVER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [972-1] LOG:  FAILOVER lock is currently FREE
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [972-2] DETAIL:  request was from remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [979-1] LOG:  remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" is checking the status of [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [980-1] LOG:  FOLLOW MASTER lock is currently LOCKED
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [980-2] DETAIL:  request was from remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [987-1] LOG:  remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" is checking the status of [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [988-1] LOG:  FOLLOW MASTER lock is currently LOCKED
Nov 10 14:28:32 srv-2268613 pgpool[15717]: [988-2] DETAIL:  request was from remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [995-1] LOG:  remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" is checking the status of [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [996-1] LOG:  FOLLOW MASTER lock is currently LOCKED
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [996-2] DETAIL:  request was from remote pgpool-II node "Linux_srv-2268665.aqa.int.zone_5432" and lock holder is local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:33 srv-2268613 pgpool[15715]: [2816-1] LOG:  find_primary_node: checking backend no 0
Nov 10 14:28:33 srv-2268613 pgpool[15715]: [2817-1] LOG:  find_primary_node: checking backend no 1
Nov 10 14:28:33 srv-2268613 pgpool[15715]: [2839-1] LOG:  find_primary_node: primary node id is 1
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [998-1] LOG:  new IPC connection received
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [999-1] LOG:  processing sync request from IPC socket
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [1000-1] LOG:  local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432" is requesting to release [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [1001-1] LOG:  local pgpool-II node "Linux_srv-2268613.aqa.int.zone_5432" has released the [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:33 srv-2268613 pgpool[15715]: [2840-1] LOG:  failover: set new primary node: 1
Nov 10 14:28:33 srv-2268613 pgpool[15715]: [2841-1] LOG:  failover: set new master node: 1
Nov 10 14:28:33 srv-2268613 pgpool[16657]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16657]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16658]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16658]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16686]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16686]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16689]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16689]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16690]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16690]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16691]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16691]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16676]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16674]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16674]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16679]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16679]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16677]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16677]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16680]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16680]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16684]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16684]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16685]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16685]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16683]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16683]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16682]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16682]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16681]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16681]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16688]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16688]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16673]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16673]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16687]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16687]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16672]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16672]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16653]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16653]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16675]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16675]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16654]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16654]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16678]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16678]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16676]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16655]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16655]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16698]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16698]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16699]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16699]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16731]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16700]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16700]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16731]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16656]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16656]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16732]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16732]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16733]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16733]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16701]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16701]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16702]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16702]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16734]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16734]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16703]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16703]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16735]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16735]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16704]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16704]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16753]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16753]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16705]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16705]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16737]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16737]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16706]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16706]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16751]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16751]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16707]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16707]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16660]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16660]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16739]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16739]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16708]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16708]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16754]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16754]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16709]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16709]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16743]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16743]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16659]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16659]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16661]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16661]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16752]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16752]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16662]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16662]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16738]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16693]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16693]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16738]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16694]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16694]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16741]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16741]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16663]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16663]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16746]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16746]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16664]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16664]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16742]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16742]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16695]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16695]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16749]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16749]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16697]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16697]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16745]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16745]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16696]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16696]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16665]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16665]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16758]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16758]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16666]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16666]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16750]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16750]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16667]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16667]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16670]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16670]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16759]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16759]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16748]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16748]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16671]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16671]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16744]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16744]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16692]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16692]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16740]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16740]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16710]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16710]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16747]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16747]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16755]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16755]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16711]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16711]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16736]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16736]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16757]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16757]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16712]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16712]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16756]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16756]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16713]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16713]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16760]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16760]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16714]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16714]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16715]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16715]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16716]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16716]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16761]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16652]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16761]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16652]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16717]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16717]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16718]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16718]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16762]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16762]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16719]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16719]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16720]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16720]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16763]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16763]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16721]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16721]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16722]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16722]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16726]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16726]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16723]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16723]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16727]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16727]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16724]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16724]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16728]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16728]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16725]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16725]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16651]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16651]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16729]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16729]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16730]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16730]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16771]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16771]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[15717]: [1008-1] LOG:  remote pgpool-II node "Linux_POAMN-2268531.aqa.int.zone_5432" is checking the status of [FOLLOW MASTER] lock for failover ID 37
Nov 10 14:28:33 srv-2268613 rsyslogd-2177: imuxsock begins to drop messages from pid 15717 due to rate-limiting
Nov 10 14:28:33 srv-2268613 pgpool[16767]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16767]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16768]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16768]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16769]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16770]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16770]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16769]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16772]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16772]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16766]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16766]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16773]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16773]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16774]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16774]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16764]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16764]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16775]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16775]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16791]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16791]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16792]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16792]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16793]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16793]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16794]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16794]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16776]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16776]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16795]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16795]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16777]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16777]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16796]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16796]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16778]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16778]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16779]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16779]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16798]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16798]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16780]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16780]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16800]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16800]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16781]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16781]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16765]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16765]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16802]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16802]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16782]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16782]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16803]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16803]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16783]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16783]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16799]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16799]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16784]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16784]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16797]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16797]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16785]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16785]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16801]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16801]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16786]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16786]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16787]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16787]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16805]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16805]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16788]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16788]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16789]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16789]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16807]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16790]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16790]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16806]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16804]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16807]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16804]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16806]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16809]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16809]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16810]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16810]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16817]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16817]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16813]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16813]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16818]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16818]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16814]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16814]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16812]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16812]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16820]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16820]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16815]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16815]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16821]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16821]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16816]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16816]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16822]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16822]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16808]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16811]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16808]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16811]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16819]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16819]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16823]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16823]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16824]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16824]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16829]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16829]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16830]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16830]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16831]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16831]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16832]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16832]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16833]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16833]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16834]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16834]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16848]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16827]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16827]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16848]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16828]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16828]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16836]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16836]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16849]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16849]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16837]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16837]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16850]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16850]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16838]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16838]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16839]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16839]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16851]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16851]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16840]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16840]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16852]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16852]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16835]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16835]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16853]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16853]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16841]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16841]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16842]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16842]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16854]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16854]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16843]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16843]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16855]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16855]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16845]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16845]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16856]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16856]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16844]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16844]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16846]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16846]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16857]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16857]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16847]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16847]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16858]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16858]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16859]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16859]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16825]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16825]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16826]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16826]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16860]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16860]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16866]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16866]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16861]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16861]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16862]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16862]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16863]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16863]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16864]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16864]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16584]: [2390-1] LOG:  worker process received restart request
Nov 10 14:28:33 srv-2268613 pgpool[16865]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16865]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[15715]: [2842-1] LOG:  failover done. shutdown host a.db.node(15432)
Nov 10 14:28:33 srv-2268613 pgpool[16869]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16869]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16870]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16870]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16871]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16871]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16872]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16872]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16867]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16867]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16873]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16873]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16874]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16874]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16868]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16868]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16875]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16875]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16876]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16876]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16877]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16877]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16878]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16878]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16879]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16879]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16880]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16880]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16881]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16881]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16882]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16882]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16883]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16883]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16884]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16884]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16885]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16885]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16886]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16886]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16887]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16887]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16888]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16888]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16889]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16889]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:33 srv-2268613 pgpool[16890]: [2844-1] LOG:  failback event detected
Nov 10 14:28:33 srv-2268613 pgpool[16890]: [2844-2] DETAIL:  restarting myself
Nov 10 14:28:34 srv-2268613 pgpool[16583]: [2400-1] LOG:  restart request received in pcp child process
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2843-1] LOG:  PCP child 16583 exits with status 0 in failover()
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2844-1] LOG:  fork a new PCP child pid 16891 in failover()
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2847-1] LOG:  child process with pid: 16278 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2848-1] LOG:  child process with pid: 16278 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2850-1] LOG:  child process with pid: 16279 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2851-1] LOG:  child process with pid: 16279 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2853-1] LOG:  child process with pid: 16280 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2854-1] LOG:  child process with pid: 16280 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2856-1] LOG:  child process with pid: 16281 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2857-1] LOG:  child process with pid: 16281 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2859-1] LOG:  child process with pid: 16282 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2860-1] LOG:  child process with pid: 16282 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2862-1] LOG:  child process with pid: 16283 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2863-1] LOG:  child process with pid: 16283 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2865-1] LOG:  child process with pid: 16284 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2866-1] LOG:  child process with pid: 16284 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2868-1] LOG:  child process with pid: 16285 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2869-1] LOG:  child process with pid: 16285 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2871-1] LOG:  child process with pid: 16286 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2872-1] LOG:  child process with pid: 16286 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2874-1] LOG:  child process with pid: 16287 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2875-1] LOG:  child process with pid: 16287 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2877-1] LOG:  child process with pid: 16288 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2878-1] LOG:  child process with pid: 16288 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2880-1] LOG:  child process with pid: 16289 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2881-1] LOG:  child process with pid: 16289 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2883-1] LOG:  child process with pid: 16290 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2884-1] LOG:  child process with pid: 16290 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2886-1] LOG:  child process with pid: 16291 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2887-1] LOG:  child process with pid: 16291 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2889-1] LOG:  child process with pid: 16292 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2890-1] LOG:  child process with pid: 16292 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2892-1] LOG:  child process with pid: 16293 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2893-1] LOG:  child process with pid: 16293 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2895-1] LOG:  child process with pid: 16294 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2896-1] LOG:  child process with pid: 16294 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2898-1] LOG:  child process with pid: 16295 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2899-1] LOG:  child process with pid: 16295 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2901-1] LOG:  child process with pid: 16296 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2902-1] LOG:  child process with pid: 16296 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2904-1] LOG:  child process with pid: 16297 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2905-1] LOG:  child process with pid: 16297 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2907-1] LOG:  child process with pid: 16298 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2908-1] LOG:  child process with pid: 16298 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2910-1] LOG:  child process with pid: 16299 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2911-1] LOG:  child process with pid: 16299 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2913-1] LOG:  child process with pid: 16300 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2914-1] LOG:  child process with pid: 16300 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2916-1] LOG:  child process with pid: 16301 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2917-1] LOG:  child process with pid: 16301 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2919-1] LOG:  child process with pid: 16303 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2920-1] LOG:  child process with pid: 16303 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2922-1] LOG:  child process with pid: 16304 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2923-1] LOG:  child process with pid: 16304 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2925-1] LOG:  child process with pid: 16305 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2926-1] LOG:  child process with pid: 16305 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2928-1] LOG:  child process with pid: 16306 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2929-1] LOG:  child process with pid: 16306 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2931-1] LOG:  child process with pid: 16307 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2932-1] LOG:  child process with pid: 16307 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2934-1] LOG:  child process with pid: 16308 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2935-1] LOG:  child process with pid: 16308 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2937-1] LOG:  child process with pid: 16309 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2938-1] LOG:  child process with pid: 16309 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2940-1] LOG:  child process with pid: 16310 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2941-1] LOG:  child process with pid: 16310 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2943-1] LOG:  child process with pid: 16311 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2944-1] LOG:  child process with pid: 16311 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2946-1] LOG:  child process with pid: 16312 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2947-1] LOG:  child process with pid: 16312 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2949-1] LOG:  child process with pid: 16313 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2950-1] LOG:  child process with pid: 16313 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2952-1] LOG:  child process with pid: 16314 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2953-1] LOG:  child process with pid: 16314 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2955-1] LOG:  child process with pid: 16315 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2956-1] LOG:  child process with pid: 16315 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2958-1] LOG:  child process with pid: 16316 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2959-1] LOG:  child process with pid: 16316 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2961-1] LOG:  child process with pid: 16317 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2962-1] LOG:  child process with pid: 16317 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2964-1] LOG:  child process with pid: 16318 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2965-1] LOG:  child process with pid: 16318 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2967-1] LOG:  child process with pid: 16319 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2968-1] LOG:  child process with pid: 16319 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2970-1] LOG:  child process with pid: 16320 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2971-1] LOG:  child process with pid: 16320 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2973-1] LOG:  child process with pid: 16321 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2974-1] LOG:  child process with pid: 16321 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2976-1] LOG:  child process with pid: 16322 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2977-1] LOG:  child process with pid: 16322 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2979-1] LOG:  child process with pid: 16323 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2980-1] LOG:  child process with pid: 16323 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2982-1] LOG:  child process with pid: 16324 exits with status 0
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2983-1] LOG:  child process with pid: 16324 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268613 pgpool[15715]: [2985-1] LOG:  child process with pid: 16325 exits with status 0
Nov 10 14:28:34 srv-2268613 rsyslogd-2177: imuxsock begins to drop messages from pid 15715 due to rate-limiting
Nov 10 14:28:42 srv-2268613 rsyslogd-2177: imuxsock lost 19 messages from pid 15717 due to rate-limiting
^C
[root@srv-2268613 ~]# 

server_1-3 (91,125 bytes)

supp_k

2016-11-10 21:17

reporter  

server_2 (51,386 bytes)

Nov 10 14:28:31 srv-2268665 pgpool[3587]: [24-1] LOG:  received degenerate backend request for node_id: 0 from pid [3587]
Nov 10 14:28:31 srv-2268665 pgpool[3587]: [25-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268665 pgpool[3587]: [26-1] LOG:  processing sync request from IPC socket
Nov 10 14:28:31 srv-2268665 pgpool[3587]: [27-1] LOG:  sync request from IPC socket is forwarded to master watchdog node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:31 srv-2268665 pgpool[3587]: [27-2] DETAIL:  waiting for the reply from master node...
Nov 10 14:28:31 srv-2268665 pgpool[3585]: [10-1] LOG:  starting degeneration. shutdown host a.db.node(15432)
Nov 10 14:28:31 srv-2268665 pgpool[3585]: [11-1] LOG:  Restart all children
Nov 10 14:28:31 srv-2268665 pgpool[3807]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3590]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3587]: [28-1] LOG:  new IPC connection received
Nov 10 14:28:31 srv-2268665 pgpool[3587]: [29-1] LOG:  processing sync request from IPC socket
Nov 10 14:28:31 srv-2268665 pgpool[3587]: [30-1] LOG:  sync request from IPC socket is forwarded to master watchdog node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:31 srv-2268665 pgpool[3587]: [30-2] DETAIL:  waiting for the reply from master node...
Nov 10 14:28:31 srv-2268665 pgpool[3592]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3806]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3594]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3805]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3596]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3804]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3597]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3803]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3598]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3802]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3599]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3801]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3600]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3800]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3601]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3799]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3602]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3798]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3603]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3797]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3604]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3796]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3605]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3606]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3795]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3607]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3653]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3611]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3654]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3609]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3655]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3614]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3656]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3610]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3657]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3612]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3658]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3615]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3666]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3620]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3667]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3621]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3668]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3622]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3669]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3619]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3670]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3617]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3671]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3618]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3677]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3624]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3678]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3629]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3628]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3679]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3680]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3630]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3626]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3681]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3682]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3631]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3685]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3633]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3686]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3632]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3687]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3778]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3688]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3637]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3689]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3635]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3690]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3780]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3695]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3643]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3640]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3696]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3777]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3697]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3641]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3698]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3644]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3639]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3699]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3642]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3707]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3650]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3651]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3708]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3646]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3709]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3648]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3710]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3649]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3711]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3647]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3828]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3719]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3660]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3661]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3720]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3662]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3721]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3665]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3722]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3664]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3663]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3723]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3673]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3724]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3676]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3725]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3675]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3726]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3672]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3674]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3727]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3684]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3728]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3683]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3729]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3691]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3730]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3693]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3731]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3692]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3636]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3732]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3694]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3733]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3779]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3734]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3638]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3735]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3781]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3634]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3736]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3700]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3737]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3703]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3738]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3702]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3739]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3704]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3740]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3705]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3741]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3706]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3645]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3742]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3717]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3743]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3829]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3652]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3744]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3659]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3745]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3716]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3746]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3713]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3747]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3701]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3748]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3831]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3749]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3718]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3750]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3714]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3715]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3751]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3775]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3752]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3712]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3753]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3830]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3832]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3754]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3776]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3755]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3827]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3756]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3782]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3757]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3786]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3758]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3759]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3826]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3760]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3825]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3761]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3823]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3762]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3822]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3763]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3772]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3818]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3819]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3773]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3817]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3820]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3774]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3821]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3808]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3627]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3809]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3625]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3764]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3788]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3765]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3783]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3766]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3792]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3784]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3790]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3793]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3789]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3787]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3794]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3767]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3771]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3768]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3811]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3769]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3813]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3770]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3814]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3810]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3785]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3812]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3791]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3815]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3613]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3816]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3616]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3608]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3623]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:31 srv-2268665 pgpool[3824]: [8-1] LOG:  child process received shutdown request signal 3
Nov 10 14:28:32 srv-2268665 pgpool[3587]: [31-1] LOG:  new IPC connection received
Nov 10 14:28:32 srv-2268665 pgpool[3587]: [32-1] LOG:  processing sync request from IPC socket
Nov 10 14:28:32 srv-2268665 pgpool[3587]: [33-1] LOG:  sync request from IPC socket is forwarded to master watchdog node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:32 srv-2268665 pgpool[3587]: [33-2] DETAIL:  waiting for the reply from master node...
Nov 10 14:28:32 srv-2268665 pgpool[3587]: [34-1] LOG:  new IPC connection received
Nov 10 14:28:32 srv-2268665 pgpool[3587]: [35-1] LOG:  processing sync request from IPC socket
Nov 10 14:28:32 srv-2268665 pgpool[3587]: [36-1] LOG:  sync request from IPC socket is forwarded to master watchdog node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:32 srv-2268665 pgpool[3587]: [36-2] DETAIL:  waiting for the reply from master node...
Nov 10 14:28:32 srv-2268665 pgpool[3585]: [12-1] LOG:  find_primary_node_repeatedly: waiting for finding a primary node
Nov 10 14:28:32 srv-2268665 pgpool[3585]: [13-1] LOG:  find_primary_node: checking backend no 0
Nov 10 14:28:32 srv-2268665 pgpool[3585]: [14-1] LOG:  find_primary_node: checking backend no 1
Nov 10 14:28:32 srv-2268665 pgpool[3585]: [15-1] LOG:  find_primary_node: primary node id is 1
Nov 10 14:28:32 srv-2268665 pgpool[3587]: [37-1] LOG:  new IPC connection received
Nov 10 14:28:32 srv-2268665 pgpool[3587]: [38-1] LOG:  processing sync request from IPC socket
Nov 10 14:28:32 srv-2268665 pgpool[3587]: [39-1] LOG:  sync request from IPC socket is forwarded to master watchdog node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:32 srv-2268665 pgpool[3587]: [39-2] DETAIL:  waiting for the reply from master node...
Nov 10 14:28:33 srv-2268665 pgpool[3587]: [40-1] LOG:  new IPC connection received
Nov 10 14:28:33 srv-2268665 pgpool[3587]: [41-1] LOG:  processing sync request from IPC socket
Nov 10 14:28:33 srv-2268665 pgpool[3587]: [42-1] LOG:  sync request from IPC socket is forwarded to master watchdog node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:33 srv-2268665 pgpool[3587]: [42-2] DETAIL:  waiting for the reply from master node...
Nov 10 14:28:33 srv-2268665 pgpool[3587]: [43-1] LOG:  new IPC connection received
Nov 10 14:28:33 srv-2268665 pgpool[3587]: [44-1] LOG:  processing sync request from IPC socket
Nov 10 14:28:33 srv-2268665 pgpool[3587]: [45-1] LOG:  sync request from IPC socket is forwarded to master watchdog node "Linux_srv-2268613.aqa.int.zone_5432"
Nov 10 14:28:33 srv-2268665 pgpool[3587]: [45-2] DETAIL:  waiting for the reply from master node...
Nov 10 14:28:33 srv-2268665 pgpool[3585]: [16-1] LOG:  failover: set new primary node: 1
Nov 10 14:28:33 srv-2268665 pgpool[3585]: [17-1] LOG:  failover: set new master node: 1
Nov 10 14:28:33 srv-2268665 pgpool[3585]: [18-1] LOG:  failover done. shutdown host a.db.node(15432)
Nov 10 14:28:33 srv-2268665 pgpool[3834]: [8-1] LOG:  worker process received restart request
Nov 10 14:28:34 srv-2268665 pgpool[3833]: [8-1] LOG:  restart request received in pcp child process
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [19-1] LOG:  PCP child 3833 exits with status 0 in failover()
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [20-1] LOG:  fork a new PCP child pid 4161 in failover()
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [21-1] LOG:  child process with pid: 3590 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [22-1] LOG:  child process with pid: 3590 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [23-1] LOG:  child process with pid: 3592 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [24-1] LOG:  child process with pid: 3592 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [25-1] LOG:  child process with pid: 3594 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [26-1] LOG:  child process with pid: 3594 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [27-1] LOG:  child process with pid: 3596 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [28-1] LOG:  child process with pid: 3596 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [29-1] LOG:  child process with pid: 3597 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [30-1] LOG:  child process with pid: 3597 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [31-1] LOG:  child process with pid: 3598 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [32-1] LOG:  child process with pid: 3598 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [33-1] LOG:  child process with pid: 3599 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [34-1] LOG:  child process with pid: 3599 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [35-1] LOG:  child process with pid: 3600 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [36-1] LOG:  child process with pid: 3600 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [37-1] LOG:  child process with pid: 3601 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [38-1] LOG:  child process with pid: 3601 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [39-1] LOG:  child process with pid: 3602 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [40-1] LOG:  child process with pid: 3602 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [41-1] LOG:  child process with pid: 3603 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [42-1] LOG:  child process with pid: 3603 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [43-1] LOG:  child process with pid: 3604 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [44-1] LOG:  child process with pid: 3604 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [45-1] LOG:  child process with pid: 3605 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [46-1] LOG:  child process with pid: 3605 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [47-1] LOG:  child process with pid: 3606 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [48-1] LOG:  child process with pid: 3606 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [49-1] LOG:  child process with pid: 3607 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [50-1] LOG:  child process with pid: 3607 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [51-1] LOG:  child process with pid: 3608 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [52-1] LOG:  child process with pid: 3608 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [53-1] LOG:  child process with pid: 3609 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [54-1] LOG:  child process with pid: 3609 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [55-1] LOG:  child process with pid: 3610 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [56-1] LOG:  child process with pid: 3610 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [57-1] LOG:  child process with pid: 3611 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [58-1] LOG:  child process with pid: 3611 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [59-1] LOG:  child process with pid: 3612 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [60-1] LOG:  child process with pid: 3612 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [61-1] LOG:  child process with pid: 3613 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [62-1] LOG:  child process with pid: 3613 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [63-1] LOG:  child process with pid: 3614 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [64-1] LOG:  child process with pid: 3614 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [65-1] LOG:  child process with pid: 3615 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [66-1] LOG:  child process with pid: 3615 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [67-1] LOG:  child process with pid: 3616 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [68-1] LOG:  child process with pid: 3616 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [69-1] LOG:  child process with pid: 3617 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [70-1] LOG:  child process with pid: 3617 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [71-1] LOG:  child process with pid: 3618 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [72-1] LOG:  child process with pid: 3618 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [73-1] LOG:  child process with pid: 3619 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [74-1] LOG:  child process with pid: 3619 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [75-1] LOG:  child process with pid: 3620 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [76-1] LOG:  child process with pid: 3620 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [77-1] LOG:  child process with pid: 3621 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [78-1] LOG:  child process with pid: 3621 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [79-1] LOG:  child process with pid: 3622 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [80-1] LOG:  child process with pid: 3622 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [81-1] LOG:  child process with pid: 3623 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [82-1] LOG:  child process with pid: 3623 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [83-1] LOG:  child process with pid: 3624 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [84-1] LOG:  child process with pid: 3624 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [85-1] LOG:  child process with pid: 3625 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [86-1] LOG:  child process with pid: 3625 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [87-1] LOG:  child process with pid: 3626 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [88-1] LOG:  child process with pid: 3626 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [89-1] LOG:  child process with pid: 3627 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [90-1] LOG:  child process with pid: 3627 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [91-1] LOG:  child process with pid: 3628 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [92-1] LOG:  child process with pid: 3628 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [93-1] LOG:  child process with pid: 3629 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [94-1] LOG:  child process with pid: 3629 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [95-1] LOG:  child process with pid: 3630 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [96-1] LOG:  child process with pid: 3630 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [97-1] LOG:  child process with pid: 3631 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [98-1] LOG:  child process with pid: 3631 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [99-1] LOG:  child process with pid: 3632 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [100-1] LOG:  child process with pid: 3632 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [101-1] LOG:  child process with pid: 3633 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [102-1] LOG:  child process with pid: 3633 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [103-1] LOG:  child process with pid: 3634 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [104-1] LOG:  child process with pid: 3634 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [105-1] LOG:  child process with pid: 3635 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [106-1] LOG:  child process with pid: 3635 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [107-1] LOG:  child process with pid: 3636 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [108-1] LOG:  child process with pid: 3636 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [109-1] LOG:  child process with pid: 3637 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [110-1] LOG:  child process with pid: 3637 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [111-1] LOG:  child process with pid: 3638 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [112-1] LOG:  child process with pid: 3638 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [113-1] LOG:  child process with pid: 3639 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [114-1] LOG:  child process with pid: 3639 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [115-1] LOG:  child process with pid: 3640 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [116-1] LOG:  child process with pid: 3640 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [117-1] LOG:  child process with pid: 3641 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [118-1] LOG:  child process with pid: 3641 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [119-1] LOG:  child process with pid: 3642 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [120-1] LOG:  child process with pid: 3642 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [121-1] LOG:  child process with pid: 3643 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [122-1] LOG:  child process with pid: 3643 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [123-1] LOG:  child process with pid: 3644 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [124-1] LOG:  child process with pid: 3644 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [125-1] LOG:  child process with pid: 3645 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [126-1] LOG:  child process with pid: 3645 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [127-1] LOG:  child process with pid: 3646 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [128-1] LOG:  child process with pid: 3646 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [129-1] LOG:  child process with pid: 3647 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [130-1] LOG:  child process with pid: 3647 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [131-1] LOG:  child process with pid: 3648 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [132-1] LOG:  child process with pid: 3648 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [133-1] LOG:  child process with pid: 3649 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [134-1] LOG:  child process with pid: 3649 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [135-1] LOG:  child process with pid: 3650 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [136-1] LOG:  child process with pid: 3650 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [137-1] LOG:  child process with pid: 3651 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [138-1] LOG:  child process with pid: 3651 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [139-1] LOG:  child process with pid: 3652 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [140-1] LOG:  child process with pid: 3652 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [141-1] LOG:  child process with pid: 3653 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [142-1] LOG:  child process with pid: 3653 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [143-1] LOG:  child process with pid: 3654 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [144-1] LOG:  child process with pid: 3654 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [145-1] LOG:  child process with pid: 3655 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [146-1] LOG:  child process with pid: 3655 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [147-1] LOG:  child process with pid: 3656 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [148-1] LOG:  child process with pid: 3656 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [149-1] LOG:  child process with pid: 3657 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [150-1] LOG:  child process with pid: 3657 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [151-1] LOG:  child process with pid: 3658 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [152-1] LOG:  child process with pid: 3658 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [153-1] LOG:  child process with pid: 3659 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [154-1] LOG:  child process with pid: 3659 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [155-1] LOG:  child process with pid: 3660 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [156-1] LOG:  child process with pid: 3660 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [157-1] LOG:  child process with pid: 3661 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [158-1] LOG:  child process with pid: 3661 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [159-1] LOG:  child process with pid: 3662 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [160-1] LOG:  child process with pid: 3662 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [161-1] LOG:  child process with pid: 3663 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [162-1] LOG:  child process with pid: 3663 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [163-1] LOG:  child process with pid: 3664 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [164-1] LOG:  child process with pid: 3664 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [165-1] LOG:  child process with pid: 3665 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [166-1] LOG:  child process with pid: 3665 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [167-1] LOG:  child process with pid: 3666 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [168-1] LOG:  child process with pid: 3666 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [169-1] LOG:  child process with pid: 3667 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [170-1] LOG:  child process with pid: 3667 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [171-1] LOG:  child process with pid: 3668 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [172-1] LOG:  child process with pid: 3668 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [173-1] LOG:  child process with pid: 3669 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [174-1] LOG:  child process with pid: 3669 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [175-1] LOG:  child process with pid: 3670 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [176-1] LOG:  child process with pid: 3670 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [177-1] LOG:  child process with pid: 3671 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [178-1] LOG:  child process with pid: 3671 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [179-1] LOG:  child process with pid: 3672 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [180-1] LOG:  child process with pid: 3672 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [181-1] LOG:  child process with pid: 3673 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [182-1] LOG:  child process with pid: 3673 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [183-1] LOG:  child process with pid: 3674 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [184-1] LOG:  child process with pid: 3674 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [185-1] LOG:  child process with pid: 3675 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [186-1] LOG:  child process with pid: 3675 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [187-1] LOG:  child process with pid: 3676 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [188-1] LOG:  child process with pid: 3676 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [189-1] LOG:  child process with pid: 3677 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [190-1] LOG:  child process with pid: 3677 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [191-1] LOG:  child process with pid: 3678 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [192-1] LOG:  child process with pid: 3678 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [193-1] LOG:  child process with pid: 3679 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [194-1] LOG:  child process with pid: 3679 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [195-1] LOG:  child process with pid: 3680 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [196-1] LOG:  child process with pid: 3680 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [197-1] LOG:  child process with pid: 3681 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [198-1] LOG:  child process with pid: 3681 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [199-1] LOG:  child process with pid: 3682 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [200-1] LOG:  child process with pid: 3682 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [201-1] LOG:  child process with pid: 3683 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [202-1] LOG:  child process with pid: 3683 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [203-1] LOG:  child process with pid: 3684 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [204-1] LOG:  child process with pid: 3684 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [205-1] LOG:  child process with pid: 3685 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [206-1] LOG:  child process with pid: 3685 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [207-1] LOG:  child process with pid: 3686 exits with status 0
Nov 10 14:28:34 srv-2268665 pgpool[3585]: [208-1] LOG:  child process with pid: 3686 exited with success and will not be restarted
Nov 10 14:28:34 srv-2268665 rsyslogd-2177: imuxsock begins to drop messages from pid 3585 due to rate-limiting
server_2 (51,386 bytes)

Muhammad Usama

2016-11-10 22:47

developer   ~0001163

Hi

I have looked at the attached log and everything looks just fine, Also tested the said scenario and it is working fine at my end.
What behavior you are experiencing after the failover? Do the new connections get stuck or if the connections are successful, and query results are not coming back?
This is also strange if you are experiencing some issue with the client connections after this patch since this patch does not touch the area of client connection and query handling.

Thanks
Regards

supp_k

2016-11-10 22:57

reporter   ~0001164

Hi,

maybe these problems are issues of my building environment. Ok, you state that the fix of the problem 0000227 is available on your side I believe.

Now can you share the patched 3.5.X version or commit it into the corresponding branch?

Muhammad Usama

2016-11-11 00:12

developer   ~0001165

Thanks for the verification. I will commit it to the master branch after some more testing and will discuss about the back porting of fix with the pgpool-II development group and update you accordingly.

Best regards!

supp_k

2016-11-11 17:11

reporter   ~0001167

Do you have any estimations when it will be possible to obtain the patched build? We need it since it impacts out environment.

Muhammad Usama

2016-11-15 06:42

developer   ~0001170

I have checked-in the fix to the master branch, and we have decided to port it to the 3.5 branch as well. I will try to push it to the 3.5 branch by the end of the week.

supp_k

2016-11-23 17:55

reporter   ~0001187

Our tests revealed no problems with new failovering mechanizm.
I think it is OK now.

Issue History

Date Modified Username Field Change
2016-07-30 09:19 supp_k New Issue
2016-07-30 09:19 supp_k File Added: pgpool.conf
2016-08-01 21:25 guptesh.cg4@gmail.com Tag Attached: master slave
2016-08-01 21:25 guptesh.cg4@gmail.com Tag Attached: streaming replication
2016-08-01 23:42 guptesh.cg4@gmail.com Note Added: 0000952
2016-08-02 00:09 supp_k Note Added: 0000954
2016-08-02 10:28 t-ishii Assigned To => Muhammad Usama
2016-08-02 10:28 t-ishii Status new => assigned
2016-08-02 13:43 t-ishii Tag Attached: watchdog
2016-08-03 05:23 Muhammad Usama Note Added: 0000959
2016-08-03 20:57 supp_k Note Added: 0000962
2016-08-05 19:32 z0rb1n0 File Added: pgpool_no_failover_fabio.tar.bz2
2016-08-05 19:32 z0rb1n0 Note Added: 0000968
2016-08-05 21:44 z0rb1n0 Note Added: 0000969
2016-08-10 16:09 guptesh.cg4@gmail.com Note Added: 0000977
2016-08-19 04:03 cohavisi Note Added: 0001015
2016-08-23 20:21 z0rb1n0 Note Added: 0001020
2016-08-24 23:20 tscheuren Note Added: 0001021
2016-08-25 23:40 Muhammad Usama Note Added: 0001024
2016-09-21 00:53 gabrimonfa Note Added: 0001073
2016-09-27 21:39 supp_k Note Added: 0001088
2016-09-27 22:09 supp_k Note Added: 0001089
2016-09-27 23:23 Muhammad Usama File Added: failover_standby_fix.diff
2016-09-27 23:30 Muhammad Usama Status assigned => feedback
2016-09-27 23:30 Muhammad Usama Note Added: 0001090
2016-09-28 00:46 supp_k Note Added: 0001092
2016-09-28 00:46 supp_k Status feedback => assigned
2016-09-28 01:15 supp_k Note Added: 0001093
2016-09-28 09:44 t-ishii Note Added: 0001096
2016-09-28 15:49 Dang Minh Huong Note Added: 0001104
2016-09-28 16:09 supp_k Note Added: 0001105
2016-09-28 17:15 Muhammad Usama Note Added: 0001106
2016-09-28 22:01 supp_k File Added: data.tar.gz
2016-09-28 22:01 supp_k Note Added: 0001107
2016-09-28 22:37 Muhammad Usama File Added: failover_standby_fix_v2.diff
2016-09-28 22:37 Muhammad Usama Note Added: 0001108
2016-09-30 00:11 supp_k Note Added: 0001111
2016-09-30 01:24 z0rb1n0 Note Added: 0001112
2016-09-30 16:23 supp_k Note Added: 0001114
2016-10-05 20:26 supp_k Note Added: 0001116
2016-10-10 20:00 gabrimonfa Note Added: 0001117
2016-10-13 18:07 supp_k Note Added: 0001118
2016-11-01 00:33 Muhammad Usama File Added: wd_rewamp_failover.diff
2016-11-01 00:43 Muhammad Usama Status assigned => feedback
2016-11-01 00:43 Muhammad Usama Note Added: 0001143
2016-11-01 20:01 supp_k Note Added: 0001145
2016-11-01 20:01 supp_k Status feedback => assigned
2016-11-01 22:29 Muhammad Usama Note Added: 0001146
2016-11-02 17:39 supp_k Note Added: 0001150
2016-11-02 19:36 Muhammad Usama Note Added: 0001151
2016-11-10 21:16 supp_k File Added: server_1
2016-11-10 21:16 supp_k Note Added: 0001162
2016-11-10 21:16 supp_k File Added: server_1-2
2016-11-10 21:16 supp_k File Added: server_1-3
2016-11-10 21:17 supp_k File Added: server_2
2016-11-10 22:47 Muhammad Usama Note Added: 0001163
2016-11-10 22:57 supp_k Note Added: 0001164
2016-11-11 00:12 Muhammad Usama Note Added: 0001165
2016-11-11 17:11 supp_k Note Added: 0001167
2016-11-15 06:42 Muhammad Usama Note Added: 0001170
2016-11-23 17:55 supp_k Note Added: 0001187
2017-08-29 09:34 pengbo Status assigned => closed