View Issue Details

IDProjectCategoryView StatusLast Update
0000248Pgpool-IIBugpublic2016-09-28 09:34
Reportersupp_kAssigned Tot-ishii 
PriorityhighSeveritymajorReproducibilityalways
Status resolvedResolutionopen 
Platformx86 64OSCentOsOS Version6.x & 7.x
Product Version3.5.3 
Target Version3.5.5Fixed in Version 
Summary0000248: pgpool doesnt restores backend connections
DescriptionThe pcp_attach_node tool doesnt give any result if all backends were down.

Detail: all backend nodes are down, pgpool requires at least one valid node.
Steps To ReproduceEnvironment:
- 3 pgpool nodes (watchdog cluster)
- 2 postgresql backends


Actions:
- stop 2 backends

The pgpool states there are valid backends and it is ok. But restoring at least 1 PostgreSQL instance and attaching it to pgpool (master) with pcp_attach_node gives no result. Pgpool still states there no valid backends though pgpool_status reflects the attached backend is up.


Only full restart of all pgpool nodes helps.



TagsNo tags attached.

Activities

t-ishii

2016-09-21 09:18

developer   ~0001074

Last edited: 2016-09-28 07:37

View 2 revisions

Good catch. Just we did not consider the case (all backends are down then 1 node is recovered by pcp_attach_node). Attached patch should fix the problem. Please try.

t-ishii

2016-09-21 09:21

developer  

all_backend_down.diff (2,294 bytes)
diff --git a/src/main/pgpool_main.c b/src/main/pgpool_main.c
index 1f0b8e0..1f7aa27 100644
--- a/src/main/pgpool_main.c
+++ b/src/main/pgpool_main.c
@@ -1487,6 +1487,7 @@ static void failover(void)
 	int status;
 	int sts;
 	bool need_to_restart_pcp = false;
+	bool all_backend_down = true;
 
 	ereport(DEBUG1,
 		(errmsg("failover handler called")));
@@ -1594,6 +1595,20 @@ static void failover(void)
 					 BACKEND_INFO(node_id).backend_hostname,
 					 BACKEND_INFO(node_id).backend_port)));
 
+			/* Check to see if all backends are down */
+			for (i=0;i<NUM_BACKENDS;i++)
+			{
+				if (BACKEND_INFO(i).backend_status != CON_DOWN &&
+					BACKEND_INFO(i).backend_status != CON_UNUSED)
+				{
+					ereport(LOG,
+							(errmsg("Node %d is not down (status: %d)",
+									i, BACKEND_INFO(i).backend_status)));
+					all_backend_down = false;
+					break;
+				}
+			}
+
 			BACKEND_INFO(node_id).backend_status = CON_CONNECT_WAIT;	/* unset down status */
 			(void)write_status_file();
 
@@ -1735,11 +1750,22 @@ static void failover(void)
 		* attached node, but load balanced node is not changed until this
 		* session ends, so it's harmless anyway.
 		*/
-		if (MASTER_SLAVE && !strcmp(pool_config->master_slave_sub_mode, MODE_STREAMREP)	&&
-			reqkind == NODE_UP_REQUEST)
+
+		/*
+		 * On 2015/9/21 Tatsuo Ishii says: this judgment is not sufficient if
+		 * all backends were down. Child process has local status in which all
+		 * backends are down. In this case even if new connection arrives from
+		 * frontend, the child will not accept it because the local status
+		 * shows all backends are down. For this purpose we refer to
+		 * "all_backend_down" variable, which was set before updating backend status.
+		 *
+		 * See bug 248 for more details.
+		 */
+
+		if (STREAM && reqkind == NODE_UP_REQUEST &&	all_backend_down == false)
 		{
 			ereport(LOG,
-					(errmsg("Do not restart children because we are failbacking node id %d host: %s port: %d and we are in streaming replication mode", node_id,
+					(errmsg("Do not restart children because we are failbacking node id %d host: %s port: %d and we are in streaming replication mode and not all backends were down", node_id,
 					 BACKEND_INFO(node_id).backend_hostname,
 					 BACKEND_INFO(node_id).backend_port)));
 
all_backend_down.diff (2,294 bytes)

supp_k

2016-09-28 05:07

reporter   ~0001094

Hi,

quick test reveals the issue is fixed. When will it be applied to the major cosdebase - Say V3_5?

t-ishii

2016-09-28 07:39

developer   ~0001095

It has been already applied to V3_5 and other branches.

http://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=b306e04dd5c4a6c95cc5d97e0203e280d3983cb4

Issue History

Date Modified Username Field Change
2016-09-20 05:45 supp_k New Issue
2016-09-21 09:07 t-ishii Assigned To => t-ishii
2016-09-21 09:07 t-ishii Status new => assigned
2016-09-21 09:07 t-ishii Target Version => 3.5.4
2016-09-21 09:18 t-ishii Note Added: 0001074
2016-09-21 09:21 t-ishii File Added: all_backend_down.diff
2016-09-21 14:59 t-ishii Target Version 3.5.4 => 3.5.5
2016-09-21 14:59 t-ishii Status assigned => feedback
2016-09-28 05:07 supp_k Note Added: 0001094
2016-09-28 05:07 supp_k Status feedback => assigned
2016-09-28 07:37 t-ishii Note Edited: 0001074 View Revisions
2016-09-28 07:39 t-ishii Note Added: 0001095
2016-09-28 09:34 t-ishii Status assigned => resolved