View Issue Details

IDProjectCategoryView StatusLast Update
0000164Pgpool-IIBugpublic2016-02-01 17:11
ReporterharukatAssigned Tonagata 
PrioritynormalSeverityminorReproducibilityunable to reproduce
Status resolvedResolutionfixed 
Product Version3.4.0 
Target VersionFixed in Version 
Summary0000164: watchdog authentication failed
DescriptionWe took a report that pgpool-II failed in backend's failback with wachdog
from our customer. (environment: pgpool-II 3.4.1 / RHEL6.x / x86_64)
There are 4 pgpool-II nodes. An pgpool-II couldn't get information about
failback from other node.

The parameter "authkey" of 4 nodes are accorded, but the following
messages are outputted at that time.
  2015-12-08 18:40:50: pid 12345: LOG: failed sending watchdog response
  2015-12-08 18:40:50: pid 12345: DETAIL: watchdog authentication failed

I think a network malfunction or pgpool's runtime error caused this.

I wrote a patch that provides more log messages about authkey
hash calculation. It will help analyzing this phenomenon.
I tested it lightly (with gdb) on recent V3_4_STABLE snapshot.
TagsNo tags attached.

Activities

harukat

2016-01-23 17:23

developer  

V3_4_auth_key_error_check.patch (2,502 bytes)
diff --git a/src/watchdog/wd_child.c b/src/watchdog/wd_child.c
index 3a262fc..ed92be1 100644
--- a/src/watchdog/wd_child.c
+++ b/src/watchdog/wd_child.c
@@ -189,7 +189,14 @@ wd_send_response(int sock, WdPacket * recv_pack)
 		pack_str_len = wd_packet_to_string(recv_pack, pack_str, sizeof(pack_str));
 		wd_calc_hash(pack_str, pack_str_len, hash);
 
-		if (strcmp(recv_pack->hash, hash))
+		if (hash[0] == '\0')
+		{
+			ereport(LOG,
+				(errmsg("failed calculation of own wd_authkey hash")));
+		}
+
+		if (hash[0] == '\0' || (recv_pack->hash)[0] == '\0'
+			 || strcmp(recv_pack->hash, hash))
 		{
 			ereport(LOG,
 				(errmsg("failed sending watchdog response"),
diff --git a/src/watchdog/wd_packet.c b/src/watchdog/wd_packet.c
index 857a3d2..4ec2fab 100644
--- a/src/watchdog/wd_packet.c
+++ b/src/watchdog/wd_packet.c
@@ -551,6 +551,12 @@ wd_thread_negotiation(void * arg)
 		/* calculate hash from packet */
 		pack_str_len = wd_packet_to_string(thread_arg->packet, pack_str, sizeof(pack_str));
 		wd_calc_hash(pack_str, pack_str_len, thread_arg->packet->hash);
+
+		if ((thread_arg->packet->hash)[0] == '\0')
+		{
+            ereport(LOG,
+                (errmsg("failed calculation of own wd_authkey hash for watchdog negotiation")));
+		}
 	}
 
 	/* packet send to target watchdog */
@@ -1211,17 +1217,29 @@ wd_calc_hash(const char *str, int len, char *buf)
 	/* use first half of authkey as username, last half as password */
 	authkey_len = strlen(pool_config->wd_authkey);
 
+	if (len <= 0 || authkey_len <= 0)
+		goto wd_calc_hash_error;
+
 	username_len = authkey_len / 2;
 	pass_len = authkey_len - username_len;
-	snprintf(username, username_len + 1, "%s", pool_config->wd_authkey);
-	snprintf(pass, pass_len + 1, "%s", pool_config->wd_authkey + username_len);
+	if ( snprintf(username, username_len + 1, "%s", pool_config->wd_authkey) < 0
+	  || snprintf(pass, pass_len + 1, "%s", pool_config->wd_authkey + username_len) < 0)
+		goto wd_calc_hash_error;
 
 	/* calculate hash using md5 encrypt */
-	pool_md5_encrypt(pass, username, strlen(username), buf + MD5_PASSWD_LEN + 1);
+	if (! pool_md5_encrypt(pass, username, strlen(username), buf + MD5_PASSWD_LEN + 1))
+		goto wd_calc_hash_error;
 	buf[(MD5_PASSWD_LEN+1)*2-1] = '\0';
 
-	pool_md5_encrypt(buf+MD5_PASSWD_LEN+1, str, len, buf);
+	if (! pool_md5_encrypt(buf+MD5_PASSWD_LEN+1, str, len, buf))
+		goto wd_calc_hash_error;
 	buf[MD5_PASSWD_LEN] = '\0';
+
+	return;
+
+wd_calc_hash_error:
+	buf[0] = '\0';
+	return;
 }
 
 int

nagata

2016-02-01 17:10

developer   ~0000644

Thank you for your reporting and providing the patch.
I pushed this with some changes;

- some fixes for warning messages
- change log level from LOG to warning
- add same changes for heartbeat packet handling

http://git.postgresql.org/gitweb/?p=pgpool2.git;a=commitdiff;h=864c7d6fdafb644f28f8025d3a441f83ba53e64d

I back-ported this to V3_3_STABLE and applied this to V3_5_STABLE and master, tool

Issue History

Date Modified Username Field Change
2016-01-23 17:23 harukat New Issue
2016-01-23 17:23 harukat File Added: V3_4_auth_key_error_check.patch
2016-01-30 10:48 t-ishii Assigned To => nagata
2016-01-30 10:48 t-ishii Status new => assigned
2016-01-30 10:48 t-ishii Description Updated View Revisions
2016-02-01 17:10 nagata Note Added: 0000644
2016-02-01 17:11 nagata Status assigned => resolved
2016-02-01 17:11 nagata Resolution open => fixed