View Issue Details

IDProjectCategoryView StatusLast Update
0000062Pgpool-IIBugpublic2013-07-24 07:47
ReportertuomasAssigned Tot-ishii 
PrioritynormalSeveritymajorReproducibilitysometimes
Status resolvedResolutionopen 
PlatformLinux 64bitOSUbuntuOS Version12.04
Product Version 
Target VersionFixed in Version 
Summary0000062: Slave network outage causes a segmentation fault on main process
Descriptionpgpool log is attached.

postgresql 9.1.x, pgpool 3.2.4. Streaming replication mode, load balancing off, parallel off, memory cache off.

Network outage (packet loss) on slave caused pgpool main process to segfault.

gdb gives only this:
Core was generated by `pgpool -n'.
Program terminated with signal 11, Segmentation fault.
#0 __memmove_ssse3_back () at ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:1676
1676 ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S: No such file or directory.
(gdb) bt
#0 __memmove_ssse3_back () at ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:1676
0000001 0x0000000000000000 in ?? ()

Steps To ReproduceIt has happened a couple of times so far, but not sure how to reproduce it.
Additional InformationThe main(?) process dies, but the child processes stay alive and the port reserved, so it can't just be automatically restarted.
TagsNo tags attached.

Activities

tuomas

2013-06-26 01:29

reporter  

pgpool.log (19,139 bytes)

t-ishii

2013-06-26 08:26

developer   ~0000294

The log shows that pgpool main process detected an error in reading the socket to backend. I examined the code related to this and found the low layer function does not handle the error case correctly when used in main process and possibbly causes memory copy overrun later on. I believe this is the cause of the segfault. Attached patch should fix the problem. Could you try it out?

t-ishii

2013-06-26 08:27

developer  

pool_stream.c.patch (1,399 bytes)
diff --git a/pool_stream.c b/pool_stream.c
index b62bd61..f1e2990 100644
--- a/pool_stream.c
+++ b/pool_stream.c
@@ -137,7 +137,7 @@ int pool_read(POOL_CONNECTION *cp, void *buf, int len)
 	{
 		if (pool_check_fd(cp))
 		{
-			if (!IS_MASTER_NODE_ID(cp->db_node_id))
+			if (!IS_MASTER_NODE_ID(cp->db_node_id) && (getpid() == mypid))
 			{
 				pool_log("pool_read: data is not ready in DB node: %d. abort this session",
 						 cp->db_node_id);
@@ -173,6 +173,8 @@ int pool_read(POOL_CONNECTION *cp, void *buf, int len)
 				{
 					notice_backend_error(cp->db_node_id);
 					child_exit(1);
+					pool_log("pool_read: do not failover because I am the main process");
+					return -1;
 				}
 				else
 				{
@@ -296,6 +298,8 @@ char *pool_read2(POOL_CONNECTION *cp, int len)
 				{
 					notice_backend_error(cp->db_node_id);
 					child_exit(1);
+					pool_log("pool_read2: do not failover because I am the main process");
+					return NULL;
 				}
 				else
 				{
@@ -508,6 +512,8 @@ int pool_flush(POOL_CONNECTION *cp)
 			{
 				notice_backend_error(cp->db_node_id);
 				child_exit(1);
+				pool_log("pool_flush: do not failover because I am the main process");
+				return -1;
 			}
 			else
 			{
@@ -650,6 +656,7 @@ char *pool_read_string(POOL_CONNECTION *cp, int *len, int line)
 			{
 				notice_backend_error(cp->db_node_id);
 				child_exit(1);
+				return NULL;
 			}
 			else
 			{
pool_stream.c.patch (1,399 bytes)

tuomas

2013-06-26 23:59

reporter   ~0000296

Thanks! I'll give it a try.

tuomas

2013-07-24 04:27

reporter   ~0000308

The problem hasn't reappeared after installing the patch, so this seems to fix it. Thanks!

t-ishii

2013-07-24 07:47

developer   ~0000309

Thanks for the report! The fix will appear in the next pgpool-II minor version up release.

Issue History

Date Modified Username Field Change
2013-06-26 01:29 tuomas New Issue
2013-06-26 01:29 tuomas File Added: pgpool.log
2013-06-26 08:26 t-ishii Note Added: 0000294
2013-06-26 08:26 t-ishii Assigned To => t-ishii
2013-06-26 08:26 t-ishii Status new => assigned
2013-06-26 08:27 t-ishii File Added: pool_stream.c.patch
2013-06-26 11:57 t-ishii Status assigned => feedback
2013-06-26 23:59 tuomas Note Added: 0000296
2013-06-26 23:59 tuomas Status feedback => assigned
2013-07-01 11:57 t-ishii Status assigned => feedback
2013-07-20 13:01 t-ishii Changeset attached => pgpool2 master 7c09d884
2013-07-24 04:27 tuomas Note Added: 0000308
2013-07-24 04:27 tuomas Status feedback => assigned
2013-07-24 07:47 t-ishii Note Added: 0000309
2013-07-24 07:47 t-ishii Status assigned => resolved