View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0000732Pgpool-IIBugpublic2021-09-24 08:452021-12-07 14:05
Reporterharukat Assigned Tot-ishii  
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionfixed 
Product Version4.2.5 
Target Version4.2.6Fixed in Version4.2.6 
Summary0000732: Segmentation fault at failover in standby only streaming_replication mode.
DescriptionI met SEGV at failover in our standby only cluster on streaming_replication mode.
I can avoid SEGV by not to use %P, %N, %S for failover_command.
I think it can be fixed to add appropriate null pointer check code.
Steps To Reproduce(1) Use setting based on pgpool.conf.sample-stream
(2) All backend postgres are hot standby, no primary postgres.
(3) Stop a backend postrges to trigger failover => SEGV occurs
Additional Informationcore inspection:

Program terminated with signal 11, Segmentation fault.
#0 0x00007f57c767d721 in __wmemcmp_ssse3 () at ../sysdeps/x86_64/multiarch/memcmp-ssse3.S:1111
1111 jnz L(exit)
Missing separate debuginfos, use: debuginfo-install audit-libs-2.7.6-3.el7.x86_64 cyrus-sasl-lib-2.1.26-23.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-37.el7_6.x86_64 libcap-ng-0.7.5-4.el7.x86_64 libcom_err-1.42.9-13.el7.x86_64 libmemcached-1.0.16-5.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 nspr-4.25.0-2.el7_9.x86_64 nss-3.53.1-3.el7_9.x86_64 nss-softokn-freebl-3.53.1-6.el7_9.x86_64 nss-util-3.53.1-1.el7_9.x86_64 openldap-2.4.44-21.el7_6.x86_64 openssl-libs-1.0.2k-16.el7_6.1.x86_64 pam-1.1.8-18.el7.x86_64 pcre-8.32-17.el7.x86_64 postgresql13-libs-13.4-1PGDG.rhel7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) bt
#0 0x00007f57c767d721 in __wmemcmp_ssse3 () at ../sysdeps/x86_64/multiarch/memcmp-ssse3.S:1111
0000001 0x00000000004a6226 in string_append_char (string=string@entry=0x7f57c9824280, append_data=0x0) at pool_string.c:59
0000002 0x000000000040c701 in trigger_failover_command (node=3, command_line=<optimized out>, old_main_node=0,
    new_main_node=0, old_primary=-1) at main/pgpool_main.c:2846
0000003 0x0000000000410de7 in failover () at main/pgpool_main.c:1797
0000004 0x000000000040f713 in sigusr1_interupt_processor () at main/pgpool_main.c:1329
0000005 0x00000000004129ef in PgpoolMain (discard_status=discard_status@entry=0 '\000',
    clear_memcache_oidmaps=clear_memcache_oidmaps@entry=0 '\000') at main/pgpool_main.c:473
0000006 0x0000000000409aea in main (argc=<optimized out>, argv=<optimized out>) at main/main.c:365
(gdb) f 2
0000002 0x000000000040c701 in trigger_failover_command (node=3, command_line=<optimized out>, old_main_node=0,
    new_main_node=0, old_primary=-1) at main/pgpool_main.c:2846
2846 string_append_char(exec_cmd, "%");
(gdb) f 1
0000001 0x00000000004a6226 in string_append_char (string=string@entry=0x7f57c9824280, append_data=0x0) at pool_string.c:59
59 int len = strlen(append_data);


core inspection: (-O0 option build)

(gdb) bt
#0 0x00007f91837f2721 in __wmemcmp_ssse3 () at ../sysdeps/x86_64/multiarch/memcmp-ssse3.S:1111
0000001 0x00000000004e9d38 in string_append_char (string=0x7f91850f1280, append_data=0x0)
    at pool_string.c:59
0000002 0x000000000040db5b in trigger_failover_command (node=3,
    command_line=0x7f91850f3db1 "%N %S", old_main_node=0, new_main_node=0, old_primary=-1)
    at main/pgpool_main.c:2836
0000003 0x000000000040b78b in failover () at main/pgpool_main.c:1797
0000004 0x0000000000409dda in sigusr1_interupt_processor () at main/pgpool_main.c:1329
0000005 0x00000000004079dc in PgpoolMain (discard_status=0 '\000', clear_memcache_oidmaps=0 '\000')
    at main/pgpool_main.c:473
0000006 0x0000000000405c20 in main (argc=2, argv=0x7ffdd25001c8) at main/main.c:365
(gdb) f 2
0000002 0x000000000040db5b in trigger_failover_command (node=3,
    command_line=0x7f91850f3db1 "%N %S", old_main_node=0, new_main_node=0, old_primary=-1)
    at main/pgpool_main.c:2836
2836 string_append_char(exec_cmd, oldprimary->backend_hostname);
(gdb) p command_line
$1 = 0x7f91850f3db1 "%N %S"
(gdb) p *command_line
$2 = 37 '%'
(gdb) p *exec_cmd
$3 = {size = 1, len = 120,
  data = 0x7f91850f68d8 "/etc/pgpool-II/failover.sh 3 192.168.3.106 5432 /var/lib/pgsql/13/data 0 192.168.3.101 0 -1 5432 /var/lib/pgsql/13/data "}
(gdb) p oldprimary
$4 = (BackendInfo *) 0x0
TagsNo tags attached.

Activities

Muhammad Usama

2021-09-24 13:49

developer   ~0003938

I have cooked up a quick patch for the fix (against the master branch).
You can try if it works
fix_bug_732_master.diff (1,123 bytes)    
diff --git a/src/main/pgpool_main.c b/src/main/pgpool_main.c
index 45039c3d..fda8e6db 100644
--- a/src/main/pgpool_main.c
+++ b/src/main/pgpool_main.c
@@ -2866,13 +2866,23 @@ trigger_failover_command(int node, const char *command_line,
 
 					case 'N':	/* old primary host name */
 						oldprimary = pool_get_node_info(old_primary);
-						string_append_char(exec_cmd, oldprimary->backend_hostname);
+						if (oldprimary)
+							string_append_char(exec_cmd, oldprimary->backend_hostname);
+						else
+							/* no valid old primary */
+							string_append_char(exec_cmd, "\"\"");
 						break;
 
 					case 'S':	/* old primary port */
 						oldprimary = pool_get_node_info(old_primary);
-						snprintf(port_buf, sizeof(port_buf), "%d", oldprimary->backend_port);
-						string_append_char(exec_cmd, port_buf);
+						if (oldprimary)
+						{
+							snprintf(port_buf, sizeof(port_buf), "%d", oldprimary->backend_port);
+							string_append_char(exec_cmd, port_buf);
+						}
+						else
+							/* no valid old primary */
+							string_append_char(exec_cmd, "\"\"");
 						break;
 
 					case '%':	/* escape */
fix_bug_732_master.diff (1,123 bytes)   

t-ishii

2021-09-24 14:56

developer   ~0003939

Thanks Usama! Let's commit this as soon as the reporter confirms the patch fixes the issue.

Muhammad Usama

2021-09-24 16:07

developer   ~0003940

Sure will do.

harukat

2021-09-27 14:54

developer   ~0003941

Thanks.
I tested the patch and confirmed that it prevents SEGV.

Muhammad Usama

2021-09-28 02:23

developer   ~0003942

Thanks for the testing and confirmation. I have pushed the fix for all affected branches.

https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commitdiff;h=52704b01b0acf1fccd025d9ad0fcf296506e12f8

Issue History

Date Modified Username Field Change
2021-09-24 08:45 harukat New Issue
2021-09-24 11:44 t-ishii Assigned To => t-ishii
2021-09-24 11:44 t-ishii Status new => assigned
2021-09-24 13:49 Muhammad Usama Note Added: 0003938
2021-09-24 13:49 Muhammad Usama File Added: fix_bug_732_master.diff
2021-09-24 14:56 t-ishii Note Added: 0003939
2021-09-24 14:57 t-ishii Status assigned => feedback
2021-09-24 16:07 Muhammad Usama Note Added: 0003940
2021-09-27 14:54 harukat Note Added: 0003941
2021-09-27 14:54 harukat Status feedback => assigned
2021-09-28 02:23 Muhammad Usama Note Added: 0003942
2021-09-28 02:23 Muhammad Usama Status assigned => resolved
2021-09-28 02:23 Muhammad Usama Resolution open => fixed
2021-12-07 14:05 administrator Status resolved => closed
2021-12-07 14:05 administrator Fixed in Version => 4.2.6
2021-12-07 14:05 administrator Target Version => 4.2.6