View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0000281Pgpool-IIBugpublic2017-01-27 06:132017-08-29 09:44
Reportersupp_k Assigned ToMuhammad Usama  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
Platformx86 64OSCentOSOS Version7.x
Product Version3.6.1 
Summary0000281: "segmentation fault" when execute pcp_attach_node
DescriptionIn case the listed below configuration parameters are activated the "pcp_attach_node 0 -h 127.0.0.1" results in "segmentation fault".

debug_level = 5
log_error_verbosity = verbose
client_min_messages = error
log_min_messages = debug5



Environment consists of:
3x Pgpool instances
2x PostgreSQL backends
Additional Informationlog records:

Jan 26 23:34:58 OAA-72c02c29743e pgpool[4467]: [255-1] LOG: informing the node status change to watchdog
Jan 26 23:34:58 OAA-72c02c29743e pgpool[4467]: [255-2] DETAIL: node id :1 status = "NODE DEAD" message:"No heartbeat signal from node"
Jan 26 23:34:58 OAA-72c02c29743e pgpool[4467]: [255-3] LOCATION: wd_lifecheck.c:509
Jan 26 23:34:58 OAA-72c02c29743e pgpool[4466]: [409-1] LOG: new IPC connection received
Jan 26 23:34:58 OAA-72c02c29743e pgpool[4466]: [409-2] LOCATION: watchdog.c:3085
Jan 26 23:34:58 OAA-72c02c29743e pgpool[4466]: [410-1] LOG: received node status change ipc message
Jan 26 23:34:58 OAA-72c02c29743e pgpool[4466]: [410-2] DETAIL: No heartbeat signal from node
Jan 26 23:34:58 OAA-72c02c29743e pgpool[4466]: [410-3] LOCATION: watchdog.c:1807
Jan 26 23:34:58 OAA-72c02c29743e pgpool[4466]: [413-1] LOG: remote node "OAB-72c02c29743e.aqa.int.zone:5432 Linux OAB-72c02c29743e.aqa.int.zone" is shutting down
Jan 26 23:34:58 OAA-72c02c29743e pgpool[4466]: [413-2] LOCATION: watchdog.c:4489
Jan 26 23:35:12 OAA-72c02c29743e su: (to postgres) root on none
Jan 26 23:36:00 OAA-72c02c29743e su: (to postgres) root on none
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4466]: [499-1] LOG: new watchdog node connection is received from "192.168.102.95:49802"
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4466]: [499-2] LOCATION: watchdog.c:3053
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4466]: [506-1] LOG: new node joined the cluster hostname:"OAB-72c02c29743e.aqa.int.zone" port:9000 pgpool_port:5432
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4466]: [506-2] LOCATION: watchdog.c:1402
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4466]: [510-1] LOG: new outbond connection to 192.168.102.95:9000
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4466]: [510-2] LOCATION: watchdog.c:3126
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4712]: [188-1] LOG: forked new pcp worker, pid=4905 socket=8
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4712]: [188-2] LOCATION: pcp_child.c:275
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4905]: [196-1] LOG: received failback request for node_id: 1 from pid [4905] wd_failover_id [0]
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4905]: [196-2] LOCATION: pgpool_main.c:1323
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4466]: [529-1] LOG: new IPC connection received
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4466]: [529-2] LOCATION: watchdog.c:3085
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4466]: [530-1] LOG: watchdog received the failover command from local pgpool-II on IPC interface
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4466]: [530-2] LOCATION: watchdog.c:1988
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4466]: [531-1] LOG: watchdog is processing the failover command [FAILBACK_REQUEST] received from local pgpool-II on IPC interface
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4466]: [531-2] LOCATION: watchdog.c:2073
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4905]: [197-1] LOG: read from socket failed, remote end closed the connection
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4905]: [197-2] LOCATION: pool_stream.c:1219
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4905]: [198-1] WARNING: error reading from IPC command socket for ipc command f
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4905]: [198-2] DETAIL: read from socket failed with error "Success"
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4905]: [198-3] LOCATION: wd_commands.c:199
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4905]: [199-1] ERROR: failback request for node_id: 1 from pid [4905] is canceled by other pgpool
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4905]: [199-2] LOCATION: pgpool_main.c:1363
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4465]: [193-1] WARNING: watchdog child process with pid: 4466 was terminated by segmentation fault
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4465]: [193-2] LOCATION: pgpool_main.c:2352
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4465]: [194-1] LOG: fork a new watchdog child process with pid: 4906
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4465]: [194-2] LOCATION: pgpool_main.c:2445
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4906]: [194-1] LOG: setting the local watchdog node name to "OAA-72c02c29743e.aqa.int.zone:5432 Linux OAA-72c02c29743e.aqa.int.zone"
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4906]: [194-2] LOCATION: watchdog.c:631
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4906]: [195-1] LOG: watchdog cluster is configured with 2 remote nodes
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4906]: [195-2] LOCATION: watchdog.c:639
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4906]: [196-1] LOG: watchdog remote node:0 on 192.168.102.95:9000
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4906]: [196-2] LOCATION: watchdog.c:650
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4906]: [197-1] LOG: watchdog remote node:1 on 192.168.224.75:9000
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4906]: [197-2] LOCATION: watchdog.c:650
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4906]: [198-1] LOG: interface monitoring is disabled in watchdog
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4906]: [198-2] LOCATION: watchdog.c:532
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4906]: [199-1] INFO: IPC socket path: "/var/run/pgpool/.s.PGPOOLWD_CMD.9000"
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4906]: [199-2] LOCATION: watchdog.c:1148
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4906]: [200-1] FATAL: failed to create watchdog command server socket
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4906]: [200-2] DETAIL: bind on "/var/run/pgpool/.s.PGPOOLWD_CMD.9000" failed with reason: "Address already in use"
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4906]: [200-3] LOCATION: watchdog.c:1159
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4712]: [190-1] LOG: PCP process with pid: 4905 exit with SUCCESS.
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4712]: [190-2] LOCATION: pcp_child.c:327
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4712]: [191-1] LOG: PCP process with pid: 4905 exits with status 0
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4712]: [191-2] LOCATION: pcp_child.c:341
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4465]: [215-1] LOG: watchdog child process with pid: 4906 exits with status 768
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4465]: [215-2] LOCATION: pgpool_main.c:2359
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4465]: [216-1] FATAL: watchdog child process exit with fatal error. exiting pgpool-II
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4465]: [216-2] LOCATION: pgpool_main.c:2439
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4908]: [216-1] LOG: setting the local watchdog node name to "OAA-72c02c29743e.aqa.int.zone:5432 Linux OAA-72c02c29743e.aqa.int.zone"
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4908]: [216-2] LOCATION: watchdog.c:631
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4908]: [217-1] LOG: watchdog cluster is configured with 2 remote nodes
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4908]: [217-2] LOCATION: watchdog.c:639
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4908]: [218-1] LOG: watchdog remote node:0 on 192.168.102.95:9000
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4908]: [218-2] LOCATION: watchdog.c:650
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4908]: [219-1] LOG: watchdog remote node:1 on 192.168.224.75:9000
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4908]: [219-2] LOCATION: watchdog.c:650
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4908]: [220-1] LOG: interface monitoring is disabled in watchdog
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4908]: [220-2] LOCATION: watchdog.c:532
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4908]: [221-1] INFO: IPC socket path: "/var/run/pgpool/.s.PGPOOLWD_CMD.9000"
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4908]: [221-2] LOCATION: watchdog.c:1148
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4908]: [222-1] LOG: watchdog node state changed from [DEAD] to [LOADING]
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4908]: [222-2] LOCATION: watchdog.c:5740
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4908]: [224-1] LOG: new outbond connection to 192.168.102.95:9000
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4908]: [224-2] LOCATION: watchdog.c:3126
Jan 26 23:36:00 OAA-72c02c29743e pgpool[4908]: [227-1] LOG: new outbond connection to 192.168.224.75:9000

TagsNo tags attached.

Activities

Muhammad Usama

2017-01-28 00:32

developer   ~0001303

Hi

Does this crash happen all the time? And if possible can you please share the core dump of the crash.

Thanks!

supp_k

2017-01-29 05:01

reporter   ~0001304

Yes,
it reproduced all the time. I'm failed to collect the core dump when the pgpool is started using systemd, so I reproduced the issue using the command #pgpool -n -f pgpool.conf

The attached file contains configuration files and the core dump.

Probably important points:
1) the pcp_attach_node command was executed in the server with the MASTER pgpool. The 1st backend was being attached ( pcp_attach_node 0 -h 127.0.0.1)
2) 2xPG backends configured. The 2nd was offline.
pgpool_fail.zip (645,229 bytes)

Muhammad Usama

2017-01-31 03:14

developer   ~0001313

Many thanks for providing the core dump. It was really helpful in identifing and fixing the issue.
I have committed the fix in master, 3.5 and 3.6 branches.

http://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=5b189f7b7c0434ec75831b51bc1647445751c0c7

Issue History

Date Modified Username Field Change
2017-01-27 06:13 supp_k New Issue
2017-01-28 00:32 Muhammad Usama Note Added: 0001303
2017-01-29 05:01 supp_k File Added: pgpool_fail.zip
2017-01-29 05:01 supp_k Note Added: 0001304
2017-01-31 03:14 Muhammad Usama Assigned To => Muhammad Usama
2017-01-31 03:14 Muhammad Usama Status new => resolved
2017-01-31 03:14 Muhammad Usama Resolution open => fixed
2017-01-31 03:14 Muhammad Usama Note Added: 0001313