View Issue Details

IDProjectCategoryView StatusLast Update
0000295Pgpool-IIBugpublic2017-03-23 09:08
Reportersupp_kAssigned ToMuhammad Usama 
PriorityhighSeveritymajorReproducibilityalways
Status resolvedResolutionopen 
Platformx86_64OSCentOSOS Version7.x
Product Version3.6.1 
Target VersionFixed in Version 
Summary0000295: Watchdog service fails to start
DescriptionThe issue is detected in the fresh version 3.6.2.
Pgpool fails to start because the Watchdog service doesn't start. Configuration file is attached:

In the logs we observe the error:
"failed to create watchdog heartbeat receive socket"
Additional InformationMar 18 00:22:25 OAA-8a7767db362e pgpool[8622]: [1-1] LOG: reading status file: 1 th backend is set to down status
Mar 18 00:22:25 OAA-8a7767db362e pgpool[8622]: [1-2] LOCATION: pgpool_main.c:3234
Mar 18 00:22:25 OAA-8a7767db362e pgpool[8622]: [2-1] LOG: waiting for watchdog to initialize
Mar 18 00:22:25 OAA-8a7767db362e pgpool[8622]: [2-2] LOCATION: pgpool_main.c:275
Mar 18 00:22:25 OAA-8a7767db362e pgpool[8623]: [2-1] LOG: setting the local watchdog node name to "OAA-8a7767db362e.aqa.int.zone:5432 Linux OAA-8a7767db362e.aqa.int.zone"
Mar 18 00:22:25 OAA-8a7767db362e pgpool[8623]: [2-2] LOCATION: watchdog.c:631
Mar 18 00:22:25 OAA-8a7767db362e pgpool[8623]: [3-1] LOG: watchdog cluster is configured with 2 remote nodes
Mar 18 00:22:25 OAA-8a7767db362e pgpool[8623]: [3-2] LOCATION: watchdog.c:639
Mar 18 00:22:25 OAA-8a7767db362e pgpool[8623]: [4-1] LOG: watchdog remote node:0 on 192.168.64.244:9000
Mar 18 00:22:25 OAA-8a7767db362e pgpool[8623]: [4-2] LOCATION: watchdog.c:650
Mar 18 00:22:25 OAA-8a7767db362e pgpool[8623]: [5-1] LOG: watchdog remote node:1 on 192.168.72.190:9000
Mar 18 00:22:25 OAA-8a7767db362e pgpool[8623]: [5-2] LOCATION: watchdog.c:650
Mar 18 00:22:25 OAA-8a7767db362e pgpool[8623]: [6-1] LOG: interface monitoring is disabled in watchdog
Mar 18 00:22:25 OAA-8a7767db362e pgpool[8623]: [6-2] LOCATION: watchdog.c:532
Mar 18 00:22:25 OAA-8a7767db362e pgpool[8623]: [7-1] LOG: watchdog node state changed from [DEAD] to [LOADING]
Mar 18 00:22:25 OAA-8a7767db362e pgpool[8623]: [7-2] LOCATION: watchdog.c:5758
Mar 18 00:22:30 OAA-8a7767db362e pgpool[8623]: [8-1] LOG: watchdog node state changed from [LOADING] to [JOINING]
Mar 18 00:22:30 OAA-8a7767db362e pgpool[8623]: [8-2] LOCATION: watchdog.c:5758
Mar 18 00:22:34 OAA-8a7767db362e pgpool[8623]: [9-1] LOG: watchdog node state changed from [JOINING] to [INITIALIZING]
Mar 18 00:22:34 OAA-8a7767db362e pgpool[8623]: [9-2] LOCATION: watchdog.c:5758
Mar 18 00:22:35 OAA-8a7767db362e pgpool[8623]: [10-1] LOG: I am the only alive node in the watchdog cluster
Mar 18 00:22:35 OAA-8a7767db362e pgpool[8623]: [10-2] HINT: skiping stand for coordinator state
Mar 18 00:22:35 OAA-8a7767db362e pgpool[8623]: [10-3] LOCATION: watchdog.c:4858
Mar 18 00:22:35 OAA-8a7767db362e pgpool[8623]: [11-1] LOG: watchdog node state changed from [INITIALIZING] to [MASTER]
Mar 18 00:22:35 OAA-8a7767db362e pgpool[8623]: [11-2] LOCATION: watchdog.c:5758
Mar 18 00:22:35 OAA-8a7767db362e pgpool[8623]: [12-1] LOG: I am announcing my self as master/coordinator watchdog node
Mar 18 00:22:35 OAA-8a7767db362e pgpool[8623]: [12-2] LOCATION: watchdog.c:5039
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8623]: [13-1] LOG: I am the cluster leader node
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8623]: [13-2] DETAIL: our declare coordinator message is accepted by all nodes
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8623]: [13-3] LOCATION: watchdog.c:5072
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8623]: [14-1] LOG: I am the cluster leader node but we do not have enough nodes in cluster
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8623]: [14-2] DETAIL: waiting for the quorum to start escalation process
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8623]: [14-3] LOCATION: watchdog.c:5085
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8622]: [3-1] LOG: watchdog process is initialized
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8622]: [3-2] LOCATION: pgpool_main.c:289
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8623]: [15-1] LOG: new IPC connection received
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8623]: [15-2] LOCATION: watchdog.c:3103
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8623]: [16-1] LOG: received the get data request from local pgpool-II on IPC interface
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8623]: [16-2] LOCATION: watchdog.c:2289
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8622]: [4-1] LOG: I am the master watchdog node
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8622]: [4-2] DETAIL: using the local backend node status
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8622]: [4-3] LOCATION: pgpool_main.c:3460
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8622]: [5-1] LOG: Setting up socket for 0.0.0.0:5432
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8622]: [5-2] LOCATION: pgpool_main.c:842
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8622]: [6-1] LOG: Setting up socket for :::5432
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8622]: [6-2] LOCATION: pgpool_main.c:842
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8623]: [17-1] LOG: new IPC connection received
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8623]: [17-2] LOCATION: watchdog.c:3103
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8624]: [4-1] LOG: 3 watchdog nodes are configured for lifecheck
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8624]: [4-2] LOCATION: wd_lifecheck.c:475
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8624]: [5-1] LOG: watchdog nodes ID:0 Name:"OAA-8a7767db362e.aqa.int.zone:5432 Linux OAA-8a7767db362e.aqa.int.zone"
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8624]: [5-2] DETAIL: Host:"OAA-8a7767db362e.aqa.int.zone" WD Port:9000 pgpool-II port:5432
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8624]: [5-3] LOCATION: wd_lifecheck.c:483
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8624]: [6-1] LOG: watchdog nodes ID:1 Name:"Not_Set"
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8624]: [6-2] DETAIL: Host:"192.168.64.244" WD Port:9000 pgpool-II port:5432
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8624]: [6-3] LOCATION: wd_lifecheck.c:483
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8624]: [7-1] LOG: watchdog nodes ID:2 Name:"Not_Set"
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8624]: [7-2] DETAIL: Host:"192.168.72.190" WD Port:9000 pgpool-II port:5432
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8624]: [7-3] LOCATION: wd_lifecheck.c:483
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8622]: [7-1] LOG: pgpool-II successfully started. version 3.6.2 (subaruboshi)
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8622]: [7-2] LOCATION: pgpool_main.c:392
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8622]: [8-1] LOG: find_primary_node: checking backend no 0
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8622]: [8-2] LOCATION: pgpool_main.c:2890
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8622]: [9-1] LOG: find_primary_node: primary node id is 0
Mar 18 00:22:39 OAA-8a7767db362e pgpool[8622]: [9-2] LOCATION: pgpool_main.c:2920
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8644]: [8-1] LOG: creating socket for sending heartbeat
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8644]: [8-2] DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8644]: [8-3] LOCATION: wd_heartbeat.c:131
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8644]: [9-1] LOG: set SO_REUSEPORT option to the socket
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8644]: [9-2] LOCATION: wd_heartbeat.c:683
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8644]: [10-1] LOG: creating socket for sending heartbeat
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8644]: [10-2] DETAIL: set SO_REUSEPORT
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8644]: [10-3] LOCATION: wd_heartbeat.c:143
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8643]: [8-1] LOG: failed to create watchdog heartbeat receive socket.
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8643]: [8-2] DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8643]: [8-3] LOCATION: wd_heartbeat.c:212
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8643]: [9-1] LOG: set SO_REUSEPORT option to the socket
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8643]: [9-2] LOCATION: wd_heartbeat.c:683
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8643]: [10-1] LOG: creating watchdog heartbeat receive socket.
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8643]: [10-2] DETAIL: set SO_REUSEPORT
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8643]: [10-3] LOCATION: wd_heartbeat.c:225
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8642]: [8-1] LOG: creating socket for sending heartbeat
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8642]: [8-2] DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8642]: [8-3] LOCATION: wd_heartbeat.c:131
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8642]: [9-1] LOG: set SO_REUSEPORT option to the socket
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8642]: [9-2] LOCATION: wd_heartbeat.c:683
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8642]: [10-1] LOG: creating socket for sending heartbeat
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8642]: [10-2] DETAIL: set SO_REUSEPORT
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8642]: [10-3] LOCATION: wd_heartbeat.c:143
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8641]: [8-1] LOG: failed to create watchdog heartbeat receive socket.
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8641]: [8-2] DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8641]: [8-3] LOCATION: wd_heartbeat.c:212
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8641]: [9-1] LOG: set SO_REUSEPORT option to the socket
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8641]: [9-2] LOCATION: wd_heartbeat.c:683
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8641]: [10-1] LOG: creating watchdog heartbeat receive socket.
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8641]: [10-2] DETAIL: set SO_REUSEPORT
Mar 18 00:22:40 OAA-8a7767db362e pgpool[8641]: [10-3] LOCATION: wd_heartbeat.c:225
TagsNo tags attached.

Activities

supp_k

2017-03-18 06:28

reporter  

pgpool.conf (4,189 bytes)

supp_k

2017-03-19 02:17

reporter   ~0001380

Pgpool Guys,

sorry for the wrong alarm. The issue is not an issue, the fail was detected by CI + yum that resolved dependencies incorrectly. It happen because it was targeted to download the latest version of Pgpool. But that the version was updated...

supp_k

2017-03-21 21:12

reporter   ~0001382

This issue relates to the issue 0000296

t-ishii

2017-03-23 09:07

developer   ~0001390

Issue closed.

Issue History

Date Modified Username Field Change
2017-03-18 06:28 supp_k New Issue
2017-03-18 06:28 supp_k File Added: pgpool.conf
2017-03-19 02:17 supp_k Note Added: 0001380
2017-03-21 21:12 supp_k Note Added: 0001382
2017-03-23 09:07 t-ishii Assigned To => Muhammad Usama
2017-03-23 09:07 t-ishii Status new => assigned
2017-03-23 09:07 t-ishii Note Added: 0001390
2017-03-23 09:08 t-ishii Status assigned => resolved