[pgpool-general: 6212] Add to watchdog cluster request is rejected by node

[Sebastian Lefke]

Hi,

i have the following setup
Server1D = primary database server
Server2D = hot-standby (straming repl.) database server
Server3P = pgpool2 v.  3.7.5
Server4P = pgpool2 v.  3.7.5

When I start pgpool2 on Server3P or Server4P it starts fine and it takes the virtual ip.
But when I want to start another pgpool server it can not connect to the master node and exits.
So I can only have one active pgpool Server, other nodes get rejected.

I checked the configuration files a dozen times, but I can't find any wrong settings. Notice that some
configuration parameter are at the end of the file.

When i start pgpool2 on Server4P first:
Log on Server 3P:
Aug 23 16:51:11 SERVER3P pgpool[19864]: [184-2] 2018-08-23 16:51:11: pid 19864:LOCATION:  watchdog.c:2802
Aug 23 16:51:11 SERVER3P pgpool[19864]: [185-1] 2018-08-23 16:51:11: pid 19864:DEBUG:  STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = LOADING
Aug 23 16:51:11 SERVER3P pgpool[19864]: [185-2] 2018-08-23 16:51:11: pid 19864:LOCATION:  watchdog.c:4640
Aug 23 16:51:11 SERVER3P pgpool[19864]: [186-1] 2018-08-23 16:51:11: pid 19864:DEBUG:  received packet, watchdog node:[] command id:[4] type:[REJECT] state:[LOADING]
Aug 23 16:51:11 SERVER3P pgpool[19864]: [186-2] 2018-08-23 16:51:11: pid 19864:LOCATION:  watchdog.c:6673
Aug 23 16:51:11 SERVER3P pgpool[19864]: [187-1] 2018-08-23 16:51:11: pid 19864:FATAL:  Add to watchdog cluster request is rejected by node "SERVER4P:9000"
Aug 23 16:51:11 SERVER3P pgpool[19864]: [187-2] 2018-08-23 16:51:11: pid 19864:HINT:  check the watchdog configurations.
Aug 23 16:51:11 SERVER3P pgpool[19864]: [187-3] 2018-08-23 16:51:11: pid 19864:LOCATION:  watchdog.c:4879
Aug 23 16:51:11 SERVER3P pgpool[19864]: [188-1] 2018-08-23 16:51:11: pid 19864:LOG:  Watchdog is shutting down
Aug 23 16:51:11 SERVER3P pgpool[19864]: [188-2] 2018-08-23 16:51:11: pid 19864:LOCATION:  watchdog.c:2894
Log on Server 4P:
Aug 23 16:25:43 SERVER4P pgpool[7982]: [1-1] 2018-08-23 16:25:43: pid 7982:LOG:  waiting for watchdog to initialize
Aug 23 16:25:43 SERVER4P pgpool[7990]: [1-1] 2018-08-23 16:25:43: pid 7990:LOG:  setting the local watchdog node name to "SERVER4P:5433 Linux SERVER4P"
Aug 23 16:25:43 SERVER4P pgpool[7990]: [2-1] 2018-08-23 16:25:43: pid 7990:LOG:  watchdog cluster is configured with 1 remote nodes
Aug 23 16:25:43 SERVER4P pgpool[7990]: [3-1] 2018-08-23 16:25:43: pid 7990:LOG:  watchdog remote node:0 on SERVER3P:9000
Aug 23 16:25:43 SERVER4P pgpool[7990]: [4-1] 2018-08-23 16:25:43: pid 7990:LOG:  interface monitoring is disabled in watchdog
Aug 23 16:25:43 SERVER4P pgpool[7990]: [5-1] 2018-08-23 16:25:43: pid 7990:LOG:  watchdog node state changed from [DEAD] to [LOADING]
Aug 23 16:25:48 SERVER4P pgpool[7990]: [6-1] 2018-08-23 16:25:48: pid 7990:LOG:  watchdog node state changed from [LOADING] to [JOINING]
Aug 23 16:25:52 SERVER4P pgpool[7990]: [7-1] 2018-08-23 16:25:52: pid 7990:LOG:  watchdog node state changed from [JOINING] to [INITIALIZING]
Aug 23 16:25:53 SERVER4P pgpool[7990]: [8-1] 2018-08-23 16:25:53: pid 7990:LOG:  I am the only alive node in the watchdog cluster
Aug 23 16:25:53 SERVER4P pgpool[7990]: [8-2] 2018-08-23 16:25:53: pid 7990:HINT:  skipping stand for coordinator state
Aug 23 16:25:53 SERVER4P pgpool[7990]: [9-1] 2018-08-23 16:25:53: pid 7990:LOG:  watchdog node state changed from [INITIALIZING] to [MASTER]
Aug 23 16:25:53 SERVER4P pgpool[7990]: [10-1] 2018-08-23 16:25:53: pid 7990:LOG:  I am announcing my self as master/coordinator watchdog node
Aug 23 16:25:57 SERVER4P pgpool[7990]: [11-1] 2018-08-23 16:25:57: pid 7990:LOG:  I am the cluster leader node
Aug 23 16:25:57 SERVER4P pgpool[7990]: [11-2] 2018-08-23 16:25:57: pid 7990:DETAIL:  our declare coordinator message is accepted by all nodes
Aug 23 16:25:57 SERVER4P pgpool[7990]: [12-1] 2018-08-23 16:25:57: pid 7990:LOG:  setting the local node "SERVER4P:5433 Linux SERVER4P" as watchdog cluster master
Aug 23 16:25:57 SERVER4P pgpool[7990]: [13-1] 2018-08-23 16:25:57: pid 7990:LOG:  I am the cluster leader node. Starting escalation process
Aug 23 16:25:57 SERVER4P pgpool[7982]: [2-1] 2018-08-23 16:25:57: pid 7982:LOG:  watchdog process is initialized
Aug 23 16:25:57 SERVER4P pgpool[7990]: [14-1] 2018-08-23 16:25:57: pid 7990:LOG:  escalation process started with PID:8104
Aug 23 16:25:57 SERVER4P pgpool[7990]: [15-1] 2018-08-23 16:25:57: pid 7990:LOG:  new IPC connection received
Aug 23 16:25:57 SERVER4P pgpool[8104]: [14-1] 2018-08-23 16:25:57: pid 8104:LOG:  watchdog: escalation started
Aug 23 16:25:57 SERVER4P pgpool[7990]: [16-1] 2018-08-23 16:25:57: pid 7990:LOG:  new IPC connection received
Aug 23 16:25:57 SERVER4P pgpool[7982]: [3-1] 2018-08-23 16:25:57: pid 7982:LOG:  Setting up socket for 0.0.0.0:5433
Aug 23 16:25:57 SERVER4P pgpool[7982]: [4-1] 2018-08-23 16:25:57: pid 7982:LOG:  Setting up socket for :::5433
Aug 23 16:25:57 SERVER4P pgpool[8103]: [3-1] 2018-08-23 16:25:57: pid 8103:LOG:  2 watchdog nodes are configured for lifecheck
Aug 23 16:25:57 SERVER4P pgpool[8103]: [4-1] 2018-08-23 16:25:57: pid 8103:LOG:  watchdog nodes ID:0 Name:"SERVER4P:5433 Linux SERVER4P"
Aug 23 16:25:57 SERVER4P pgpool[8103]: [4-2] 2018-08-23 16:25:57: pid 8103:DETAIL:  Host:"SERVER4P" WD Port:9000 pgpool-II port:5433
Aug 23 16:25:57 SERVER4P pgpool[8103]: [5-1] 2018-08-23 16:25:57: pid 8103:LOG:  watchdog nodes ID:1 Name:"Not_Set"
Aug 23 16:25:57 SERVER4P pgpool[8103]: [5-2] 2018-08-23 16:25:57: pid 8103:DETAIL:  Host:"SERVER3P" WD Port:9000 pgpool-II port:9999
Aug 23 16:25:57 SERVER4P pgpool[7982]: [5-1] 2018-08-23 16:25:57: pid 7982:LOG:  find_primary_node_repeatedly: waiting for finding a primary node
Aug 23 16:25:57 SERVER4P pgpool[7982]: [6-1] 2018-08-23 16:25:57: pid 7982:LOG:  find_primary_node: checking backend no 0
Aug 23 16:25:57 SERVER4P pgpool[7982]: [7-1] 2018-08-23 16:25:57: pid 7982:LOG:  find_primary_node: primary node id is 0
Aug 23 16:25:57 SERVER4P pgpool[7982]: [8-1] 2018-08-23 16:25:57: pid 7982:LOG:  pgpool-II successfully started. version 3.7.5 (amefuriboshi)
Aug 23 16:25:58 SERVER4P pgpool[8108]: [6-1] 2018-08-23 16:25:58: pid 8108:LOG:  set SO_REUSEPORT option to the socket
Aug 23 16:25:58 SERVER4P pgpool[8106]: [6-1] 2018-08-23 16:25:58: pid 8106:LOG:  set SO_REUSEPORT option to the socket
Aug 23 16:25:58 SERVER4P pgpool[8108]: [7-1] 2018-08-23 16:25:58: pid 8108:LOG:  creating socket for sending heartbeat
Aug 23 16:25:58 SERVER4P pgpool[8108]: [7-2] 2018-08-23 16:25:58: pid 8108:DETAIL:  set SO_REUSEPORT
Aug 23 16:25:58 SERVER4P pgpool[8106]: [7-1] 2018-08-23 16:25:58: pid 8106:LOG:  creating watchdog heartbeat receive socket.
Aug 23 16:25:58 SERVER4P pgpool[8106]: [7-2] 2018-08-23 16:25:58: pid 8106:DETAIL:  set SO_REUSEPORT
Aug 23 16:26:01 SERVER4P pgpool[8104]: [15-1] 2018-08-23 16:26:01: pid 8104:LOG:  successfully acquired the delegate IP:"10.2.65.88"
Aug 23 16:26:01 SERVER4P pgpool[8104]: [15-2] 2018-08-23 16:26:01: pid 8104:DETAIL:  'if_up_cmd' returned with success
Aug 23 16:26:01 SERVER4P pgpool[7990]: [17-1] 2018-08-23 16:26:01: pid 7990:LOG:  watchdog escalation process with pid: 8104 exit with SUCCESS.
Aug 23 16:51:11 SERVER4P pgpool[7990]: [18-1] 2018-08-23 16:51:11: pid 7990:LOG:  new watchdog node connection is received from "10.2.65.86:65199"
Aug 23 16:51:11 SERVER4P pgpool[7990]: [19-1] 2018-08-23 16:51:11: pid 7990:LOG:  new watchdog node connection is received from "10.2.65.86:176"

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20180823/85ad224f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pgpool_Server3P.conf
Type: application/octet-stream
Size: 35489 bytes
Desc: pgpool_Server3P.conf
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20180823/85ad224f/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pgpool_Server4P.conf
Type: application/octet-stream
Size: 35486 bytes
Desc: pgpool_Server4P.conf
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20180823/85ad224f/attachment-0003.obj>