View Issue Details

IDProjectCategoryView StatusLast Update
0000562pgpool-HA[All Projects] Generalpublic2020-04-14 15:42
ReportermnievaAssigned Tohoshiai 
PrioritynormalSeveritymajorReproducibilityalways
Status feedbackResolutionopen 
Product Version 
Target VersionFixed in Version 
Summary0000562: Add to watchdog cluster request is rejected by node 9000
DescriptionI have 2 db servers with repmgr set up for replication. pgpool is installed in both servers and is configured. I followed the steps in https://www.pgpool.net/pgpool-web/contrib_docs/watchdog_master_slave/en.html#thisis.

My problem is when pgpool is started on the 2 host, I am getting this.

2019-12-04 09:43:01: db: [No Connection] pid 23568: user: [No Connection] client [No Connection]: 16 FATAL: Add to watchdog cluster request is rejected by node "pgsql01-lcp-prd.vz.points.com:9000"
2019-12-04 09:43:01: db: [No Connection] pid 23568: user: [No Connection] client [No Connection]: 17 HINT: check the watchdog configurations.
2019-12-04 09:43:01: db: [No Connection] pid 23568: user: [No Connection] client [No Connection]: 18 LOG: Watchdog is shutting down
2019-12-04 09:43:01: db: [No Connection] pid 23566: user: [No Connection] client [No Connection]: 17 DEBUG: reaper handler
2019-12-04 09:43:01: db: [No Connection] pid 23566: user: [No Connection] client [No Connection]: 18 DEBUG: watchdog child process with pid: 23568 exit with FATAL ERROR. pgpool-II will be shutdown
2019-12-04 09:43:01: db: [No Connection] pid 23566: user: [No Connection] client [No Connection]: 19 LOG: watchdog child process with pid: 23568 exits with status 768
2019-12-04 09:43:01: db: [No Connection] pid 23566: user: [No Connection] client [No Connection]: 20 FATAL: watchdog child process exit with fatal error. exiting pgpool-II
2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 1 LOG: setting the local watchdog node name to "pgsql02-lcp-prd.vz.points.com:9999 Linux pgsql02-lcp-prd.vz.points.com"
2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 2 LOG: watchdog cluster is configured with 1 remote nodes
2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 3 LOG: watchdog remote node:0 on pgsql01-lcp-prd.vz.points.com:9000
2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 4 LOG: interface monitoring is disabled in watchdog
2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 5 INFO: IPC socket path: "/home/postgres/config/9999/run/.s.PGPOOLWD_CMD.9000"
2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 6 LOG: watchdog node state changed from [DEAD] to [LOADING]
2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 7 DEBUG: STATE MACHINE INVOKED WITH EVENT = STATE CHANGED Current State = LOADING
2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 8 LOG: new outbound connection to pgsql01-lcp-prd.vz.points.com:9000
2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 9 DEBUG: STATE MACHINE INVOKED WITH EVENT = NEW OUTBOUND_CONNECTION Current State = LOADING
2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 10 DEBUG: sending packet, watchdog node:[] command id:[4] type:[ADD NODE] state:[LOADING]
2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 11 DEBUG: sending watchdog packet to socket:7, type:[A], command ID:4, data Length:383
2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 12 LOG: Watchdog is shutting down


Not so sure how to resolve this issue. I am unsure on which configuration did I do wrong. Hope you can help me on this.
Steps To Reproducestart pgpool on host 1
start pgpool on host2

pgpool on host2 is shutdown
Additional InformationI am using postgres 11.6
repmgr 5.0
pgpool 4.1

I am attaching the pgpool.conf and the pgpool logs
TagsNo tags attached.

Activities

mnieva

2019-12-05 05:31

reporter  

pgpool server 1.zip (18,882 bytes)
pgpool server 2.zip (10,153 bytes)

hoshiai

2019-12-05 13:06

developer   ~0003001

The config paramter is wrong.
your config fie set :
  other_pgpool_port0 = 5432
  port = 9999

port number is non-match. If pgpool(on host 1) use port 9999, pgpool(on host 2) should set other_pgpool_port0 to 9999:

mnieva

2019-12-07 05:37

reporter   ~0003012

Hi, thanks for your help. I have updated he other_pgpool_port and now able to start up both pgpool.

However, pgpool is unable to do the failover. I started both pgpool pool. Reboot the master host. The db slave was not promoted to the master.
When I do not use watchdog, and start only one pgpool, the slave gets promoted. Failover happens. But when watchdog is on and both pgpool is started, the failover is not done.

I am attaching the pgpool logs for the slave pgpool. Am I missing anything on the configuration?

pgpool.log (363,466 bytes)

hoshiai

2019-12-10 14:05

developer   ~0003017

Ok, I wll check your log later.

This ticket mistake project name(pgpool-HA). Actually, 'Pgpool-II' is correct.
So, could you re-create ticket with correct project?

hoshiai

2020-02-05 09:39

developer   ~0003132

> This ticket mistake project name(pgpool-HA). Actually, 'Pgpool-II' is correct.
> So, could you re-create ticket with correct project?

Could you create new issue which it is same content on Pgpool-II project?

> I am attaching the pgpool logs for the slave pgpool. Am I missing anything on the configuration?

both Pgpools was not communicated yet.
I think that you sould set 'haertbeat_device0' to value other than empty.

hoshiai

2020-04-14 15:42

developer   ~0003323

It's been more than a month since I replay to you ... Is your problem solved?
If you don't have any problems, I would like to close this issue.

Issue History

Date Modified Username Field Change
2019-12-05 05:31 mnieva New Issue
2019-12-05 05:31 mnieva File Added: pgpool server 2.zip
2019-12-05 05:31 mnieva File Added: pgpool server 1.zip
2019-12-05 09:55 hoshiai Assigned To => hoshiai
2019-12-05 09:55 hoshiai Status new => assigned
2019-12-05 13:06 hoshiai Status assigned => feedback
2019-12-05 13:06 hoshiai Note Added: 0003001
2019-12-07 05:37 mnieva File Added: pgpool.log
2019-12-07 05:37 mnieva Note Added: 0003012
2019-12-07 05:37 mnieva Status feedback => assigned
2019-12-10 14:05 hoshiai Status assigned => feedback
2019-12-10 14:05 hoshiai Note Added: 0003017
2020-02-05 09:39 hoshiai Note Added: 0003132
2020-04-14 15:42 hoshiai Note Added: 0003323