View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000562 | pgpool-HA | General | public | 2019-12-05 05:31 | 2020-04-14 15:42 |
| Reporter | mnieva | Assigned To | hoshiai | ||
| Priority | normal | Severity | major | Reproducibility | always |
| Status | feedback | Resolution | open | ||
| Summary | 0000562: Add to watchdog cluster request is rejected by node 9000 | ||||
| Description | I have 2 db servers with repmgr set up for replication. pgpool is installed in both servers and is configured. I followed the steps in https://www.pgpool.net/pgpool-web/contrib_docs/watchdog_master_slave/en.html#thisis. My problem is when pgpool is started on the 2 host, I am getting this. 2019-12-04 09:43:01: db: [No Connection] pid 23568: user: [No Connection] client [No Connection]: 16 FATAL: Add to watchdog cluster request is rejected by node "pgsql01-lcp-prd.vz.points.com:9000" 2019-12-04 09:43:01: db: [No Connection] pid 23568: user: [No Connection] client [No Connection]: 17 HINT: check the watchdog configurations. 2019-12-04 09:43:01: db: [No Connection] pid 23568: user: [No Connection] client [No Connection]: 18 LOG: Watchdog is shutting down 2019-12-04 09:43:01: db: [No Connection] pid 23566: user: [No Connection] client [No Connection]: 17 DEBUG: reaper handler 2019-12-04 09:43:01: db: [No Connection] pid 23566: user: [No Connection] client [No Connection]: 18 DEBUG: watchdog child process with pid: 23568 exit with FATAL ERROR. pgpool-II will be shutdown 2019-12-04 09:43:01: db: [No Connection] pid 23566: user: [No Connection] client [No Connection]: 19 LOG: watchdog child process with pid: 23568 exits with status 768 2019-12-04 09:43:01: db: [No Connection] pid 23566: user: [No Connection] client [No Connection]: 20 FATAL: watchdog child process exit with fatal error. exiting pgpool-II 2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 1 LOG: setting the local watchdog node name to "pgsql02-lcp-prd.vz.points.com:9999 Linux pgsql02-lcp-prd.vz.points.com" 2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 2 LOG: watchdog cluster is configured with 1 remote nodes 2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 3 LOG: watchdog remote node:0 on pgsql01-lcp-prd.vz.points.com:9000 2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 4 LOG: interface monitoring is disabled in watchdog 2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 5 INFO: IPC socket path: "/home/postgres/config/9999/run/.s.PGPOOLWD_CMD.9000" 2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 6 LOG: watchdog node state changed from [DEAD] to [LOADING] 2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 7 DEBUG: STATE MACHINE INVOKED WITH EVENT = STATE CHANGED Current State = LOADING 2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 8 LOG: new outbound connection to pgsql01-lcp-prd.vz.points.com:9000 2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 9 DEBUG: STATE MACHINE INVOKED WITH EVENT = NEW OUTBOUND_CONNECTION Current State = LOADING 2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 10 DEBUG: sending packet, watchdog node:[] command id:[4] type:[ADD NODE] state:[LOADING] 2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 11 DEBUG: sending watchdog packet to socket:7, type:[A], command ID:4, data Length:383 2019-12-04 09:43:01: db: [No Connection] pid 23569: user: [No Connection] client [No Connection]: 12 LOG: Watchdog is shutting down Not so sure how to resolve this issue. I am unsure on which configuration did I do wrong. Hope you can help me on this. | ||||
| Steps To Reproduce | start pgpool on host 1 start pgpool on host2 pgpool on host2 is shutdown | ||||
| Additional Information | I am using postgres 11.6 repmgr 5.0 pgpool 4.1 I am attaching the pgpool.conf and the pgpool logs | ||||
| Tags | No tags attached. | ||||
|
|
|
|
|
The config paramter is wrong. your config fie set : other_pgpool_port0 = 5432 port = 9999 port number is non-match. If pgpool(on host 1) use port 9999, pgpool(on host 2) should set other_pgpool_port0 to 9999: |
|
|
Hi, thanks for your help. I have updated he other_pgpool_port and now able to start up both pgpool. However, pgpool is unable to do the failover. I started both pgpool pool. Reboot the master host. The db slave was not promoted to the master. When I do not use watchdog, and start only one pgpool, the slave gets promoted. Failover happens. But when watchdog is on and both pgpool is started, the failover is not done. I am attaching the pgpool logs for the slave pgpool. Am I missing anything on the configuration? |
|
|
Ok, I wll check your log later. This ticket mistake project name(pgpool-HA). Actually, 'Pgpool-II' is correct. So, could you re-create ticket with correct project? |
|
|
> This ticket mistake project name(pgpool-HA). Actually, 'Pgpool-II' is correct. > So, could you re-create ticket with correct project? Could you create new issue which it is same content on Pgpool-II project? > I am attaching the pgpool logs for the slave pgpool. Am I missing anything on the configuration? both Pgpools was not communicated yet. I think that you sould set 'haertbeat_device0' to value other than empty. |
|
|
It's been more than a month since I replay to you ... Is your problem solved? If you don't have any problems, I would like to close this issue. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2019-12-05 05:31 | mnieva | New Issue | |
| 2019-12-05 05:31 | mnieva | File Added: pgpool server 2.zip | |
| 2019-12-05 05:31 | mnieva | File Added: pgpool server 1.zip | |
| 2019-12-05 09:55 | hoshiai | Assigned To | => hoshiai |
| 2019-12-05 09:55 | hoshiai | Status | new => assigned |
| 2019-12-05 13:06 | hoshiai | Status | assigned => feedback |
| 2019-12-05 13:06 | hoshiai | Note Added: 0003001 | |
| 2019-12-07 05:37 | mnieva | File Added: pgpool.log | |
| 2019-12-07 05:37 | mnieva | Note Added: 0003012 | |
| 2019-12-07 05:37 | mnieva | Status | feedback => assigned |
| 2019-12-10 14:05 | hoshiai | Status | assigned => feedback |
| 2019-12-10 14:05 | hoshiai | Note Added: 0003017 | |
| 2020-02-05 09:39 | hoshiai | Note Added: 0003132 | |
| 2020-04-14 15:42 | hoshiai | Note Added: 0003323 |
