[pgpool-general: 7822] Pgpool2 example with watchdog: Unable to test

[Nirav Bhatt]

After doing all the config (config files, passwordless ssh, firewall-cmd
etc) as given in the following link, I am unable to test my setup. My OS is
Ubuntu 20.04.3 LTS.

https://www.pgpool.net/docs/41/en/html/example-cluster.html#EXAMPLE-CLUSTER-STRUCTURE

All the port numbers are as listed in the example.

The part in example section: 8.3.7.1 and onwards is not working.

When I run on server1 (my primary):
psql -h 20.20.20.20 -p 9999 -U pgpool postgres -c "show pool_nodes"

I got error:

psql: error: connection to server at "20.20.20.20", port 9999 failed:
Connection refused
Is the server running on that host and accepting TCP/IP connections?

I restarted postgresql and pgpool2 services time and again.

pgpool2 service most recent logs:

server1 (current primary):
Oct 24 17:01:41 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:01:41: pid
104691: LOG:  new watchdog node connection is received from "<server1
host>:16553"
Oct 24 17:01:41 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:01:41: pid
104691: LOG:  new watchdog node connection is received from "<server1
host>:17577"
Oct 24 17:04:02 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:04:02: pid
104691: LOG:  new watchdog node connection is received from "<server2
host>:49304"
Oct 24 17:04:02 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:04:02: pid
104691: LOG:  new watchdog node connection is received from "<server2
host>:50328"
Oct 24 17:11:12 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:11:12: pid
104691: LOG:  new watchdog node connection is received from "<server3
host>:51864"
Oct 24 17:11:12 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:11:12: pid
104691: LOG:  new watchdog node connection is received from "<server3
host>:52888"

server2 and server3 (looks like pgpool2 service is failing):

Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
125018: LOG:  setting the local watchdog node name to "server3:5433 Linux
ip-<server 3 host>"
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
125018: LOG:  watchdog cluster is configured with 2 remote nodes
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
125018: LOG:  watchdog remote node:0 on server1:9000
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
125018: LOG:  watchdog remote node:1 on server2:9000
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
125018: LOG:  interface monitoring is disabled in watchdog
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
125018: LOG:  watchdog node state changed from [DEAD] to [LOADING]
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
125018: LOG:  new outbound connection to server1:9000
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
125018: FATAL:  Add to watchdog cluster request is rejected by node
"server1:9000"
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
125018: HINT:  check the watchdog configurations.
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
125018: LOG:  Watchdog is shutting down
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125014]: 2021-10-24 17:11:12: pid
125014: LOG:  watchdog child process with pid: 125018 exits with status 768
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125014]: 2021-10-24 17:11:12: pid
125014: FATAL:  watchdog child process exit with fatal error. exiting
pgpool-II
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
125019: LOG:  setting the local watchdog node name to "server3:5433 Linux
ip-<server 3 host>"
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
125019: LOG:  watchdog cluster is configured with 2 remote nodes
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
125019: LOG:  watchdog remote node:0 on server1:9000
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
125019: LOG:  watchdog remote node:1 on server2:9000
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
125019: LOG:  interface monitoring is disabled in watchdog
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
125019: LOG:  watchdog node state changed from [DEAD] to [LOADING]
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
125019: LOG:  new outbound connection to server1:9000
Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
125019: LOG:  Watchdog is shutting down
Oct 24 17:11:12 ip-10-192-12-30 systemd[1]: pgpool2.service: Main process
exited, code=exited, status=3/NOTIMPLEMENTED
Oct 24 17:11:12 ip-10-192-12-30 systemd[1]: pgpool2.service: Failed with
result 'exit-code'.

What steps am I missing?

Thank you for the assistance,
-Nirav
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20211024/98ab6d68/attachment.htm>