[pgpool-general: 7828] Re: pgpool-general Digest, Vol 120, Issue 45

[Nirav Bhatt]

I ran sudo ufw status and firewall is already inactive.

Not sure what firewall-cmd did for all the pgpool ports. (they are showing
as list of open ports)

Should add them all via ufw also?

-Nirav

On Mon, Oct 25, 2021 at 7:46 PM Luca Maranzano <liuk001 at gmail.com> wrote:

> Try to disable firewall to see it it's related, all the pgpool
> configuration files are identical on the 3 nodes?
>
> On Mon, Oct 25, 2021 at 6:41 PM Nirav Bhatt <niravbhatt.cpp at gmail.com>
> wrote:
>
>> Hello,
>>
>> Thanks for the response.
>>
>> I restarted pgpool2 service on all 3 nodes (with some delay, and in the
>> order: primary, secondary1, secondary2).
>> My primary pgpool2 service logs do show
>>
>> LOG:  new watchdog node connection is received from <secondary host>
>>
>> But my secondary pgpool2 service logs (among other normal messages) show:
>>
>> FATAL:  Add to watchdog cluster request is rejected by node "server1:9000"
>> FATAL:  watchdog child process exit with fatal error. exiting pgpool-II
>> LOG:  watchdog node state changed from [DEAD] to [LOADING]
>> LOG:  new outbound connection to server1:9000
>> FATAL:  Add to watchdog cluster request is rejected by node "server1:9000"
>> HINT:  check the watchdog configurations.
>> LOG:  Watchdog is shutting down
>>
>> I checked that server1 has 9000 port open, along with 9999, 9898 and 9694:
>>
>> > firewall-cmd --list-ports
>> 9999/tcp 9898/tcp 9000/tcp 9694/udp
>>
>> What could be going on?
>>
>> -Nirav
>>
>>
>> On Mon, Oct 25, 2021 at 7:13 PM Luca Maranzano <liuk001 at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> if you cannot connect to 20.20.20.20 -p 9999 it means that hte PGPool
>>> cluster is not well formed and that the VIP has not been enabled.
>>>
>>> You can check that with pcp_watchdog_info.
>>> The logs from pgpool should also help to diagnose the problem.
>>>
>>> HTH
>>> Regards
>>> Luca
>>>
>>>
>>> ---------- Forwarded message ---------
>>> From: <pgpool-general-request at pgpool.net>
>>> Date: Mon, Oct 25, 2021 at 5:00 AM
>>> Subject: pgpool-general Digest, Vol 120, Issue 45
>>> To: <pgpool-general at pgpool.net>
>>>
>>>
>>> Send pgpool-general mailing list submissions to
>>>         pgpool-general at pgpool.net
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>         https://www.pgpool.net/mailman/listinfo/pgpool-general
>>> or, via email, send a message with subject or body 'help' to
>>>         pgpool-general-request at pgpool.net
>>>
>>> You can reach the person managing the list at
>>>         pgpool-general-owner at pgpool.net
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of pgpool-general digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>>    1. [pgpool-general: 7822] Pgpool2 example with watchdog: Unable
>>>       to test (Nirav Bhatt)
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Sun, 24 Oct 2021 20:18:28 +0300
>>> From: Nirav Bhatt <niravbhatt.cpp at gmail.com>
>>> To: pgpool-general at pgpool.net
>>> Subject: [pgpool-general: 7822] Pgpool2 example with watchdog: Unable
>>>         to test
>>> Message-ID:
>>>         <CABz_77PMu9ksuYM2QGBaAuf=m_edGoKcidmWqC=K7Rpr=
>>> J_ZGw at mail.gmail.com>
>>> Content-Type: text/plain; charset="utf-8"
>>>
>>> After doing all the config (config files, passwordless ssh, firewall-cmd
>>> etc) as given in the following link, I am unable to test my setup. My OS
>>> is
>>> Ubuntu 20.04.3 LTS.
>>>
>>>
>>> https://www.pgpool.net/docs/41/en/html/example-cluster.html#EXAMPLE-CLUSTER-STRUCTURE
>>>
>>> All the port numbers are as listed in the example.
>>>
>>> The part in example section: 8.3.7.1 and onwards is not working.
>>>
>>> When I run on server1 (my primary):
>>> psql -h 20.20.20.20 -p 9999 -U pgpool postgres -c "show pool_nodes"
>>>
>>> I got error:
>>>
>>> psql: error: connection to server at "20.20.20.20", port 9999 failed:
>>> Connection refused
>>> Is the server running on that host and accepting TCP/IP connections?
>>>
>>> I restarted postgresql and pgpool2 services time and again.
>>>
>>> pgpool2 service most recent logs:
>>>
>>> server1 (current primary):
>>> Oct 24 17:01:41 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:01:41: pid
>>> 104691: LOG:  new watchdog node connection is received from "<server1
>>> host>:16553"
>>> Oct 24 17:01:41 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:01:41: pid
>>> 104691: LOG:  new watchdog node connection is received from "<server1
>>> host>:17577"
>>> Oct 24 17:04:02 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:04:02: pid
>>> 104691: LOG:  new watchdog node connection is received from "<server2
>>> host>:49304"
>>> Oct 24 17:04:02 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:04:02: pid
>>> 104691: LOG:  new watchdog node connection is received from "<server2
>>> host>:50328"
>>> Oct 24 17:11:12 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:11:12: pid
>>> 104691: LOG:  new watchdog node connection is received from "<server3
>>> host>:51864"
>>> Oct 24 17:11:12 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:11:12: pid
>>> 104691: LOG:  new watchdog node connection is received from "<server3
>>> host>:52888"
>>>
>>> server2 and server3 (looks like pgpool2 service is failing):
>>>
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>>> 125018: LOG:  setting the local watchdog node name to "server3:5433 Linux
>>> ip-<server 3 host>"
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>>> 125018: LOG:  watchdog cluster is configured with 2 remote nodes
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>>> 125018: LOG:  watchdog remote node:0 on server1:9000
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>>> 125018: LOG:  watchdog remote node:1 on server2:9000
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>>> 125018: LOG:  interface monitoring is disabled in watchdog
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>>> 125018: LOG:  watchdog node state changed from [DEAD] to [LOADING]
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>>> 125018: LOG:  new outbound connection to server1:9000
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>>> 125018: FATAL:  Add to watchdog cluster request is rejected by node
>>> "server1:9000"
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>>> 125018: HINT:  check the watchdog configurations.
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>>> 125018: LOG:  Watchdog is shutting down
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125014]: 2021-10-24 17:11:12: pid
>>> 125014: LOG:  watchdog child process with pid: 125018 exits with status
>>> 768
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125014]: 2021-10-24 17:11:12: pid
>>> 125014: FATAL:  watchdog child process exit with fatal error. exiting
>>> pgpool-II
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
>>> 125019: LOG:  setting the local watchdog node name to "server3:5433 Linux
>>> ip-<server 3 host>"
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
>>> 125019: LOG:  watchdog cluster is configured with 2 remote nodes
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
>>> 125019: LOG:  watchdog remote node:0 on server1:9000
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
>>> 125019: LOG:  watchdog remote node:1 on server2:9000
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
>>> 125019: LOG:  interface monitoring is disabled in watchdog
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
>>> 125019: LOG:  watchdog node state changed from [DEAD] to [LOADING]
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
>>> 125019: LOG:  new outbound connection to server1:9000
>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
>>> 125019: LOG:  Watchdog is shutting down
>>> Oct 24 17:11:12 ip-10-192-12-30 systemd[1]: pgpool2.service: Main process
>>> exited, code=exited, status=3/NOTIMPLEMENTED
>>> Oct 24 17:11:12 ip-10-192-12-30 systemd[1]: pgpool2.service: Failed with
>>> result 'exit-code'.
>>>
>>> What steps am I missing?
>>>
>>> Thank you for the assistance,
>>> -Nirav
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL: <
>>> http://www.pgpool.net/pipermail/pgpool-general/attachments/20211024/98ab6d68/attachment-0001.htm
>>> >
>>>
>>> ------------------------------
>>>
>>> Subject: Digest Footer
>>>
>>> _______________________________________________
>>> pgpool-general mailing list
>>> pgpool-general at pgpool.net
>>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>>>
>>>
>>> ------------------------------
>>>
>>> End of pgpool-general Digest, Vol 120, Issue 45
>>> ***********************************************
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20211025/1c29bb98/attachment.htm>