[pgpool-general: 7830] Re: pgpool-general Digest, Vol 120, Issue 45
Nirav Bhatt
niravbhatt.cpp at gmail.com
Tue Oct 26 04:46:35 JST 2021
I added all ports via ufw on server1: 22,9999,9898,9000,9694,5433,5432
I enabled firewall past that point.
I diffed all conf files across servers - all files are same, including hba
and .pgpass and .pcppass.
Only pgpool.conf files differ, that too in parameters like server hosts for
pgpool and watchdog, no other differences.
No error is observed on primary server1. The last log lines on server1
says: new watchdog node connection is received from <server2> or <server3>
But secondary server pgpool2 keeps exiting with:
Add to watchdog cluster request is rejected by node "server1:9000"
I somehow suspect it has to do with wd_authkey which I leave empty:
https://www.sraoss.jp/pipermail/pgpool-hackers/2017-August/002497.html
But it is an old issue, so I can't say for sure.
-Nirav
On Mon, Oct 25, 2021 at 8:07 PM Luca Maranzano <liuk001 at gmail.com> wrote:
> Firewall-cmd has to be used on Redhat derived flavours, on Ubuntu standard
> is ufw, so "ufw disable" should open all ports
>
> On Mon, Oct 25, 2021 at 6:59 PM Nirav Bhatt <niravbhatt.cpp at gmail.com>
> wrote:
>
>> I ran sudo ufw status and firewall is already inactive.
>>
>> Not sure what firewall-cmd did for all the pgpool ports. (they are
>> showing as list of open ports)
>>
>> Should add them all via ufw also?
>>
>> -Nirav
>>
>> On Mon, Oct 25, 2021 at 7:46 PM Luca Maranzano <liuk001 at gmail.com> wrote:
>>
>>> Try to disable firewall to see it it's related, all the pgpool
>>> configuration files are identical on the 3 nodes?
>>>
>>> On Mon, Oct 25, 2021 at 6:41 PM Nirav Bhatt <niravbhatt.cpp at gmail.com>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> Thanks for the response.
>>>>
>>>> I restarted pgpool2 service on all 3 nodes (with some delay, and in the
>>>> order: primary, secondary1, secondary2).
>>>> My primary pgpool2 service logs do show
>>>>
>>>> LOG: new watchdog node connection is received from <secondary host>
>>>>
>>>> But my secondary pgpool2 service logs (among other normal messages)
>>>> show:
>>>>
>>>> FATAL: Add to watchdog cluster request is rejected by node
>>>> "server1:9000"
>>>> FATAL: watchdog child process exit with fatal error. exiting pgpool-II
>>>> LOG: watchdog node state changed from [DEAD] to [LOADING]
>>>> LOG: new outbound connection to server1:9000
>>>> FATAL: Add to watchdog cluster request is rejected by node
>>>> "server1:9000"
>>>> HINT: check the watchdog configurations.
>>>> LOG: Watchdog is shutting down
>>>>
>>>> I checked that server1 has 9000 port open, along with 9999, 9898 and
>>>> 9694:
>>>>
>>>> > firewall-cmd --list-ports
>>>> 9999/tcp 9898/tcp 9000/tcp 9694/udp
>>>>
>>>> What could be going on?
>>>>
>>>> -Nirav
>>>>
>>>>
>>>> On Mon, Oct 25, 2021 at 7:13 PM Luca Maranzano <liuk001 at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> if you cannot connect to 20.20.20.20 -p 9999 it means that hte PGPool
>>>>> cluster is not well formed and that the VIP has not been enabled.
>>>>>
>>>>> You can check that with pcp_watchdog_info.
>>>>> The logs from pgpool should also help to diagnose the problem.
>>>>>
>>>>> HTH
>>>>> Regards
>>>>> Luca
>>>>>
>>>>>
>>>>> ---------- Forwarded message ---------
>>>>> From: <pgpool-general-request at pgpool.net>
>>>>> Date: Mon, Oct 25, 2021 at 5:00 AM
>>>>> Subject: pgpool-general Digest, Vol 120, Issue 45
>>>>> To: <pgpool-general at pgpool.net>
>>>>>
>>>>>
>>>>> Send pgpool-general mailing list submissions to
>>>>> pgpool-general at pgpool.net
>>>>>
>>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>>> https://www.pgpool.net/mailman/listinfo/pgpool-general
>>>>> or, via email, send a message with subject or body 'help' to
>>>>> pgpool-general-request at pgpool.net
>>>>>
>>>>> You can reach the person managing the list at
>>>>> pgpool-general-owner at pgpool.net
>>>>>
>>>>> When replying, please edit your Subject line so it is more specific
>>>>> than "Re: Contents of pgpool-general digest..."
>>>>>
>>>>>
>>>>> Today's Topics:
>>>>>
>>>>> 1. [pgpool-general: 7822] Pgpool2 example with watchdog: Unable
>>>>> to test (Nirav Bhatt)
>>>>>
>>>>>
>>>>> ----------------------------------------------------------------------
>>>>>
>>>>> Message: 1
>>>>> Date: Sun, 24 Oct 2021 20:18:28 +0300
>>>>> From: Nirav Bhatt <niravbhatt.cpp at gmail.com>
>>>>> To: pgpool-general at pgpool.net
>>>>> Subject: [pgpool-general: 7822] Pgpool2 example with watchdog: Unable
>>>>> to test
>>>>> Message-ID:
>>>>> <CABz_77PMu9ksuYM2QGBaAuf=m_edGoKcidmWqC=K7Rpr=
>>>>> J_ZGw at mail.gmail.com>
>>>>> Content-Type: text/plain; charset="utf-8"
>>>>>
>>>>> After doing all the config (config files, passwordless ssh,
>>>>> firewall-cmd
>>>>> etc) as given in the following link, I am unable to test my setup. My
>>>>> OS is
>>>>> Ubuntu 20.04.3 LTS.
>>>>>
>>>>>
>>>>> https://www.pgpool.net/docs/41/en/html/example-cluster.html#EXAMPLE-CLUSTER-STRUCTURE
>>>>>
>>>>> All the port numbers are as listed in the example.
>>>>>
>>>>> The part in example section: 8.3.7.1 and onwards is not working.
>>>>>
>>>>> When I run on server1 (my primary):
>>>>> psql -h 20.20.20.20 -p 9999 -U pgpool postgres -c "show pool_nodes"
>>>>>
>>>>> I got error:
>>>>>
>>>>> psql: error: connection to server at "20.20.20.20", port 9999 failed:
>>>>> Connection refused
>>>>> Is the server running on that host and accepting TCP/IP connections?
>>>>>
>>>>> I restarted postgresql and pgpool2 services time and again.
>>>>>
>>>>> pgpool2 service most recent logs:
>>>>>
>>>>> server1 (current primary):
>>>>> Oct 24 17:01:41 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:01:41:
>>>>> pid
>>>>> 104691: LOG: new watchdog node connection is received from "<server1
>>>>> host>:16553"
>>>>> Oct 24 17:01:41 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:01:41:
>>>>> pid
>>>>> 104691: LOG: new watchdog node connection is received from "<server1
>>>>> host>:17577"
>>>>> Oct 24 17:04:02 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:04:02:
>>>>> pid
>>>>> 104691: LOG: new watchdog node connection is received from "<server2
>>>>> host>:49304"
>>>>> Oct 24 17:04:02 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:04:02:
>>>>> pid
>>>>> 104691: LOG: new watchdog node connection is received from "<server2
>>>>> host>:50328"
>>>>> Oct 24 17:11:12 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 104691: LOG: new watchdog node connection is received from "<server3
>>>>> host>:51864"
>>>>> Oct 24 17:11:12 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 104691: LOG: new watchdog node connection is received from "<server3
>>>>> host>:52888"
>>>>>
>>>>> server2 and server3 (looks like pgpool2 service is failing):
>>>>>
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125018: LOG: setting the local watchdog node name to "server3:5433
>>>>> Linux
>>>>> ip-<server 3 host>"
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125018: LOG: watchdog cluster is configured with 2 remote nodes
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125018: LOG: watchdog remote node:0 on server1:9000
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125018: LOG: watchdog remote node:1 on server2:9000
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125018: LOG: interface monitoring is disabled in watchdog
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125018: LOG: watchdog node state changed from [DEAD] to [LOADING]
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125018: LOG: new outbound connection to server1:9000
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125018: FATAL: Add to watchdog cluster request is rejected by node
>>>>> "server1:9000"
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125018: HINT: check the watchdog configurations.
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125018: LOG: Watchdog is shutting down
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125014]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125014: LOG: watchdog child process with pid: 125018 exits with
>>>>> status 768
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125014]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125014: FATAL: watchdog child process exit with fatal error. exiting
>>>>> pgpool-II
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125019: LOG: setting the local watchdog node name to "server3:5433
>>>>> Linux
>>>>> ip-<server 3 host>"
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125019: LOG: watchdog cluster is configured with 2 remote nodes
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125019: LOG: watchdog remote node:0 on server1:9000
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125019: LOG: watchdog remote node:1 on server2:9000
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125019: LOG: interface monitoring is disabled in watchdog
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125019: LOG: watchdog node state changed from [DEAD] to [LOADING]
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125019: LOG: new outbound connection to server1:9000
>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12:
>>>>> pid
>>>>> 125019: LOG: Watchdog is shutting down
>>>>> Oct 24 17:11:12 ip-10-192-12-30 systemd[1]: pgpool2.service: Main
>>>>> process
>>>>> exited, code=exited, status=3/NOTIMPLEMENTED
>>>>> Oct 24 17:11:12 ip-10-192-12-30 systemd[1]: pgpool2.service: Failed
>>>>> with
>>>>> result 'exit-code'.
>>>>>
>>>>> What steps am I missing?
>>>>>
>>>>> Thank you for the assistance,
>>>>> -Nirav
>>>>> -------------- next part --------------
>>>>> An HTML attachment was scrubbed...
>>>>> URL: <
>>>>> http://www.pgpool.net/pipermail/pgpool-general/attachments/20211024/98ab6d68/attachment-0001.htm
>>>>> >
>>>>>
>>>>> ------------------------------
>>>>>
>>>>> Subject: Digest Footer
>>>>>
>>>>> _______________________________________________
>>>>> pgpool-general mailing list
>>>>> pgpool-general at pgpool.net
>>>>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>>>>>
>>>>>
>>>>> ------------------------------
>>>>>
>>>>> End of pgpool-general Digest, Vol 120, Issue 45
>>>>> ***********************************************
>>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20211025/af8dfce2/attachment.htm>
More information about the pgpool-general
mailing list