[pgpool-general: 7827] Re: pgpool-general Digest, Vol 120, Issue 45

Nirav Bhatt niravbhatt.cpp at gmail.com
Tue Oct 26 01:57:31 JST 2021 

Hello,

Thanks for the response.

I restarted pgpool2 service on all 3 nodes (with some delay, and in the
order: primary, secondary1, secondary2).
My primary pgpool2 service logs do show

LOG:  new watchdog node connection is received from <secondary host>

But my secondary pgpool2 service logs (among other normal messages) show:

FATAL:  Add to watchdog cluster request is rejected by node "server1:9000"
FATAL:  watchdog child process exit with fatal error. exiting pgpool-II
LOG:  watchdog node state changed from [DEAD] to [LOADING]
LOG:  new outbound connection to server1:9000
FATAL:  Add to watchdog cluster request is rejected by node "server1:9000"
HINT:  check the watchdog configurations.
LOG:  Watchdog is shutting down

I checked that server1 has 9000 port open, along with 9999, 9898 and 9694:

> firewall-cmd --list-ports
9999/tcp 9898/tcp 9000/tcp 9694/udp

What could be going on?

-Nirav



On Mon, Oct 25, 2021 at 7:41 PM Nirav Bhatt <niravbhatt.cpp at gmail.com>
wrote:

> Hello,
>
> Thanks for the response.
>
> I restarted pgpool2 service on all 3 nodes (with some delay, and in the
> order: primary, secondary1, secondary2).
> My primary pgpool2 service logs do show
>
> LOG:  new watchdog node connection is received from <secondary host>
>
> But my secondary pgpool2 service logs (among other normal messages) show:
>
> FATAL:  Add to watchdog cluster request is rejected by node "server1:9000"
> FATAL:  watchdog child process exit with fatal error. exiting pgpool-II
> LOG:  watchdog node state changed from [DEAD] to [LOADING]
> LOG:  new outbound connection to server1:9000
> FATAL:  Add to watchdog cluster request is rejected by node "server1:9000"
> HINT:  check the watchdog configurations.
> LOG:  Watchdog is shutting down
>
> I checked that server1 has 9000 port open, along with 9999, 9898 and 9694:
>
> > firewall-cmd --list-ports
> 9999/tcp 9898/tcp 9000/tcp 9694/udp
>
> What could be going on?
>
> -Nirav
>
>
> On Mon, Oct 25, 2021 at 7:13 PM Luca Maranzano <liuk001 at gmail.com> wrote:
>
>> Hi,
>>
>> if you cannot connect to 20.20.20.20 -p 9999 it means that hte PGPool
>> cluster is not well formed and that the VIP has not been enabled.
>>
>> You can check that with pcp_watchdog_info.
>> The logs from pgpool should also help to diagnose the problem.
>>
>> HTH
>> Regards
>> Luca
>>
>>
>> ---------- Forwarded message ---------
>> From: <pgpool-general-request at pgpool.net>
>> Date: Mon, Oct 25, 2021 at 5:00 AM
>> Subject: pgpool-general Digest, Vol 120, Issue 45
>> To: <pgpool-general at pgpool.net>
>>
>>
>> Send pgpool-general mailing list submissions to
>>         pgpool-general at pgpool.net
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>         https://www.pgpool.net/mailman/listinfo/pgpool-general
>> or, via email, send a message with subject or body 'help' to
>>         pgpool-general-request at pgpool.net
>>
>> You can reach the person managing the list at
>>         pgpool-general-owner at pgpool.net
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of pgpool-general digest..."
>>
>>
>> Today's Topics:
>>
>>    1. [pgpool-general: 7822] Pgpool2 example with watchdog: Unable
>>       to test (Nirav Bhatt)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Sun, 24 Oct 2021 20:18:28 +0300
>> From: Nirav Bhatt <niravbhatt.cpp at gmail.com>
>> To: pgpool-general at pgpool.net
>> Subject: [pgpool-general: 7822] Pgpool2 example with watchdog: Unable
>>         to test
>> Message-ID:
>>         <CABz_77PMu9ksuYM2QGBaAuf=m_edGoKcidmWqC=K7Rpr=
>> J_ZGw at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> After doing all the config (config files, passwordless ssh, firewall-cmd
>> etc) as given in the following link, I am unable to test my setup. My OS
>> is
>> Ubuntu 20.04.3 LTS.
>>
>>
>> https://www.pgpool.net/docs/41/en/html/example-cluster.html#EXAMPLE-CLUSTER-STRUCTURE
>>
>> All the port numbers are as listed in the example.
>>
>> The part in example section: 8.3.7.1 and onwards is not working.
>>
>> When I run on server1 (my primary):
>> psql -h 20.20.20.20 -p 9999 -U pgpool postgres -c "show pool_nodes"
>>
>> I got error:
>>
>> psql: error: connection to server at "20.20.20.20", port 9999 failed:
>> Connection refused
>> Is the server running on that host and accepting TCP/IP connections?
>>
>> I restarted postgresql and pgpool2 services time and again.
>>
>> pgpool2 service most recent logs:
>>
>> server1 (current primary):
>> Oct 24 17:01:41 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:01:41: pid
>> 104691: LOG:  new watchdog node connection is received from "<server1
>> host>:16553"
>> Oct 24 17:01:41 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:01:41: pid
>> 104691: LOG:  new watchdog node connection is received from "<server1
>> host>:17577"
>> Oct 24 17:04:02 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:04:02: pid
>> 104691: LOG:  new watchdog node connection is received from "<server2
>> host>:49304"
>> Oct 24 17:04:02 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:04:02: pid
>> 104691: LOG:  new watchdog node connection is received from "<server2
>> host>:50328"
>> Oct 24 17:11:12 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:11:12: pid
>> 104691: LOG:  new watchdog node connection is received from "<server3
>> host>:51864"
>> Oct 24 17:11:12 ip-10-192-10-10 pgpool[104691]: 2021-10-24 17:11:12: pid
>> 104691: LOG:  new watchdog node connection is received from "<server3
>> host>:52888"
>>
>> server2 and server3 (looks like pgpool2 service is failing):
>>
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>> 125018: LOG:  setting the local watchdog node name to "server3:5433 Linux
>> ip-<server 3 host>"
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>> 125018: LOG:  watchdog cluster is configured with 2 remote nodes
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>> 125018: LOG:  watchdog remote node:0 on server1:9000
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>> 125018: LOG:  watchdog remote node:1 on server2:9000
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>> 125018: LOG:  interface monitoring is disabled in watchdog
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>> 125018: LOG:  watchdog node state changed from [DEAD] to [LOADING]
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>> 125018: LOG:  new outbound connection to server1:9000
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>> 125018: FATAL:  Add to watchdog cluster request is rejected by node
>> "server1:9000"
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>> 125018: HINT:  check the watchdog configurations.
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24 17:11:12: pid
>> 125018: LOG:  Watchdog is shutting down
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125014]: 2021-10-24 17:11:12: pid
>> 125014: LOG:  watchdog child process with pid: 125018 exits with status
>> 768
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125014]: 2021-10-24 17:11:12: pid
>> 125014: FATAL:  watchdog child process exit with fatal error. exiting
>> pgpool-II
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
>> 125019: LOG:  setting the local watchdog node name to "server3:5433 Linux
>> ip-<server 3 host>"
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
>> 125019: LOG:  watchdog cluster is configured with 2 remote nodes
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
>> 125019: LOG:  watchdog remote node:0 on server1:9000
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
>> 125019: LOG:  watchdog remote node:1 on server2:9000
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
>> 125019: LOG:  interface monitoring is disabled in watchdog
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
>> 125019: LOG:  watchdog node state changed from [DEAD] to [LOADING]
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
>> 125019: LOG:  new outbound connection to server1:9000
>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24 17:11:12: pid
>> 125019: LOG:  Watchdog is shutting down
>> Oct 24 17:11:12 ip-10-192-12-30 systemd[1]: pgpool2.service: Main process
>> exited, code=exited, status=3/NOTIMPLEMENTED
>> Oct 24 17:11:12 ip-10-192-12-30 systemd[1]: pgpool2.service: Failed with
>> result 'exit-code'.
>>
>> What steps am I missing?
>>
>> Thank you for the assistance,
>> -Nirav
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://www.pgpool.net/pipermail/pgpool-general/attachments/20211024/98ab6d68/attachment-0001.htm
>> >
>>
>> ------------------------------
>>
>> Subject: Digest Footer
>>
>> _______________________________________________
>> pgpool-general mailing list
>> pgpool-general at pgpool.net
>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>>
>>
>> ------------------------------
>>
>> End of pgpool-general Digest, Vol 120, Issue 45
>> ***********************************************
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20211025/e823b8a3/attachment-0001.htm>

More information about the pgpool-general mailing list