[pgpool-general: 7839] Re: pgpool-general Digest, Vol 120, Issue 45

Nirav Bhatt niravbhatt.cpp at gmail.com
Sun Oct 31 05:26:21 JST 2021 

Hi Luca

Thanks for the response.

Those logs belonged to the case when I put same pgpool.conf on all 3
servers, based on your suggestion.
As I saw later, this should not be done for 4.1.
So we should probably discard that case.

With separate pgpool.conf file, it is logging the things I sent in earlier
mail. (connection refused by server1:9000)

Plus:

server2:
LOG:  setting the local watchdog node name to "server2:5433 Linux
ip-10-192-11-20"
LOG:  watchdog remote node:0 on server1:9000
LOG:  watchdog remote node:1 on server3:9000

server3:
LOG:  setting the local watchdog node name to "server3:5433 Linux
ip-10-192-12-30"
LOG:  watchdog remote node:0 on server1:9000
LOG:  watchdog remote node:1 on server2:9000

Thanks,
-Nirav

On Sat, Oct 30, 2021 at 11:01 PM Luca Maranzano <liuk001 at gmail.com> wrote:

> Hi Nirav,
> you're right, the node id is only from 4.2, sorry for my oversight, I
> thought it was valid for versione 4.X. It's really strange that there are
> still no 4.2 debian packages available for Debian and Ubuntu, 4.2 was
> released in 2020.
>
> I took a look at the logs, I found strange this:
>
> server1: LOG:  setting the local node "server1:5433 Linux ip-10-192-10-10"
> as watchdog cluster master    (it seems ok)
> server2: LOG:  setting the local watchdog node name to "*server1:5433 *Linux
> ip-10-192-11-20"
> server3: LOG:  setting the local watchdog node name to "*server1:5433*
> Linux ip-10-192-12-30"
>
> why server2 and server3 are referencing themselves as server1?
>
> Besides that you should set the VIP on the same subnet of the servers, for
> example 10.192.11.100, not with an IP from a different subnet (20.20.20.20)
> I still have to understand how parameters must be differentiated in 4.1 if
> you cannot use the node id file.
>
> I'll try to investigate later.
> Cheers,
> Luca
>
>
>
>
>
>
>
> On Sat, Oct 30, 2021 at 7:43 PM Nirav Bhatt <niravbhatt.cpp at gmail.com>
> wrote:
>
>> To reiterate, this pgpool2 4.1 setup.
>>
>> 1 - I have now restored my different pgpool.conf files on each server, as
>> described in the link for 4.1:
>> https://www.pgpool.net/docs/41/en/html/example-cluster.html
>>
>> They are now no longer identical.
>>
>> They only differ in
>> - wd_hostname (this server),
>> - heartbeat_destination0 & heartbeat_destination1 (other servers eg for
>> server1, this is server2 and server3, and vice versa),
>> - other_pgpool_hostname0 & other_pgpool_hostname1 (again, other servers)
>>
>> I also diffed other files under /etc/pgpool2 and postgres home, including
>> password files, they are also same across the 3 servers.
>>
>> 2 - I restarted the servers just in case port change was not in the
>> effect.
>> I also tried telnet and I could see connection going from server2 &
>> server3 to server1 on 9000.
>> So port or connectivity isn't issue.
>>
>> 3 - The issue is: WD on server1 is refusing connection to server2 and
>> server3 watchdogs on 9000.
>>
>> On server2 and server3 log:
>>
>> LOG:  watchdog node state changed from [DEAD] to [LOADING]
>> LOG:  new outbound connection to server1:9000
>> FATAL:  Add to watchdog cluster request is rejected by node "server1:9000"
>> HINT:  check the watchdog configurations.
>> LOG:  Watchdog is shutting down
>>
>> while server1 log:
>>
>> LOG:  new watchdog node connection is received from "10.192.11.20:23277"
>> <-- server2
>> LOG:  new watchdog node connection is received from "10.192.12.30:16596"
>> <-- server3
>>
>> (nothing else about why it refused connection - just this thing above)
>>
>> I believe that's enough debug info, if you could spot any problems,
>> please let me know.
>>
>> Thanks for the assistance so far,
>> -Nirav
>>
>>
>>
>> On Sat, Oct 30, 2021 at 7:24 PM Nirav Bhatt <niravbhatt.cpp at gmail.com>
>> wrote:
>>
>>> I also suspect if following line is true for 4.1 setup:
>>>
>>>
>>> *From Pgpool-II 4.2, now all configuration parameters are identical on
>>> all hosts*
>>>
>>> -Nirav
>>>
>>> On Sat, Oct 30, 2021 at 7:22 PM Nirav Bhatt <niravbhatt.cpp at gmail.com>
>>> wrote:
>>>
>>>> Hello
>>>>
>>>> I am following this link, which is pgpool2 v4.1.
>>>>
>>>> https://www.pgpool.net/docs/41/en/html/example-cluster.html
>>>>
>>>> On my Ubuntu setup also that is the latest I can get:
>>>> *4.1.4-3.pgdg20.04+2*
>>>>
>>>> If you still insist, I will try to set it up, but I guess you are
>>>> quoting from this link:
>>>>
>>>>
>>>> https://www.pgpool.net/docs/latest/en/html/example-cluster.html#EXAMPLE-CLUSTER-PGPOOL-NODE-ID
>>>>
>>>> It is a more recent version of pgpool2 than what I have.
>>>>
>>>> -Nirav
>>>>
>>>> On Sat, Oct 30, 2021 at 7:03 PM Luca Maranzano <liuk001 at gmail.com>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>> I think that the pgpool_node_id file is mandatory since it is used by
>>>>> every server to identify itself in the cluster with its node id.
>>>>> Where is it documented that it is not necessary?
>>>>>
>>>>> Try to configure it
>>>>>
>>>>> HTH
>>>>> Regards
>>>>> Luca
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Oct 30, 2021 at 5:42 PM Nirav Bhatt <niravbhatt.cpp at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello
>>>>>> Sorry for the late answer.
>>>>>>
>>>>>> I now made pgpool.conf identical on all 3 servers, then restarted
>>>>>> postgresql and pgpool2 services.
>>>>>>
>>>>>> I opened ports on each server: 22, 9999/tcp, 9898/tcp, 9000/tcp,
>>>>>> 9694/udp, 5433, 5432
>>>>>>
>>>>>> Yet the problem persists on server2 and server3 - it can not connect
>>>>>> to 9000 port (on themselves)
>>>>>> Server1 pgpool seems to be running without problem.
>>>>>>
>>>>>> I am attaching my pgpool.conf file. Also, I am attaching pgpool2
>>>>>> service logs from all 3 servers.
>>>>>>
>>>>>> 10.192.10.10 server1
>>>>>> 10.192.11.20 server2
>>>>>> 10.192.12.30 server3
>>>>>>
>>>>>> Virtual IP is 20.20.20.20
>>>>>>
>>>>>> I am running the following pgpool2 version so I do not think I need
>>>>>> pgpool_node_id file.
>>>>>>
>>>>>> 4.1.4-3.pgdg20.04+2
>>>>>>
>>>>>> Thanks for the support,
>>>>>> -Nirav
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 27, 2021 at 8:48 AM Luca Maranzano <liuk001 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi, still not clear to me, from the doc:
>>>>>>>
>>>>>>> 8.2.5. Create pgpool_node_id
>>>>>>>
>>>>>>> From Pgpool-II 4.2, now all configuration parameters are identical
>>>>>>> on all hosts. If watchdog feature is enabled, to distingusish which
>>>>>>> host is which, a pgpool_node_id file is required. You need to
>>>>>>> create a pgpool_node_id file and specify the pgpool (watchdog) node
>>>>>>> number (e.g. 0, 1, 2 ...) to identify pgpool (watchdog) host.
>>>>>>>
>>>>>>> In our systems the pgpool.conf are *identical *
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 26, 2021 at 8:09 PM Nirav Bhatt <
>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>
>>>>>>>> The difference lies in “other” server host names.
>>>>>>>>
>>>>>>>> -Nirav
>>>>>>>>
>>>>>>>> On Tue, 26 Oct 2021 at 21.07, Luca Maranzano <liuk001 at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> AFAIK the pgpool.conf file must be identical on all cluster nodes.
>>>>>>>>> Which are the differences in detail?
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>> Luca
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Oct 25, 2021 at 9:46 PM Nirav Bhatt <
>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I added all ports via ufw on server1:
>>>>>>>>>> 22,9999,9898,9000,9694,5433,5432
>>>>>>>>>>
>>>>>>>>>> I enabled firewall past that point.
>>>>>>>>>>
>>>>>>>>>> I diffed all conf files across servers - all files are same,
>>>>>>>>>> including hba and .pgpass and .pcppass.
>>>>>>>>>> Only pgpool.conf files differ, that too in parameters like server
>>>>>>>>>> hosts for pgpool and watchdog, no other differences.
>>>>>>>>>>
>>>>>>>>>> No error is observed on primary server1. The last log lines on
>>>>>>>>>> server1 says: new watchdog node connection is received from <server2> or
>>>>>>>>>> <server3>
>>>>>>>>>>
>>>>>>>>>> But secondary server pgpool2 keeps exiting with:
>>>>>>>>>>
>>>>>>>>>> Add to watchdog cluster request is rejected by node "server1:9000"
>>>>>>>>>>
>>>>>>>>>> I somehow suspect it has to do with wd_authkey which I leave
>>>>>>>>>> empty:
>>>>>>>>>> https://www.sraoss.jp/pipermail/pgpool-hackers/2017-August/002497.html
>>>>>>>>>>
>>>>>>>>>> But it is an old issue, so I can't say for sure.
>>>>>>>>>>
>>>>>>>>>> -Nirav
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 25, 2021 at 8:07 PM Luca Maranzano <liuk001 at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Firewall-cmd has to be used on Redhat derived flavours, on
>>>>>>>>>>> Ubuntu standard is ufw, so "ufw disable" should open all ports
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Oct 25, 2021 at 6:59 PM Nirav Bhatt <
>>>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I ran sudo ufw status and firewall is already inactive.
>>>>>>>>>>>>
>>>>>>>>>>>> Not sure what firewall-cmd did for all the pgpool ports. (they
>>>>>>>>>>>> are showing as list of open ports)
>>>>>>>>>>>>
>>>>>>>>>>>> Should add them all via ufw also?
>>>>>>>>>>>>
>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Oct 25, 2021 at 7:46 PM Luca Maranzano <
>>>>>>>>>>>> liuk001 at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Try to disable firewall to see it it's related, all the pgpool
>>>>>>>>>>>>> configuration files are identical on the 3 nodes?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 6:41 PM Nirav Bhatt <
>>>>>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the response.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I restarted pgpool2 service on all 3 nodes (with some delay,
>>>>>>>>>>>>>> and in the order: primary, secondary1, secondary2).
>>>>>>>>>>>>>> My primary pgpool2 service logs do show
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> LOG:  new watchdog node connection is received from
>>>>>>>>>>>>>> <secondary host>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But my secondary pgpool2 service logs (among other normal
>>>>>>>>>>>>>> messages) show:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> FATAL:  Add to watchdog cluster request is rejected by node
>>>>>>>>>>>>>> "server1:9000"
>>>>>>>>>>>>>> FATAL:  watchdog child process exit with fatal error. exiting
>>>>>>>>>>>>>> pgpool-II
>>>>>>>>>>>>>> LOG:  watchdog node state changed from [DEAD] to [LOADING]
>>>>>>>>>>>>>> LOG:  new outbound connection to server1:9000
>>>>>>>>>>>>>> FATAL:  Add to watchdog cluster request is rejected by node
>>>>>>>>>>>>>> "server1:9000"
>>>>>>>>>>>>>> HINT:  check the watchdog configurations.
>>>>>>>>>>>>>> LOG:  Watchdog is shutting down
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I checked that server1 has 9000 port open, along with 9999,
>>>>>>>>>>>>>> 9898 and 9694:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> > firewall-cmd --list-ports
>>>>>>>>>>>>>> 9999/tcp 9898/tcp 9000/tcp 9694/udp
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What could be going on?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 7:13 PM Luca Maranzano <
>>>>>>>>>>>>>> liuk001 at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> if you cannot connect to 20.20.20.20 -p 9999 it means that
>>>>>>>>>>>>>>> hte PGPool cluster is not well formed and that the VIP has not been enabled.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You can check that with pcp_watchdog_info.
>>>>>>>>>>>>>>> The logs from pgpool should also help to diagnose
>>>>>>>>>>>>>>> the problem.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> HTH
>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>> Luca
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------- Forwarded message ---------
>>>>>>>>>>>>>>> From: <pgpool-general-request at pgpool.net>
>>>>>>>>>>>>>>> Date: Mon, Oct 25, 2021 at 5:00 AM
>>>>>>>>>>>>>>> Subject: pgpool-general Digest, Vol 120, Issue 45
>>>>>>>>>>>>>>> To: <pgpool-general at pgpool.net>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Send pgpool-general mailing list submissions to
>>>>>>>>>>>>>>>         pgpool-general at pgpool.net
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://www.pgpool.net/mailman/listinfo/pgpool-general
>>>>>>>>>>>>>>> or, via email, send a message with subject or body 'help' to
>>>>>>>>>>>>>>>         pgpool-general-request at pgpool.net
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You can reach the person managing the list at
>>>>>>>>>>>>>>>         pgpool-general-owner at pgpool.net
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When replying, please edit your Subject line so it is more
>>>>>>>>>>>>>>> specific
>>>>>>>>>>>>>>> than "Re: Contents of pgpool-general digest..."
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Today's Topics:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    1. [pgpool-general: 7822] Pgpool2 example with watchdog:
>>>>>>>>>>>>>>> Unable
>>>>>>>>>>>>>>>       to test (Nirav Bhatt)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Message: 1
>>>>>>>>>>>>>>> Date: Sun, 24 Oct 2021 20:18:28 +0300
>>>>>>>>>>>>>>> From: Nirav Bhatt <niravbhatt.cpp at gmail.com>
>>>>>>>>>>>>>>> To: pgpool-general at pgpool.net
>>>>>>>>>>>>>>> Subject: [pgpool-general: 7822] Pgpool2 example with
>>>>>>>>>>>>>>> watchdog: Unable
>>>>>>>>>>>>>>>         to test
>>>>>>>>>>>>>>> Message-ID:
>>>>>>>>>>>>>>>         <CABz_77PMu9ksuYM2QGBaAuf=m_edGoKcidmWqC=K7Rpr=
>>>>>>>>>>>>>>> J_ZGw at mail.gmail.com>
>>>>>>>>>>>>>>> Content-Type: text/plain; charset="utf-8"
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> After doing all the config (config files, passwordless ssh,
>>>>>>>>>>>>>>> firewall-cmd
>>>>>>>>>>>>>>> etc) as given in the following link, I am unable to test my
>>>>>>>>>>>>>>> setup. My OS is
>>>>>>>>>>>>>>> Ubuntu 20.04.3 LTS.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://www.pgpool.net/docs/41/en/html/example-cluster.html#EXAMPLE-CLUSTER-STRUCTURE
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> All the port numbers are as listed in the example.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The part in example section: 8.3.7.1 and onwards is not
>>>>>>>>>>>>>>> working.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When I run on server1 (my primary):
>>>>>>>>>>>>>>> psql -h 20.20.20.20 -p 9999 -U pgpool postgres -c "show
>>>>>>>>>>>>>>> pool_nodes"
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I got error:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> psql: error: connection to server at "20.20.20.20", port
>>>>>>>>>>>>>>> 9999 failed:
>>>>>>>>>>>>>>> Connection refused
>>>>>>>>>>>>>>> Is the server running on that host and accepting TCP/IP
>>>>>>>>>>>>>>> connections?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I restarted postgresql and pgpool2 services time and again.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> pgpool2 service most recent logs:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> server1 (current primary):
>>>>>>>>>>>>>>> Oct 24 17:01:41 ip-10-192-10-10 pgpool[104691]: 2021-10-24
>>>>>>>>>>>>>>> 17:01:41: pid
>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is received from
>>>>>>>>>>>>>>> "<server1
>>>>>>>>>>>>>>> host>:16553"
>>>>>>>>>>>>>>> Oct 24 17:01:41 ip-10-192-10-10 pgpool[104691]: 2021-10-24
>>>>>>>>>>>>>>> 17:01:41: pid
>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is received from
>>>>>>>>>>>>>>> "<server1
>>>>>>>>>>>>>>> host>:17577"
>>>>>>>>>>>>>>> Oct 24 17:04:02 ip-10-192-10-10 pgpool[104691]: 2021-10-24
>>>>>>>>>>>>>>> 17:04:02: pid
>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is received from
>>>>>>>>>>>>>>> "<server2
>>>>>>>>>>>>>>> host>:49304"
>>>>>>>>>>>>>>> Oct 24 17:04:02 ip-10-192-10-10 pgpool[104691]: 2021-10-24
>>>>>>>>>>>>>>> 17:04:02: pid
>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is received from
>>>>>>>>>>>>>>> "<server2
>>>>>>>>>>>>>>> host>:50328"
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-10-10 pgpool[104691]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is received from
>>>>>>>>>>>>>>> "<server3
>>>>>>>>>>>>>>> host>:51864"
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-10-10 pgpool[104691]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is received from
>>>>>>>>>>>>>>> "<server3
>>>>>>>>>>>>>>> host>:52888"
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> server2 and server3 (looks like pgpool2 service is failing):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125018: LOG:  setting the local watchdog node name to
>>>>>>>>>>>>>>> "server3:5433 Linux
>>>>>>>>>>>>>>> ip-<server 3 host>"
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125018: LOG:  watchdog cluster is configured with 2 remote
>>>>>>>>>>>>>>> nodes
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125018: LOG:  watchdog remote node:0 on server1:9000
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125018: LOG:  watchdog remote node:1 on server2:9000
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125018: LOG:  interface monitoring is disabled in watchdog
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125018: LOG:  watchdog node state changed from [DEAD] to
>>>>>>>>>>>>>>> [LOADING]
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125018: LOG:  new outbound connection to server1:9000
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125018: FATAL:  Add to watchdog cluster request is rejected
>>>>>>>>>>>>>>> by node
>>>>>>>>>>>>>>> "server1:9000"
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125018: HINT:  check the watchdog configurations.
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125018: LOG:  Watchdog is shutting down
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125014]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125014: LOG:  watchdog child process with pid: 125018 exits
>>>>>>>>>>>>>>> with status 768
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125014]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125014: FATAL:  watchdog child process exit with fatal
>>>>>>>>>>>>>>> error. exiting
>>>>>>>>>>>>>>> pgpool-II
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125019: LOG:  setting the local watchdog node name to
>>>>>>>>>>>>>>> "server3:5433 Linux
>>>>>>>>>>>>>>> ip-<server 3 host>"
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125019: LOG:  watchdog cluster is configured with 2 remote
>>>>>>>>>>>>>>> nodes
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125019: LOG:  watchdog remote node:0 on server1:9000
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125019: LOG:  watchdog remote node:1 on server2:9000
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125019: LOG:  interface monitoring is disabled in watchdog
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125019: LOG:  watchdog node state changed from [DEAD] to
>>>>>>>>>>>>>>> [LOADING]
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125019: LOG:  new outbound connection to server1:9000
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]: 2021-10-24
>>>>>>>>>>>>>>> 17:11:12: pid
>>>>>>>>>>>>>>> 125019: LOG:  Watchdog is shutting down
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 systemd[1]: pgpool2.service:
>>>>>>>>>>>>>>> Main process
>>>>>>>>>>>>>>> exited, code=exited, status=3/NOTIMPLEMENTED
>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 systemd[1]: pgpool2.service:
>>>>>>>>>>>>>>> Failed with
>>>>>>>>>>>>>>> result 'exit-code'.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What steps am I missing?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you for the assistance,
>>>>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>>>> -------------- next part --------------
>>>>>>>>>>>>>>> An HTML attachment was scrubbed...
>>>>>>>>>>>>>>> URL: <
>>>>>>>>>>>>>>> http://www.pgpool.net/pipermail/pgpool-general/attachments/20211024/98ab6d68/attachment-0001.htm
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Subject: Digest Footer
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> pgpool-general mailing list
>>>>>>>>>>>>>>> pgpool-general at pgpool.net
>>>>>>>>>>>>>>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> End of pgpool-general Digest, Vol 120, Issue 45
>>>>>>>>>>>>>>> ***********************************************
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20211030/aa74acfd/attachment-0001.htm>

More information about the pgpool-general mailing list