[pgpool-general: 7842] Re: pgpool-general Digest, Vol 120, Issue 45

Nirav Bhatt niravbhatt.cpp at gmail.com
Sun Oct 31 06:39:40 JST 2021 

pool_key problem was only observed in detached mode i.e. pgpool -n option.
pgpool was unable to decrypt password key somehow.

It is not happening with systemctl start pgpool2.
(I do not know what's the difference though, and whether process invocation
could provide any debug info)

I can't see any error in service logs when I start using systemctl.
So I assume things are good from password standpoint.

You are right - Routes in AWS enable routing to specific instance using
virtual ip (attached to interface via runtime script - up cmd in this case)

https://aws.amazon.com/blogs/apn/making-application-failover-seamless-by-failing-over-your-private-virtual-ip-across-availability-zones/

Regarding pgpool2, it's a pity though such a well documented, highly
customized & fine grained tool can't operate after all the config, and it
is not obvious to find the reason from the service logs.
(I have been on this now since more than 1.5 months, but cannot get it
working - my limited knowledge of linux is also playing its part)

To see if any of my server config was wrong, lastly I even switched servers
i.e started server2 process first (which became the master) and started
server1 process later - which failed just like the earlier case while
connecting to server2:9000

Will continue tomorrow about it.

Thanks all along,
-Nirav

On Sat, Oct 30, 2021 at 11:51 PM Nirav Bhatt <niravbhatt.cpp at gmail.com>
wrote:

> 3 subnets in 3 AWS Availability zones for redundancy. They still share the
> same network.
> This is a perfectly valid use case. (I have read about it earlier).
>
> There seems some pool_key problem on server1, which I am investigating now.
>
> -Nirav
>
>
> On Sat, Oct 30, 2021 at 11:43 PM Luca Maranzano <liuk001 at gmail.com> wrote:
>
>> How different? Are the 3 nodes on 3 separate networks? If yes why?
>>
>> Il sab 30 ott 2021, 22:40 Nirav Bhatt <niravbhatt.cpp at gmail.com> ha
>> scritto:
>>
>>> They are different, but everything including ssh copy and ping is
>>> working between them.
>>>
>>>
>>>
>>> On Sat, Oct 30, 2021 at 11:31 PM Luca Maranzano <liuk001 at gmail.com>
>>> wrote:
>>>
>>>> What about the netmask of the network interface on 3 nodes?
>>>>
>>>> Il sab 30 ott 2021, 22:26 Nirav Bhatt <niravbhatt.cpp at gmail.com> ha
>>>> scritto:
>>>>
>>>>> Hi Luca
>>>>>
>>>>> Thanks for the response.
>>>>>
>>>>> Those logs belonged to the case when I put same pgpool.conf on all 3
>>>>> servers, based on your suggestion.
>>>>> As I saw later, this should not be done for 4.1.
>>>>> So we should probably discard that case.
>>>>>
>>>>> With separate pgpool.conf file, it is logging the things I sent in
>>>>> earlier mail. (connection refused by server1:9000)
>>>>>
>>>>> Plus:
>>>>>
>>>>> server2:
>>>>> LOG:  setting the local watchdog node name to "server2:5433 Linux
>>>>> ip-10-192-11-20"
>>>>> LOG:  watchdog remote node:0 on server1:9000
>>>>> LOG:  watchdog remote node:1 on server3:9000
>>>>>
>>>>> server3:
>>>>> LOG:  setting the local watchdog node name to "server3:5433 Linux
>>>>> ip-10-192-12-30"
>>>>> LOG:  watchdog remote node:0 on server1:9000
>>>>> LOG:  watchdog remote node:1 on server2:9000
>>>>>
>>>>> Thanks,
>>>>> -Nirav
>>>>>
>>>>> On Sat, Oct 30, 2021 at 11:01 PM Luca Maranzano <liuk001 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Nirav,
>>>>>> you're right, the node id is only from 4.2, sorry for my oversight, I
>>>>>> thought it was valid for versione 4.X. It's really strange that there are
>>>>>> still no 4.2 debian packages available for Debian and Ubuntu, 4.2 was
>>>>>> released in 2020.
>>>>>>
>>>>>> I took a look at the logs, I found strange this:
>>>>>>
>>>>>> server1: LOG:  setting the local node "server1:5433 Linux
>>>>>> ip-10-192-10-10" as watchdog cluster master    (it seems ok)
>>>>>> server2: LOG:  setting the local watchdog node name to "*server1:5433
>>>>>> *Linux ip-10-192-11-20"
>>>>>> server3: LOG:  setting the local watchdog node name to "
>>>>>> *server1:5433* Linux ip-10-192-12-30"
>>>>>>
>>>>>> why server2 and server3 are referencing themselves as server1?
>>>>>>
>>>>>> Besides that you should set the VIP on the same subnet of the
>>>>>> servers, for example 10.192.11.100, not with an IP from a different subnet
>>>>>> (20.20.20.20)
>>>>>> I still have to understand how parameters must be differentiated in
>>>>>> 4.1 if you cannot use the node id file.
>>>>>>
>>>>>> I'll try to investigate later.
>>>>>> Cheers,
>>>>>> Luca
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Oct 30, 2021 at 7:43 PM Nirav Bhatt <niravbhatt.cpp at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> To reiterate, this pgpool2 4.1 setup.
>>>>>>>
>>>>>>> 1 - I have now restored my different pgpool.conf files on each
>>>>>>> server, as described in the link for 4.1:
>>>>>>> https://www.pgpool.net/docs/41/en/html/example-cluster.html
>>>>>>>
>>>>>>> They are now no longer identical.
>>>>>>>
>>>>>>> They only differ in
>>>>>>> - wd_hostname (this server),
>>>>>>> - heartbeat_destination0 & heartbeat_destination1 (other servers eg
>>>>>>> for server1, this is server2 and server3, and vice versa),
>>>>>>> - other_pgpool_hostname0 & other_pgpool_hostname1 (again, other
>>>>>>> servers)
>>>>>>>
>>>>>>> I also diffed other files under /etc/pgpool2 and postgres home,
>>>>>>> including password files, they are also same across the 3 servers.
>>>>>>>
>>>>>>> 2 - I restarted the servers just in case port change was not in the
>>>>>>> effect.
>>>>>>> I also tried telnet and I could see connection going from server2 &
>>>>>>> server3 to server1 on 9000.
>>>>>>> So port or connectivity isn't issue.
>>>>>>>
>>>>>>> 3 - The issue is: WD on server1 is refusing connection to server2
>>>>>>> and server3 watchdogs on 9000.
>>>>>>>
>>>>>>> On server2 and server3 log:
>>>>>>>
>>>>>>> LOG:  watchdog node state changed from [DEAD] to [LOADING]
>>>>>>> LOG:  new outbound connection to server1:9000
>>>>>>> FATAL:  Add to watchdog cluster request is rejected by node
>>>>>>> "server1:9000"
>>>>>>> HINT:  check the watchdog configurations.
>>>>>>> LOG:  Watchdog is shutting down
>>>>>>>
>>>>>>> while server1 log:
>>>>>>>
>>>>>>> LOG:  new watchdog node connection is received from "
>>>>>>> 10.192.11.20:23277" <-- server2
>>>>>>> LOG:  new watchdog node connection is received from "
>>>>>>> 10.192.12.30:16596" <-- server3
>>>>>>>
>>>>>>> (nothing else about why it refused connection - just this thing
>>>>>>> above)
>>>>>>>
>>>>>>> I believe that's enough debug info, if you could spot any problems,
>>>>>>> please let me know.
>>>>>>>
>>>>>>> Thanks for the assistance so far,
>>>>>>> -Nirav
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Oct 30, 2021 at 7:24 PM Nirav Bhatt <
>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>
>>>>>>>> I also suspect if following line is true for 4.1 setup:
>>>>>>>>
>>>>>>>>
>>>>>>>> *From Pgpool-II 4.2, now all configuration parameters are identical
>>>>>>>> on all hosts*
>>>>>>>>
>>>>>>>> -Nirav
>>>>>>>>
>>>>>>>> On Sat, Oct 30, 2021 at 7:22 PM Nirav Bhatt <
>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hello
>>>>>>>>>
>>>>>>>>> I am following this link, which is pgpool2 v4.1.
>>>>>>>>>
>>>>>>>>> https://www.pgpool.net/docs/41/en/html/example-cluster.html
>>>>>>>>>
>>>>>>>>> On my Ubuntu setup also that is the latest I can get:
>>>>>>>>> *4.1.4-3.pgdg20.04+2*
>>>>>>>>>
>>>>>>>>> If you still insist, I will try to set it up, but I guess you are
>>>>>>>>> quoting from this link:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://www.pgpool.net/docs/latest/en/html/example-cluster.html#EXAMPLE-CLUSTER-PGPOOL-NODE-ID
>>>>>>>>>
>>>>>>>>> It is a more recent version of pgpool2 than what I have.
>>>>>>>>>
>>>>>>>>> -Nirav
>>>>>>>>>
>>>>>>>>> On Sat, Oct 30, 2021 at 7:03 PM Luca Maranzano <liuk001 at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>> I think that the pgpool_node_id file is mandatory since it is
>>>>>>>>>> used by every server to identify itself in the cluster with its node id.
>>>>>>>>>> Where is it documented that it is not necessary?
>>>>>>>>>>
>>>>>>>>>> Try to configure it
>>>>>>>>>>
>>>>>>>>>> HTH
>>>>>>>>>> Regards
>>>>>>>>>> Luca
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sat, Oct 30, 2021 at 5:42 PM Nirav Bhatt <
>>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello
>>>>>>>>>>> Sorry for the late answer.
>>>>>>>>>>>
>>>>>>>>>>> I now made pgpool.conf identical on all 3 servers, then
>>>>>>>>>>> restarted postgresql and pgpool2 services.
>>>>>>>>>>>
>>>>>>>>>>> I opened ports on each server: 22, 9999/tcp, 9898/tcp, 9000/tcp,
>>>>>>>>>>> 9694/udp, 5433, 5432
>>>>>>>>>>>
>>>>>>>>>>> Yet the problem persists on server2 and server3 - it can not
>>>>>>>>>>> connect to 9000 port (on themselves)
>>>>>>>>>>> Server1 pgpool seems to be running without problem.
>>>>>>>>>>>
>>>>>>>>>>> I am attaching my pgpool.conf file. Also, I am attaching pgpool2
>>>>>>>>>>> service logs from all 3 servers.
>>>>>>>>>>>
>>>>>>>>>>> 10.192.10.10 server1
>>>>>>>>>>> 10.192.11.20 server2
>>>>>>>>>>> 10.192.12.30 server3
>>>>>>>>>>>
>>>>>>>>>>> Virtual IP is 20.20.20.20
>>>>>>>>>>>
>>>>>>>>>>> I am running the following pgpool2 version so I do not think I
>>>>>>>>>>> need pgpool_node_id file.
>>>>>>>>>>>
>>>>>>>>>>> 4.1.4-3.pgdg20.04+2
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the support,
>>>>>>>>>>> -Nirav
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Oct 27, 2021 at 8:48 AM Luca Maranzano <
>>>>>>>>>>> liuk001 at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi, still not clear to me, from the doc:
>>>>>>>>>>>>
>>>>>>>>>>>> 8.2.5. Create pgpool_node_id
>>>>>>>>>>>>
>>>>>>>>>>>> From Pgpool-II 4.2, now all configuration parameters are
>>>>>>>>>>>> identical on all hosts. If watchdog feature is enabled, to
>>>>>>>>>>>> distingusish which host is which, a pgpool_node_id file is
>>>>>>>>>>>> required. You need to create a pgpool_node_id file and specify
>>>>>>>>>>>> the pgpool (watchdog) node number (e.g. 0, 1, 2 ...) to identify pgpool
>>>>>>>>>>>> (watchdog) host.
>>>>>>>>>>>>
>>>>>>>>>>>> In our systems the pgpool.conf are *identical *
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Oct 26, 2021 at 8:09 PM Nirav Bhatt <
>>>>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> The difference lies in “other” server host names.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, 26 Oct 2021 at 21.07, Luca Maranzano <
>>>>>>>>>>>>> liuk001 at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> AFAIK the pgpool.conf file must be identical on all cluster
>>>>>>>>>>>>>> nodes.
>>>>>>>>>>>>>> Which are the differences in detail?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>> Luca
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 9:46 PM Nirav Bhatt <
>>>>>>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I added all ports via ufw on server1:
>>>>>>>>>>>>>>> 22,9999,9898,9000,9694,5433,5432
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I enabled firewall past that point.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I diffed all conf files across servers - all files are same,
>>>>>>>>>>>>>>> including hba and .pgpass and .pcppass.
>>>>>>>>>>>>>>> Only pgpool.conf files differ, that too in parameters like
>>>>>>>>>>>>>>> server hosts for pgpool and watchdog, no other differences.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> No error is observed on primary server1. The last log lines
>>>>>>>>>>>>>>> on server1 says: new watchdog node connection is received from <server2> or
>>>>>>>>>>>>>>> <server3>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But secondary server pgpool2 keeps exiting with:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Add to watchdog cluster request is rejected by node
>>>>>>>>>>>>>>> "server1:9000"
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I somehow suspect it has to do with wd_authkey which I leave
>>>>>>>>>>>>>>> empty:
>>>>>>>>>>>>>>> https://www.sraoss.jp/pipermail/pgpool-hackers/2017-August/002497.html
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But it is an old issue, so I can't say for sure.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 8:07 PM Luca Maranzano <
>>>>>>>>>>>>>>> liuk001 at gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Firewall-cmd has to be used on Redhat derived flavours, on
>>>>>>>>>>>>>>>> Ubuntu standard is ufw, so "ufw disable" should open all ports
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 6:59 PM Nirav Bhatt <
>>>>>>>>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I ran sudo ufw status and firewall is already inactive.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Not sure what firewall-cmd did for all the pgpool ports.
>>>>>>>>>>>>>>>>> (they are showing as list of open ports)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Should add them all via ufw also?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 7:46 PM Luca Maranzano <
>>>>>>>>>>>>>>>>> liuk001 at gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Try to disable firewall to see it it's related, all the
>>>>>>>>>>>>>>>>>> pgpool configuration files are identical on the 3 nodes?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 6:41 PM Nirav Bhatt <
>>>>>>>>>>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for the response.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I restarted pgpool2 service on all 3 nodes (with some
>>>>>>>>>>>>>>>>>>> delay, and in the order: primary, secondary1, secondary2).
>>>>>>>>>>>>>>>>>>> My primary pgpool2 service logs do show
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> LOG:  new watchdog node connection is received from
>>>>>>>>>>>>>>>>>>> <secondary host>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> But my secondary pgpool2 service logs (among other
>>>>>>>>>>>>>>>>>>> normal messages) show:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> FATAL:  Add to watchdog cluster request is rejected by
>>>>>>>>>>>>>>>>>>> node "server1:9000"
>>>>>>>>>>>>>>>>>>> FATAL:  watchdog child process exit with fatal error.
>>>>>>>>>>>>>>>>>>> exiting pgpool-II
>>>>>>>>>>>>>>>>>>> LOG:  watchdog node state changed from [DEAD] to
>>>>>>>>>>>>>>>>>>> [LOADING]
>>>>>>>>>>>>>>>>>>> LOG:  new outbound connection to server1:9000
>>>>>>>>>>>>>>>>>>> FATAL:  Add to watchdog cluster request is rejected by
>>>>>>>>>>>>>>>>>>> node "server1:9000"
>>>>>>>>>>>>>>>>>>> HINT:  check the watchdog configurations.
>>>>>>>>>>>>>>>>>>> LOG:  Watchdog is shutting down
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I checked that server1 has 9000 port open, along with
>>>>>>>>>>>>>>>>>>> 9999, 9898 and 9694:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> > firewall-cmd --list-ports
>>>>>>>>>>>>>>>>>>> 9999/tcp 9898/tcp 9000/tcp 9694/udp
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> What could be going on?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 7:13 PM Luca Maranzano <
>>>>>>>>>>>>>>>>>>> liuk001 at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> if you cannot connect to 20.20.20.20 -p 9999 it means
>>>>>>>>>>>>>>>>>>>> that hte PGPool cluster is not well formed and that the VIP has not been
>>>>>>>>>>>>>>>>>>>> enabled.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> You can check that with pcp_watchdog_info.
>>>>>>>>>>>>>>>>>>>> The logs from pgpool should also help to diagnose
>>>>>>>>>>>>>>>>>>>> the problem.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> HTH
>>>>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>>>> Luca
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ---------- Forwarded message ---------
>>>>>>>>>>>>>>>>>>>> From: <pgpool-general-request at pgpool.net>
>>>>>>>>>>>>>>>>>>>> Date: Mon, Oct 25, 2021 at 5:00 AM
>>>>>>>>>>>>>>>>>>>> Subject: pgpool-general Digest, Vol 120, Issue 45
>>>>>>>>>>>>>>>>>>>> To: <pgpool-general at pgpool.net>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Send pgpool-general mailing list submissions to
>>>>>>>>>>>>>>>>>>>>         pgpool-general at pgpool.net
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> To subscribe or unsubscribe via the World Wide Web,
>>>>>>>>>>>>>>>>>>>> visit
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> https://www.pgpool.net/mailman/listinfo/pgpool-general
>>>>>>>>>>>>>>>>>>>> or, via email, send a message with subject or body
>>>>>>>>>>>>>>>>>>>> 'help' to
>>>>>>>>>>>>>>>>>>>>         pgpool-general-request at pgpool.net
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> You can reach the person managing the list at
>>>>>>>>>>>>>>>>>>>>         pgpool-general-owner at pgpool.net
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> When replying, please edit your Subject line so it is
>>>>>>>>>>>>>>>>>>>> more specific
>>>>>>>>>>>>>>>>>>>> than "Re: Contents of pgpool-general digest..."
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Today's Topics:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    1. [pgpool-general: 7822] Pgpool2 example with
>>>>>>>>>>>>>>>>>>>> watchdog: Unable
>>>>>>>>>>>>>>>>>>>>       to test (Nirav Bhatt)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Message: 1
>>>>>>>>>>>>>>>>>>>> Date: Sun, 24 Oct 2021 20:18:28 +0300
>>>>>>>>>>>>>>>>>>>> From: Nirav Bhatt <niravbhatt.cpp at gmail.com>
>>>>>>>>>>>>>>>>>>>> To: pgpool-general at pgpool.net
>>>>>>>>>>>>>>>>>>>> Subject: [pgpool-general: 7822] Pgpool2 example with
>>>>>>>>>>>>>>>>>>>> watchdog: Unable
>>>>>>>>>>>>>>>>>>>>         to test
>>>>>>>>>>>>>>>>>>>> Message-ID:
>>>>>>>>>>>>>>>>>>>>         <CABz_77PMu9ksuYM2QGBaAuf=m_edGoKcidmWqC=K7Rpr=
>>>>>>>>>>>>>>>>>>>> J_ZGw at mail.gmail.com>
>>>>>>>>>>>>>>>>>>>> Content-Type: text/plain; charset="utf-8"
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> After doing all the config (config files, passwordless
>>>>>>>>>>>>>>>>>>>> ssh, firewall-cmd
>>>>>>>>>>>>>>>>>>>> etc) as given in the following link, I am unable to
>>>>>>>>>>>>>>>>>>>> test my setup. My OS is
>>>>>>>>>>>>>>>>>>>> Ubuntu 20.04.3 LTS.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> https://www.pgpool.net/docs/41/en/html/example-cluster.html#EXAMPLE-CLUSTER-STRUCTURE
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> All the port numbers are as listed in the example.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The part in example section: 8.3.7.1 and onwards is not
>>>>>>>>>>>>>>>>>>>> working.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> When I run on server1 (my primary):
>>>>>>>>>>>>>>>>>>>> psql -h 20.20.20.20 -p 9999 -U pgpool postgres -c
>>>>>>>>>>>>>>>>>>>> "show pool_nodes"
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I got error:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> psql: error: connection to server at "20.20.20.20",
>>>>>>>>>>>>>>>>>>>> port 9999 failed:
>>>>>>>>>>>>>>>>>>>> Connection refused
>>>>>>>>>>>>>>>>>>>> Is the server running on that host and accepting TCP/IP
>>>>>>>>>>>>>>>>>>>> connections?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I restarted postgresql and pgpool2 services time and
>>>>>>>>>>>>>>>>>>>> again.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> pgpool2 service most recent logs:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> server1 (current primary):
>>>>>>>>>>>>>>>>>>>> Oct 24 17:01:41 ip-10-192-10-10 pgpool[104691]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:01:41: pid
>>>>>>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is received
>>>>>>>>>>>>>>>>>>>> from "<server1
>>>>>>>>>>>>>>>>>>>> host>:16553"
>>>>>>>>>>>>>>>>>>>> Oct 24 17:01:41 ip-10-192-10-10 pgpool[104691]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:01:41: pid
>>>>>>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is received
>>>>>>>>>>>>>>>>>>>> from "<server1
>>>>>>>>>>>>>>>>>>>> host>:17577"
>>>>>>>>>>>>>>>>>>>> Oct 24 17:04:02 ip-10-192-10-10 pgpool[104691]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:04:02: pid
>>>>>>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is received
>>>>>>>>>>>>>>>>>>>> from "<server2
>>>>>>>>>>>>>>>>>>>> host>:49304"
>>>>>>>>>>>>>>>>>>>> Oct 24 17:04:02 ip-10-192-10-10 pgpool[104691]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:04:02: pid
>>>>>>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is received
>>>>>>>>>>>>>>>>>>>> from "<server2
>>>>>>>>>>>>>>>>>>>> host>:50328"
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-10-10 pgpool[104691]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is received
>>>>>>>>>>>>>>>>>>>> from "<server3
>>>>>>>>>>>>>>>>>>>> host>:51864"
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-10-10 pgpool[104691]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is received
>>>>>>>>>>>>>>>>>>>> from "<server3
>>>>>>>>>>>>>>>>>>>> host>:52888"
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> server2 and server3 (looks like pgpool2 service is
>>>>>>>>>>>>>>>>>>>> failing):
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125018: LOG:  setting the local watchdog node name to
>>>>>>>>>>>>>>>>>>>> "server3:5433 Linux
>>>>>>>>>>>>>>>>>>>> ip-<server 3 host>"
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125018: LOG:  watchdog cluster is configured with 2
>>>>>>>>>>>>>>>>>>>> remote nodes
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125018: LOG:  watchdog remote node:0 on server1:9000
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125018: LOG:  watchdog remote node:1 on server2:9000
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125018: LOG:  interface monitoring is disabled in
>>>>>>>>>>>>>>>>>>>> watchdog
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125018: LOG:  watchdog node state changed from [DEAD]
>>>>>>>>>>>>>>>>>>>> to [LOADING]
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125018: LOG:  new outbound connection to server1:9000
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125018: FATAL:  Add to watchdog cluster request is
>>>>>>>>>>>>>>>>>>>> rejected by node
>>>>>>>>>>>>>>>>>>>> "server1:9000"
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125018: HINT:  check the watchdog configurations.
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125018: LOG:  Watchdog is shutting down
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125014]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125014: LOG:  watchdog child process with pid: 125018
>>>>>>>>>>>>>>>>>>>> exits with status 768
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125014]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125014: FATAL:  watchdog child process exit with fatal
>>>>>>>>>>>>>>>>>>>> error. exiting
>>>>>>>>>>>>>>>>>>>> pgpool-II
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125019: LOG:  setting the local watchdog node name to
>>>>>>>>>>>>>>>>>>>> "server3:5433 Linux
>>>>>>>>>>>>>>>>>>>> ip-<server 3 host>"
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125019: LOG:  watchdog cluster is configured with 2
>>>>>>>>>>>>>>>>>>>> remote nodes
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125019: LOG:  watchdog remote node:0 on server1:9000
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125019: LOG:  watchdog remote node:1 on server2:9000
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125019: LOG:  interface monitoring is disabled in
>>>>>>>>>>>>>>>>>>>> watchdog
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125019: LOG:  watchdog node state changed from [DEAD]
>>>>>>>>>>>>>>>>>>>> to [LOADING]
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125019: LOG:  new outbound connection to server1:9000
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>> 125019: LOG:  Watchdog is shutting down
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 systemd[1]:
>>>>>>>>>>>>>>>>>>>> pgpool2.service: Main process
>>>>>>>>>>>>>>>>>>>> exited, code=exited, status=3/NOTIMPLEMENTED
>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 systemd[1]:
>>>>>>>>>>>>>>>>>>>> pgpool2.service: Failed with
>>>>>>>>>>>>>>>>>>>> result 'exit-code'.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> What steps am I missing?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thank you for the assistance,
>>>>>>>>>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>>>>>>>>> -------------- next part --------------
>>>>>>>>>>>>>>>>>>>> An HTML attachment was scrubbed...
>>>>>>>>>>>>>>>>>>>> URL: <
>>>>>>>>>>>>>>>>>>>> http://www.pgpool.net/pipermail/pgpool-general/attachments/20211024/98ab6d68/attachment-0001.htm
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Subject: Digest Footer
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>> pgpool-general mailing list
>>>>>>>>>>>>>>>>>>>> pgpool-general at pgpool.net
>>>>>>>>>>>>>>>>>>>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> End of pgpool-general Digest, Vol 120, Issue 45
>>>>>>>>>>>>>>>>>>>> ***********************************************
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20211031/86f8eb18/attachment-0001.htm>

More information about the pgpool-general mailing list