[pgpool-general: 7843] Re: pgpool-general Digest, Vol 120, Issue 45

Nirav Bhatt niravbhatt.cpp at gmail.com
Sun Oct 31 23:30:46 JST 2021 

Btw - I forgot to attach the 3 pgpool.conf files (4.1, so all 3 are
different) I use on each server.

I am also attaching pgpool2 service log on server1 (primary, for which
pgpool-primary.conf is there) and that on server2 (secondary, for which
pgpool-secondary.conf is there)

Thanks
-Nirav

On Sun, Oct 31, 2021 at 12:39 AM Nirav Bhatt <niravbhatt.cpp at gmail.com>
wrote:

> pool_key problem was only observed in detached mode i.e. pgpool -n option.
> pgpool was unable to decrypt password key somehow.
>
> It is not happening with systemctl start pgpool2.
> (I do not know what's the difference though, and whether process
> invocation could provide any debug info)
>
> I can't see any error in service logs when I start using systemctl.
> So I assume things are good from password standpoint.
>
> You are right - Routes in AWS enable routing to specific instance using
> virtual ip (attached to interface via runtime script - up cmd in this case)
>
>
> https://aws.amazon.com/blogs/apn/making-application-failover-seamless-by-failing-over-your-private-virtual-ip-across-availability-zones/
>
> Regarding pgpool2, it's a pity though such a well documented, highly
> customized & fine grained tool can't operate after all the config, and it
> is not obvious to find the reason from the service logs.
> (I have been on this now since more than 1.5 months, but cannot get it
> working - my limited knowledge of linux is also playing its part)
>
> To see if any of my server config was wrong, lastly I even switched
> servers i.e started server2 process first (which became the master) and
> started server1 process later - which failed just like the earlier case
> while connecting to server2:9000
>
> Will continue tomorrow about it.
>
> Thanks all along,
> -Nirav
>
> On Sat, Oct 30, 2021 at 11:51 PM Nirav Bhatt <niravbhatt.cpp at gmail.com>
> wrote:
>
>> 3 subnets in 3 AWS Availability zones for redundancy. They still share
>> the same network.
>> This is a perfectly valid use case. (I have read about it earlier).
>>
>> There seems some pool_key problem on server1, which I am investigating
>> now.
>>
>> -Nirav
>>
>>
>> On Sat, Oct 30, 2021 at 11:43 PM Luca Maranzano <liuk001 at gmail.com>
>> wrote:
>>
>>> How different? Are the 3 nodes on 3 separate networks? If yes why?
>>>
>>> Il sab 30 ott 2021, 22:40 Nirav Bhatt <niravbhatt.cpp at gmail.com> ha
>>> scritto:
>>>
>>>> They are different, but everything including ssh copy and ping is
>>>> working between them.
>>>>
>>>>
>>>>
>>>> On Sat, Oct 30, 2021 at 11:31 PM Luca Maranzano <liuk001 at gmail.com>
>>>> wrote:
>>>>
>>>>> What about the netmask of the network interface on 3 nodes?
>>>>>
>>>>> Il sab 30 ott 2021, 22:26 Nirav Bhatt <niravbhatt.cpp at gmail.com> ha
>>>>> scritto:
>>>>>
>>>>>> Hi Luca
>>>>>>
>>>>>> Thanks for the response.
>>>>>>
>>>>>> Those logs belonged to the case when I put same pgpool.conf on all 3
>>>>>> servers, based on your suggestion.
>>>>>> As I saw later, this should not be done for 4.1.
>>>>>> So we should probably discard that case.
>>>>>>
>>>>>> With separate pgpool.conf file, it is logging the things I sent in
>>>>>> earlier mail. (connection refused by server1:9000)
>>>>>>
>>>>>> Plus:
>>>>>>
>>>>>> server2:
>>>>>> LOG:  setting the local watchdog node name to "server2:5433 Linux
>>>>>> ip-10-192-11-20"
>>>>>> LOG:  watchdog remote node:0 on server1:9000
>>>>>> LOG:  watchdog remote node:1 on server3:9000
>>>>>>
>>>>>> server3:
>>>>>> LOG:  setting the local watchdog node name to "server3:5433 Linux
>>>>>> ip-10-192-12-30"
>>>>>> LOG:  watchdog remote node:0 on server1:9000
>>>>>> LOG:  watchdog remote node:1 on server2:9000
>>>>>>
>>>>>> Thanks,
>>>>>> -Nirav
>>>>>>
>>>>>> On Sat, Oct 30, 2021 at 11:01 PM Luca Maranzano <liuk001 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Nirav,
>>>>>>> you're right, the node id is only from 4.2, sorry for my oversight,
>>>>>>> I thought it was valid for versione 4.X. It's really strange that there are
>>>>>>> still no 4.2 debian packages available for Debian and Ubuntu, 4.2 was
>>>>>>> released in 2020.
>>>>>>>
>>>>>>> I took a look at the logs, I found strange this:
>>>>>>>
>>>>>>> server1: LOG:  setting the local node "server1:5433 Linux
>>>>>>> ip-10-192-10-10" as watchdog cluster master    (it seems ok)
>>>>>>> server2: LOG:  setting the local watchdog node name to "*server1:5433
>>>>>>> *Linux ip-10-192-11-20"
>>>>>>> server3: LOG:  setting the local watchdog node name to "
>>>>>>> *server1:5433* Linux ip-10-192-12-30"
>>>>>>>
>>>>>>> why server2 and server3 are referencing themselves as server1?
>>>>>>>
>>>>>>> Besides that you should set the VIP on the same subnet of the
>>>>>>> servers, for example 10.192.11.100, not with an IP from a different subnet
>>>>>>> (20.20.20.20)
>>>>>>> I still have to understand how parameters must be differentiated in
>>>>>>> 4.1 if you cannot use the node id file.
>>>>>>>
>>>>>>> I'll try to investigate later.
>>>>>>> Cheers,
>>>>>>> Luca
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Oct 30, 2021 at 7:43 PM Nirav Bhatt <
>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>
>>>>>>>> To reiterate, this pgpool2 4.1 setup.
>>>>>>>>
>>>>>>>> 1 - I have now restored my different pgpool.conf files on each
>>>>>>>> server, as described in the link for 4.1:
>>>>>>>> https://www.pgpool.net/docs/41/en/html/example-cluster.html
>>>>>>>>
>>>>>>>> They are now no longer identical.
>>>>>>>>
>>>>>>>> They only differ in
>>>>>>>> - wd_hostname (this server),
>>>>>>>> - heartbeat_destination0 & heartbeat_destination1 (other servers eg
>>>>>>>> for server1, this is server2 and server3, and vice versa),
>>>>>>>> - other_pgpool_hostname0 & other_pgpool_hostname1 (again, other
>>>>>>>> servers)
>>>>>>>>
>>>>>>>> I also diffed other files under /etc/pgpool2 and postgres home,
>>>>>>>> including password files, they are also same across the 3 servers.
>>>>>>>>
>>>>>>>> 2 - I restarted the servers just in case port change was not in the
>>>>>>>> effect.
>>>>>>>> I also tried telnet and I could see connection going from server2 &
>>>>>>>> server3 to server1 on 9000.
>>>>>>>> So port or connectivity isn't issue.
>>>>>>>>
>>>>>>>> 3 - The issue is: WD on server1 is refusing connection to server2
>>>>>>>> and server3 watchdogs on 9000.
>>>>>>>>
>>>>>>>> On server2 and server3 log:
>>>>>>>>
>>>>>>>> LOG:  watchdog node state changed from [DEAD] to [LOADING]
>>>>>>>> LOG:  new outbound connection to server1:9000
>>>>>>>> FATAL:  Add to watchdog cluster request is rejected by node
>>>>>>>> "server1:9000"
>>>>>>>> HINT:  check the watchdog configurations.
>>>>>>>> LOG:  Watchdog is shutting down
>>>>>>>>
>>>>>>>> while server1 log:
>>>>>>>>
>>>>>>>> LOG:  new watchdog node connection is received from "
>>>>>>>> 10.192.11.20:23277" <-- server2
>>>>>>>> LOG:  new watchdog node connection is received from "
>>>>>>>> 10.192.12.30:16596" <-- server3
>>>>>>>>
>>>>>>>> (nothing else about why it refused connection - just this thing
>>>>>>>> above)
>>>>>>>>
>>>>>>>> I believe that's enough debug info, if you could spot any problems,
>>>>>>>> please let me know.
>>>>>>>>
>>>>>>>> Thanks for the assistance so far,
>>>>>>>> -Nirav
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Oct 30, 2021 at 7:24 PM Nirav Bhatt <
>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I also suspect if following line is true for 4.1 setup:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *From Pgpool-II 4.2, now all configuration parameters are
>>>>>>>>> identical on all hosts*
>>>>>>>>>
>>>>>>>>> -Nirav
>>>>>>>>>
>>>>>>>>> On Sat, Oct 30, 2021 at 7:22 PM Nirav Bhatt <
>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello
>>>>>>>>>>
>>>>>>>>>> I am following this link, which is pgpool2 v4.1.
>>>>>>>>>>
>>>>>>>>>> https://www.pgpool.net/docs/41/en/html/example-cluster.html
>>>>>>>>>>
>>>>>>>>>> On my Ubuntu setup also that is the latest I can get:
>>>>>>>>>> *4.1.4-3.pgdg20.04+2*
>>>>>>>>>>
>>>>>>>>>> If you still insist, I will try to set it up, but I guess you are
>>>>>>>>>> quoting from this link:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://www.pgpool.net/docs/latest/en/html/example-cluster.html#EXAMPLE-CLUSTER-PGPOOL-NODE-ID
>>>>>>>>>>
>>>>>>>>>> It is a more recent version of pgpool2 than what I have.
>>>>>>>>>>
>>>>>>>>>> -Nirav
>>>>>>>>>>
>>>>>>>>>> On Sat, Oct 30, 2021 at 7:03 PM Luca Maranzano <liuk001 at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>> I think that the pgpool_node_id file is mandatory since it is
>>>>>>>>>>> used by every server to identify itself in the cluster with its node id.
>>>>>>>>>>> Where is it documented that it is not necessary?
>>>>>>>>>>>
>>>>>>>>>>> Try to configure it
>>>>>>>>>>>
>>>>>>>>>>> HTH
>>>>>>>>>>> Regards
>>>>>>>>>>> Luca
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Oct 30, 2021 at 5:42 PM Nirav Bhatt <
>>>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello
>>>>>>>>>>>> Sorry for the late answer.
>>>>>>>>>>>>
>>>>>>>>>>>> I now made pgpool.conf identical on all 3 servers, then
>>>>>>>>>>>> restarted postgresql and pgpool2 services.
>>>>>>>>>>>>
>>>>>>>>>>>> I opened ports on each server: 22, 9999/tcp, 9898/tcp,
>>>>>>>>>>>> 9000/tcp, 9694/udp, 5433, 5432
>>>>>>>>>>>>
>>>>>>>>>>>> Yet the problem persists on server2 and server3 - it can not
>>>>>>>>>>>> connect to 9000 port (on themselves)
>>>>>>>>>>>> Server1 pgpool seems to be running without problem.
>>>>>>>>>>>>
>>>>>>>>>>>> I am attaching my pgpool.conf file. Also, I am attaching
>>>>>>>>>>>> pgpool2 service logs from all 3 servers.
>>>>>>>>>>>>
>>>>>>>>>>>> 10.192.10.10 server1
>>>>>>>>>>>> 10.192.11.20 server2
>>>>>>>>>>>> 10.192.12.30 server3
>>>>>>>>>>>>
>>>>>>>>>>>> Virtual IP is 20.20.20.20
>>>>>>>>>>>>
>>>>>>>>>>>> I am running the following pgpool2 version so I do not think I
>>>>>>>>>>>> need pgpool_node_id file.
>>>>>>>>>>>>
>>>>>>>>>>>> 4.1.4-3.pgdg20.04+2
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the support,
>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Oct 27, 2021 at 8:48 AM Luca Maranzano <
>>>>>>>>>>>> liuk001 at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi, still not clear to me, from the doc:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 8.2.5. Create pgpool_node_id
>>>>>>>>>>>>>
>>>>>>>>>>>>> From Pgpool-II 4.2, now all configuration parameters are
>>>>>>>>>>>>> identical on all hosts. If watchdog feature is enabled, to
>>>>>>>>>>>>> distingusish which host is which, a pgpool_node_id file is
>>>>>>>>>>>>> required. You need to create a pgpool_node_id file and
>>>>>>>>>>>>> specify the pgpool (watchdog) node number (e.g. 0, 1, 2 ...) to identify
>>>>>>>>>>>>> pgpool (watchdog) host.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In our systems the pgpool.conf are *identical *
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Oct 26, 2021 at 8:09 PM Nirav Bhatt <
>>>>>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> The difference lies in “other” server host names.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, 26 Oct 2021 at 21.07, Luca Maranzano <
>>>>>>>>>>>>>> liuk001 at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> AFAIK the pgpool.conf file must be identical on all cluster
>>>>>>>>>>>>>>> nodes.
>>>>>>>>>>>>>>> Which are the differences in detail?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>> Luca
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 9:46 PM Nirav Bhatt <
>>>>>>>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I added all ports via ufw on server1:
>>>>>>>>>>>>>>>> 22,9999,9898,9000,9694,5433,5432
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I enabled firewall past that point.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I diffed all conf files across servers - all files are
>>>>>>>>>>>>>>>> same, including hba and .pgpass and .pcppass.
>>>>>>>>>>>>>>>> Only pgpool.conf files differ, that too in parameters like
>>>>>>>>>>>>>>>> server hosts for pgpool and watchdog, no other differences.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> No error is observed on primary server1. The last log lines
>>>>>>>>>>>>>>>> on server1 says: new watchdog node connection is received from <server2> or
>>>>>>>>>>>>>>>> <server3>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> But secondary server pgpool2 keeps exiting with:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Add to watchdog cluster request is rejected by node
>>>>>>>>>>>>>>>> "server1:9000"
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I somehow suspect it has to do with wd_authkey which I
>>>>>>>>>>>>>>>> leave empty:
>>>>>>>>>>>>>>>> https://www.sraoss.jp/pipermail/pgpool-hackers/2017-August/002497.html
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> But it is an old issue, so I can't say for sure.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 8:07 PM Luca Maranzano <
>>>>>>>>>>>>>>>> liuk001 at gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Firewall-cmd has to be used on Redhat derived flavours, on
>>>>>>>>>>>>>>>>> Ubuntu standard is ufw, so "ufw disable" should open all ports
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 6:59 PM Nirav Bhatt <
>>>>>>>>>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I ran sudo ufw status and firewall is already inactive.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Not sure what firewall-cmd did for all the pgpool ports.
>>>>>>>>>>>>>>>>>> (they are showing as list of open ports)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Should add them all via ufw also?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 7:46 PM Luca Maranzano <
>>>>>>>>>>>>>>>>>> liuk001 at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Try to disable firewall to see it it's related, all the
>>>>>>>>>>>>>>>>>>> pgpool configuration files are identical on the 3 nodes?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 6:41 PM Nirav Bhatt <
>>>>>>>>>>>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks for the response.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I restarted pgpool2 service on all 3 nodes (with some
>>>>>>>>>>>>>>>>>>>> delay, and in the order: primary, secondary1, secondary2).
>>>>>>>>>>>>>>>>>>>> My primary pgpool2 service logs do show
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> LOG:  new watchdog node connection is received from
>>>>>>>>>>>>>>>>>>>> <secondary host>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> But my secondary pgpool2 service logs (among other
>>>>>>>>>>>>>>>>>>>> normal messages) show:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> FATAL:  Add to watchdog cluster request is rejected by
>>>>>>>>>>>>>>>>>>>> node "server1:9000"
>>>>>>>>>>>>>>>>>>>> FATAL:  watchdog child process exit with fatal error.
>>>>>>>>>>>>>>>>>>>> exiting pgpool-II
>>>>>>>>>>>>>>>>>>>> LOG:  watchdog node state changed from [DEAD] to
>>>>>>>>>>>>>>>>>>>> [LOADING]
>>>>>>>>>>>>>>>>>>>> LOG:  new outbound connection to server1:9000
>>>>>>>>>>>>>>>>>>>> FATAL:  Add to watchdog cluster request is rejected by
>>>>>>>>>>>>>>>>>>>> node "server1:9000"
>>>>>>>>>>>>>>>>>>>> HINT:  check the watchdog configurations.
>>>>>>>>>>>>>>>>>>>> LOG:  Watchdog is shutting down
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I checked that server1 has 9000 port open, along with
>>>>>>>>>>>>>>>>>>>> 9999, 9898 and 9694:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> > firewall-cmd --list-ports
>>>>>>>>>>>>>>>>>>>> 9999/tcp 9898/tcp 9000/tcp 9694/udp
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> What could be going on?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 7:13 PM Luca Maranzano <
>>>>>>>>>>>>>>>>>>>> liuk001 at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> if you cannot connect to 20.20.20.20 -p 9999 it means
>>>>>>>>>>>>>>>>>>>>> that hte PGPool cluster is not well formed and that the VIP has not been
>>>>>>>>>>>>>>>>>>>>> enabled.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> You can check that with pcp_watchdog_info.
>>>>>>>>>>>>>>>>>>>>> The logs from pgpool should also help to diagnose
>>>>>>>>>>>>>>>>>>>>> the problem.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> HTH
>>>>>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>>>>> Luca
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> ---------- Forwarded message ---------
>>>>>>>>>>>>>>>>>>>>> From: <pgpool-general-request at pgpool.net>
>>>>>>>>>>>>>>>>>>>>> Date: Mon, Oct 25, 2021 at 5:00 AM
>>>>>>>>>>>>>>>>>>>>> Subject: pgpool-general Digest, Vol 120, Issue 45
>>>>>>>>>>>>>>>>>>>>> To: <pgpool-general at pgpool.net>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Send pgpool-general mailing list submissions to
>>>>>>>>>>>>>>>>>>>>>         pgpool-general at pgpool.net
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> To subscribe or unsubscribe via the World Wide Web,
>>>>>>>>>>>>>>>>>>>>> visit
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> https://www.pgpool.net/mailman/listinfo/pgpool-general
>>>>>>>>>>>>>>>>>>>>> or, via email, send a message with subject or body
>>>>>>>>>>>>>>>>>>>>> 'help' to
>>>>>>>>>>>>>>>>>>>>>         pgpool-general-request at pgpool.net
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> You can reach the person managing the list at
>>>>>>>>>>>>>>>>>>>>>         pgpool-general-owner at pgpool.net
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> When replying, please edit your Subject line so it is
>>>>>>>>>>>>>>>>>>>>> more specific
>>>>>>>>>>>>>>>>>>>>> than "Re: Contents of pgpool-general digest..."
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Today's Topics:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    1. [pgpool-general: 7822] Pgpool2 example with
>>>>>>>>>>>>>>>>>>>>> watchdog: Unable
>>>>>>>>>>>>>>>>>>>>>       to test (Nirav Bhatt)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Message: 1
>>>>>>>>>>>>>>>>>>>>> Date: Sun, 24 Oct 2021 20:18:28 +0300
>>>>>>>>>>>>>>>>>>>>> From: Nirav Bhatt <niravbhatt.cpp at gmail.com>
>>>>>>>>>>>>>>>>>>>>> To: pgpool-general at pgpool.net
>>>>>>>>>>>>>>>>>>>>> Subject: [pgpool-general: 7822] Pgpool2 example with
>>>>>>>>>>>>>>>>>>>>> watchdog: Unable
>>>>>>>>>>>>>>>>>>>>>         to test
>>>>>>>>>>>>>>>>>>>>> Message-ID:
>>>>>>>>>>>>>>>>>>>>>         <CABz_77PMu9ksuYM2QGBaAuf=m_edGoKcidmWqC=K7Rpr=
>>>>>>>>>>>>>>>>>>>>> J_ZGw at mail.gmail.com>
>>>>>>>>>>>>>>>>>>>>> Content-Type: text/plain; charset="utf-8"
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> After doing all the config (config files, passwordless
>>>>>>>>>>>>>>>>>>>>> ssh, firewall-cmd
>>>>>>>>>>>>>>>>>>>>> etc) as given in the following link, I am unable to
>>>>>>>>>>>>>>>>>>>>> test my setup. My OS is
>>>>>>>>>>>>>>>>>>>>> Ubuntu 20.04.3 LTS.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> https://www.pgpool.net/docs/41/en/html/example-cluster.html#EXAMPLE-CLUSTER-STRUCTURE
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> All the port numbers are as listed in the example.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The part in example section: 8.3.7.1 and onwards is
>>>>>>>>>>>>>>>>>>>>> not working.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> When I run on server1 (my primary):
>>>>>>>>>>>>>>>>>>>>> psql -h 20.20.20.20 -p 9999 -U pgpool postgres -c
>>>>>>>>>>>>>>>>>>>>> "show pool_nodes"
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I got error:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> psql: error: connection to server at "20.20.20.20",
>>>>>>>>>>>>>>>>>>>>> port 9999 failed:
>>>>>>>>>>>>>>>>>>>>> Connection refused
>>>>>>>>>>>>>>>>>>>>> Is the server running on that host and accepting
>>>>>>>>>>>>>>>>>>>>> TCP/IP connections?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I restarted postgresql and pgpool2 services time and
>>>>>>>>>>>>>>>>>>>>> again.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> pgpool2 service most recent logs:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> server1 (current primary):
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:01:41 ip-10-192-10-10 pgpool[104691]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:01:41: pid
>>>>>>>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is received
>>>>>>>>>>>>>>>>>>>>> from "<server1
>>>>>>>>>>>>>>>>>>>>> host>:16553"
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:01:41 ip-10-192-10-10 pgpool[104691]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:01:41: pid
>>>>>>>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is received
>>>>>>>>>>>>>>>>>>>>> from "<server1
>>>>>>>>>>>>>>>>>>>>> host>:17577"
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:04:02 ip-10-192-10-10 pgpool[104691]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:04:02: pid
>>>>>>>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is received
>>>>>>>>>>>>>>>>>>>>> from "<server2
>>>>>>>>>>>>>>>>>>>>> host>:49304"
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:04:02 ip-10-192-10-10 pgpool[104691]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:04:02: pid
>>>>>>>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is received
>>>>>>>>>>>>>>>>>>>>> from "<server2
>>>>>>>>>>>>>>>>>>>>> host>:50328"
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-10-10 pgpool[104691]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is received
>>>>>>>>>>>>>>>>>>>>> from "<server3
>>>>>>>>>>>>>>>>>>>>> host>:51864"
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-10-10 pgpool[104691]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is received
>>>>>>>>>>>>>>>>>>>>> from "<server3
>>>>>>>>>>>>>>>>>>>>> host>:52888"
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> server2 and server3 (looks like pgpool2 service is
>>>>>>>>>>>>>>>>>>>>> failing):
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125018: LOG:  setting the local watchdog node name to
>>>>>>>>>>>>>>>>>>>>> "server3:5433 Linux
>>>>>>>>>>>>>>>>>>>>> ip-<server 3 host>"
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125018: LOG:  watchdog cluster is configured with 2
>>>>>>>>>>>>>>>>>>>>> remote nodes
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125018: LOG:  watchdog remote node:0 on server1:9000
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125018: LOG:  watchdog remote node:1 on server2:9000
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125018: LOG:  interface monitoring is disabled in
>>>>>>>>>>>>>>>>>>>>> watchdog
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125018: LOG:  watchdog node state changed from [DEAD]
>>>>>>>>>>>>>>>>>>>>> to [LOADING]
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125018: LOG:  new outbound connection to server1:9000
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125018: FATAL:  Add to watchdog cluster request is
>>>>>>>>>>>>>>>>>>>>> rejected by node
>>>>>>>>>>>>>>>>>>>>> "server1:9000"
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125018: HINT:  check the watchdog configurations.
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125018: LOG:  Watchdog is shutting down
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125014]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125014: LOG:  watchdog child process with pid: 125018
>>>>>>>>>>>>>>>>>>>>> exits with status 768
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125014]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125014: FATAL:  watchdog child process exit with fatal
>>>>>>>>>>>>>>>>>>>>> error. exiting
>>>>>>>>>>>>>>>>>>>>> pgpool-II
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125019: LOG:  setting the local watchdog node name to
>>>>>>>>>>>>>>>>>>>>> "server3:5433 Linux
>>>>>>>>>>>>>>>>>>>>> ip-<server 3 host>"
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125019: LOG:  watchdog cluster is configured with 2
>>>>>>>>>>>>>>>>>>>>> remote nodes
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125019: LOG:  watchdog remote node:0 on server1:9000
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125019: LOG:  watchdog remote node:1 on server2:9000
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125019: LOG:  interface monitoring is disabled in
>>>>>>>>>>>>>>>>>>>>> watchdog
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125019: LOG:  watchdog node state changed from [DEAD]
>>>>>>>>>>>>>>>>>>>>> to [LOADING]
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125019: LOG:  new outbound connection to server1:9000
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>> 125019: LOG:  Watchdog is shutting down
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 systemd[1]:
>>>>>>>>>>>>>>>>>>>>> pgpool2.service: Main process
>>>>>>>>>>>>>>>>>>>>> exited, code=exited, status=3/NOTIMPLEMENTED
>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 systemd[1]:
>>>>>>>>>>>>>>>>>>>>> pgpool2.service: Failed with
>>>>>>>>>>>>>>>>>>>>> result 'exit-code'.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> What steps am I missing?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thank you for the assistance,
>>>>>>>>>>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>>>>>>>>>> -------------- next part --------------
>>>>>>>>>>>>>>>>>>>>> An HTML attachment was scrubbed...
>>>>>>>>>>>>>>>>>>>>> URL: <
>>>>>>>>>>>>>>>>>>>>> http://www.pgpool.net/pipermail/pgpool-general/attachments/20211024/98ab6d68/attachment-0001.htm
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Subject: Digest Footer
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>>> pgpool-general mailing list
>>>>>>>>>>>>>>>>>>>>> pgpool-general at pgpool.net
>>>>>>>>>>>>>>>>>>>>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> End of pgpool-general Digest, Vol 120, Issue 45
>>>>>>>>>>>>>>>>>>>>> ***********************************************
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20211031/e724b646/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Archive.zip
Type: application/zip
Size: 29283 bytes
Desc: not available
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20211031/e724b646/attachment-0001.zip>

More information about the pgpool-general mailing list