[pgpool-general: 7844] Re: pgpool-general Digest, Vol 120, Issue 45

Luca Maranzano liuk001 at gmail.com
Mon Nov 1 01:24:07 JST 2021 

Hi Nirav,
from server2.log:
2021-10-31 14:15:54: pid 7785: WARNING:  pool key file
"/var/lib/postgresql/.pgpoolkey" has group or world acc>
you must set "chmod 600 /var/lib/postgresql/.pgpoolkey" on all servers

Oct 31 14:15:54 ip-10-192-11-20 pgpool[7789]: 2021-10-31 14:15:54: pid
7789: FATAL:  Add to watchdog cluster request is rejected by node
"server1:9000"
Oct 31 14:15:54 ip-10-192-11-20 pgpool[7789]: 2021-10-31 14:15:54: pid
7789: HINT:  check the watchdog configurations.

Log says that the watchdog request is rejected, so connection is OK but
it's surely a misconfiguration problem.

Try to increase debug level and check logs again

Cheers


On Sun, Oct 31, 2021 at 3:30 PM Nirav Bhatt <niravbhatt.cpp at gmail.com>
wrote:

> Btw - I forgot to attach the 3 pgpool.conf files (4.1, so all 3 are
> different) I use on each server.
>
> I am also attaching pgpool2 service log on server1 (primary, for which
> pgpool-primary.conf is there) and that on server2 (secondary, for which
> pgpool-secondary.conf is there)
>
> Thanks
> -Nirav
>
> On Sun, Oct 31, 2021 at 12:39 AM Nirav Bhatt <niravbhatt.cpp at gmail.com>
> wrote:
>
>> pool_key problem was only observed in detached mode i.e. pgpool -n
>> option.
>> pgpool was unable to decrypt password key somehow.
>>
>> It is not happening with systemctl start pgpool2.
>> (I do not know what's the difference though, and whether process
>> invocation could provide any debug info)
>>
>> I can't see any error in service logs when I start using systemctl.
>> So I assume things are good from password standpoint.
>>
>> You are right - Routes in AWS enable routing to specific instance using
>> virtual ip (attached to interface via runtime script - up cmd in this case)
>>
>>
>> https://aws.amazon.com/blogs/apn/making-application-failover-seamless-by-failing-over-your-private-virtual-ip-across-availability-zones/
>>
>> Regarding pgpool2, it's a pity though such a well documented, highly
>> customized & fine grained tool can't operate after all the config, and it
>> is not obvious to find the reason from the service logs.
>> (I have been on this now since more than 1.5 months, but cannot get it
>> working - my limited knowledge of linux is also playing its part)
>>
>> To see if any of my server config was wrong, lastly I even switched
>> servers i.e started server2 process first (which became the master) and
>> started server1 process later - which failed just like the earlier case
>> while connecting to server2:9000
>>
>> Will continue tomorrow about it.
>>
>> Thanks all along,
>> -Nirav
>>
>> On Sat, Oct 30, 2021 at 11:51 PM Nirav Bhatt <niravbhatt.cpp at gmail.com>
>> wrote:
>>
>>> 3 subnets in 3 AWS Availability zones for redundancy. They still share
>>> the same network.
>>> This is a perfectly valid use case. (I have read about it earlier).
>>>
>>> There seems some pool_key problem on server1, which I am investigating
>>> now.
>>>
>>> -Nirav
>>>
>>>
>>> On Sat, Oct 30, 2021 at 11:43 PM Luca Maranzano <liuk001 at gmail.com>
>>> wrote:
>>>
>>>> How different? Are the 3 nodes on 3 separate networks? If yes why?
>>>>
>>>> Il sab 30 ott 2021, 22:40 Nirav Bhatt <niravbhatt.cpp at gmail.com> ha
>>>> scritto:
>>>>
>>>>> They are different, but everything including ssh copy and ping is
>>>>> working between them.
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Oct 30, 2021 at 11:31 PM Luca Maranzano <liuk001 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> What about the netmask of the network interface on 3 nodes?
>>>>>>
>>>>>> Il sab 30 ott 2021, 22:26 Nirav Bhatt <niravbhatt.cpp at gmail.com> ha
>>>>>> scritto:
>>>>>>
>>>>>>> Hi Luca
>>>>>>>
>>>>>>> Thanks for the response.
>>>>>>>
>>>>>>> Those logs belonged to the case when I put same pgpool.conf on all 3
>>>>>>> servers, based on your suggestion.
>>>>>>> As I saw later, this should not be done for 4.1.
>>>>>>> So we should probably discard that case.
>>>>>>>
>>>>>>> With separate pgpool.conf file, it is logging the things I sent in
>>>>>>> earlier mail. (connection refused by server1:9000)
>>>>>>>
>>>>>>> Plus:
>>>>>>>
>>>>>>> server2:
>>>>>>> LOG:  setting the local watchdog node name to "server2:5433 Linux
>>>>>>> ip-10-192-11-20"
>>>>>>> LOG:  watchdog remote node:0 on server1:9000
>>>>>>> LOG:  watchdog remote node:1 on server3:9000
>>>>>>>
>>>>>>> server3:
>>>>>>> LOG:  setting the local watchdog node name to "server3:5433 Linux
>>>>>>> ip-10-192-12-30"
>>>>>>> LOG:  watchdog remote node:0 on server1:9000
>>>>>>> LOG:  watchdog remote node:1 on server2:9000
>>>>>>>
>>>>>>> Thanks,
>>>>>>> -Nirav
>>>>>>>
>>>>>>> On Sat, Oct 30, 2021 at 11:01 PM Luca Maranzano <liuk001 at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Nirav,
>>>>>>>> you're right, the node id is only from 4.2, sorry for my oversight,
>>>>>>>> I thought it was valid for versione 4.X. It's really strange that there are
>>>>>>>> still no 4.2 debian packages available for Debian and Ubuntu, 4.2 was
>>>>>>>> released in 2020.
>>>>>>>>
>>>>>>>> I took a look at the logs, I found strange this:
>>>>>>>>
>>>>>>>> server1: LOG:  setting the local node "server1:5433 Linux
>>>>>>>> ip-10-192-10-10" as watchdog cluster master    (it seems ok)
>>>>>>>> server2: LOG:  setting the local watchdog node name to "*server1:5433
>>>>>>>> *Linux ip-10-192-11-20"
>>>>>>>> server3: LOG:  setting the local watchdog node name to "
>>>>>>>> *server1:5433* Linux ip-10-192-12-30"
>>>>>>>>
>>>>>>>> why server2 and server3 are referencing themselves as server1?
>>>>>>>>
>>>>>>>> Besides that you should set the VIP on the same subnet of the
>>>>>>>> servers, for example 10.192.11.100, not with an IP from a different subnet
>>>>>>>> (20.20.20.20)
>>>>>>>> I still have to understand how parameters must be differentiated in
>>>>>>>> 4.1 if you cannot use the node id file.
>>>>>>>>
>>>>>>>> I'll try to investigate later.
>>>>>>>> Cheers,
>>>>>>>> Luca
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Oct 30, 2021 at 7:43 PM Nirav Bhatt <
>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> To reiterate, this pgpool2 4.1 setup.
>>>>>>>>>
>>>>>>>>> 1 - I have now restored my different pgpool.conf files on each
>>>>>>>>> server, as described in the link for 4.1:
>>>>>>>>> https://www.pgpool.net/docs/41/en/html/example-cluster.html
>>>>>>>>>
>>>>>>>>> They are now no longer identical.
>>>>>>>>>
>>>>>>>>> They only differ in
>>>>>>>>> - wd_hostname (this server),
>>>>>>>>> - heartbeat_destination0 & heartbeat_destination1 (other servers
>>>>>>>>> eg for server1, this is server2 and server3, and vice versa),
>>>>>>>>> - other_pgpool_hostname0 & other_pgpool_hostname1 (again, other
>>>>>>>>> servers)
>>>>>>>>>
>>>>>>>>> I also diffed other files under /etc/pgpool2 and postgres home,
>>>>>>>>> including password files, they are also same across the 3 servers.
>>>>>>>>>
>>>>>>>>> 2 - I restarted the servers just in case port change was not in
>>>>>>>>> the effect.
>>>>>>>>> I also tried telnet and I could see connection going from server2
>>>>>>>>> & server3 to server1 on 9000.
>>>>>>>>> So port or connectivity isn't issue.
>>>>>>>>>
>>>>>>>>> 3 - The issue is: WD on server1 is refusing connection to server2
>>>>>>>>> and server3 watchdogs on 9000.
>>>>>>>>>
>>>>>>>>> On server2 and server3 log:
>>>>>>>>>
>>>>>>>>> LOG:  watchdog node state changed from [DEAD] to [LOADING]
>>>>>>>>> LOG:  new outbound connection to server1:9000
>>>>>>>>> FATAL:  Add to watchdog cluster request is rejected by node
>>>>>>>>> "server1:9000"
>>>>>>>>> HINT:  check the watchdog configurations.
>>>>>>>>> LOG:  Watchdog is shutting down
>>>>>>>>>
>>>>>>>>> while server1 log:
>>>>>>>>>
>>>>>>>>> LOG:  new watchdog node connection is received from "
>>>>>>>>> 10.192.11.20:23277" <-- server2
>>>>>>>>> LOG:  new watchdog node connection is received from "
>>>>>>>>> 10.192.12.30:16596" <-- server3
>>>>>>>>>
>>>>>>>>> (nothing else about why it refused connection - just this thing
>>>>>>>>> above)
>>>>>>>>>
>>>>>>>>> I believe that's enough debug info, if you could spot any
>>>>>>>>> problems, please let me know.
>>>>>>>>>
>>>>>>>>> Thanks for the assistance so far,
>>>>>>>>> -Nirav
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Oct 30, 2021 at 7:24 PM Nirav Bhatt <
>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I also suspect if following line is true for 4.1 setup:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *From Pgpool-II 4.2, now all configuration parameters are
>>>>>>>>>> identical on all hosts*
>>>>>>>>>>
>>>>>>>>>> -Nirav
>>>>>>>>>>
>>>>>>>>>> On Sat, Oct 30, 2021 at 7:22 PM Nirav Bhatt <
>>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello
>>>>>>>>>>>
>>>>>>>>>>> I am following this link, which is pgpool2 v4.1.
>>>>>>>>>>>
>>>>>>>>>>> https://www.pgpool.net/docs/41/en/html/example-cluster.html
>>>>>>>>>>>
>>>>>>>>>>> On my Ubuntu setup also that is the latest I can get:
>>>>>>>>>>> *4.1.4-3.pgdg20.04+2*
>>>>>>>>>>>
>>>>>>>>>>> If you still insist, I will try to set it up, but I guess you
>>>>>>>>>>> are quoting from this link:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> https://www.pgpool.net/docs/latest/en/html/example-cluster.html#EXAMPLE-CLUSTER-PGPOOL-NODE-ID
>>>>>>>>>>>
>>>>>>>>>>> It is a more recent version of pgpool2 than what I have.
>>>>>>>>>>>
>>>>>>>>>>> -Nirav
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Oct 30, 2021 at 7:03 PM Luca Maranzano <
>>>>>>>>>>> liuk001 at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>> I think that the pgpool_node_id file is mandatory since it is
>>>>>>>>>>>> used by every server to identify itself in the cluster with its node id.
>>>>>>>>>>>> Where is it documented that it is not necessary?
>>>>>>>>>>>>
>>>>>>>>>>>> Try to configure it
>>>>>>>>>>>>
>>>>>>>>>>>> HTH
>>>>>>>>>>>> Regards
>>>>>>>>>>>> Luca
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Oct 30, 2021 at 5:42 PM Nirav Bhatt <
>>>>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello
>>>>>>>>>>>>> Sorry for the late answer.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I now made pgpool.conf identical on all 3 servers, then
>>>>>>>>>>>>> restarted postgresql and pgpool2 services.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I opened ports on each server: 22, 9999/tcp, 9898/tcp,
>>>>>>>>>>>>> 9000/tcp, 9694/udp, 5433, 5432
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yet the problem persists on server2 and server3 - it can not
>>>>>>>>>>>>> connect to 9000 port (on themselves)
>>>>>>>>>>>>> Server1 pgpool seems to be running without problem.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am attaching my pgpool.conf file. Also, I am attaching
>>>>>>>>>>>>> pgpool2 service logs from all 3 servers.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 10.192.10.10 server1
>>>>>>>>>>>>> 10.192.11.20 server2
>>>>>>>>>>>>> 10.192.12.30 server3
>>>>>>>>>>>>>
>>>>>>>>>>>>> Virtual IP is 20.20.20.20
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am running the following pgpool2 version so I do not think I
>>>>>>>>>>>>> need pgpool_node_id file.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 4.1.4-3.pgdg20.04+2
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the support,
>>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Oct 27, 2021 at 8:48 AM Luca Maranzano <
>>>>>>>>>>>>> liuk001 at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi, still not clear to me, from the doc:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 8.2.5. Create pgpool_node_id
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> From Pgpool-II 4.2, now all configuration parameters are
>>>>>>>>>>>>>> identical on all hosts. If watchdog feature is enabled, to
>>>>>>>>>>>>>> distingusish which host is which, a pgpool_node_id file is
>>>>>>>>>>>>>> required. You need to create a pgpool_node_id file and
>>>>>>>>>>>>>> specify the pgpool (watchdog) node number (e.g. 0, 1, 2 ...) to identify
>>>>>>>>>>>>>> pgpool (watchdog) host.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In our systems the pgpool.conf are *identical *
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Oct 26, 2021 at 8:09 PM Nirav Bhatt <
>>>>>>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The difference lies in “other” server host names.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, 26 Oct 2021 at 21.07, Luca Maranzano <
>>>>>>>>>>>>>>> liuk001 at gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> AFAIK the pgpool.conf file must be identical on all cluster
>>>>>>>>>>>>>>>> nodes.
>>>>>>>>>>>>>>>> Which are the differences in detail?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>> Luca
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 9:46 PM Nirav Bhatt <
>>>>>>>>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I added all ports via ufw on server1:
>>>>>>>>>>>>>>>>> 22,9999,9898,9000,9694,5433,5432
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I enabled firewall past that point.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I diffed all conf files across servers - all files are
>>>>>>>>>>>>>>>>> same, including hba and .pgpass and .pcppass.
>>>>>>>>>>>>>>>>> Only pgpool.conf files differ, that too in parameters like
>>>>>>>>>>>>>>>>> server hosts for pgpool and watchdog, no other differences.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> No error is observed on primary server1. The last log
>>>>>>>>>>>>>>>>> lines on server1 says: new watchdog node connection is received from
>>>>>>>>>>>>>>>>> <server2> or <server3>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> But secondary server pgpool2 keeps exiting with:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Add to watchdog cluster request is rejected by node
>>>>>>>>>>>>>>>>> "server1:9000"
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I somehow suspect it has to do with wd_authkey which I
>>>>>>>>>>>>>>>>> leave empty:
>>>>>>>>>>>>>>>>> https://www.sraoss.jp/pipermail/pgpool-hackers/2017-August/002497.html
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> But it is an old issue, so I can't say for sure.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 8:07 PM Luca Maranzano <
>>>>>>>>>>>>>>>>> liuk001 at gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Firewall-cmd has to be used on Redhat derived flavours,
>>>>>>>>>>>>>>>>>> on Ubuntu standard is ufw, so "ufw disable" should open all ports
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 6:59 PM Nirav Bhatt <
>>>>>>>>>>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I ran sudo ufw status and firewall is already inactive.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Not sure what firewall-cmd did for all the pgpool ports.
>>>>>>>>>>>>>>>>>>> (they are showing as list of open ports)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Should add them all via ufw also?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 7:46 PM Luca Maranzano <
>>>>>>>>>>>>>>>>>>> liuk001 at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Try to disable firewall to see it it's related, all the
>>>>>>>>>>>>>>>>>>>> pgpool configuration files are identical on the 3 nodes?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 6:41 PM Nirav Bhatt <
>>>>>>>>>>>>>>>>>>>> niravbhatt.cpp at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for the response.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I restarted pgpool2 service on all 3 nodes (with some
>>>>>>>>>>>>>>>>>>>>> delay, and in the order: primary, secondary1, secondary2).
>>>>>>>>>>>>>>>>>>>>> My primary pgpool2 service logs do show
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> LOG:  new watchdog node connection is received from
>>>>>>>>>>>>>>>>>>>>> <secondary host>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> But my secondary pgpool2 service logs (among other
>>>>>>>>>>>>>>>>>>>>> normal messages) show:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> FATAL:  Add to watchdog cluster request is rejected by
>>>>>>>>>>>>>>>>>>>>> node "server1:9000"
>>>>>>>>>>>>>>>>>>>>> FATAL:  watchdog child process exit with fatal error.
>>>>>>>>>>>>>>>>>>>>> exiting pgpool-II
>>>>>>>>>>>>>>>>>>>>> LOG:  watchdog node state changed from [DEAD] to
>>>>>>>>>>>>>>>>>>>>> [LOADING]
>>>>>>>>>>>>>>>>>>>>> LOG:  new outbound connection to server1:9000
>>>>>>>>>>>>>>>>>>>>> FATAL:  Add to watchdog cluster request is rejected by
>>>>>>>>>>>>>>>>>>>>> node "server1:9000"
>>>>>>>>>>>>>>>>>>>>> HINT:  check the watchdog configurations.
>>>>>>>>>>>>>>>>>>>>> LOG:  Watchdog is shutting down
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I checked that server1 has 9000 port open, along with
>>>>>>>>>>>>>>>>>>>>> 9999, 9898 and 9694:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> > firewall-cmd --list-ports
>>>>>>>>>>>>>>>>>>>>> 9999/tcp 9898/tcp 9000/tcp 9694/udp
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> What could be going on?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 25, 2021 at 7:13 PM Luca Maranzano <
>>>>>>>>>>>>>>>>>>>>> liuk001 at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> if you cannot connect to 20.20.20.20 -p 9999 it
>>>>>>>>>>>>>>>>>>>>>> means that hte PGPool cluster is not well formed and that the VIP has not
>>>>>>>>>>>>>>>>>>>>>> been enabled.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> You can check that with pcp_watchdog_info.
>>>>>>>>>>>>>>>>>>>>>> The logs from pgpool should also help to diagnose
>>>>>>>>>>>>>>>>>>>>>> the problem.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> HTH
>>>>>>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>>>>>> Luca
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ---------- Forwarded message ---------
>>>>>>>>>>>>>>>>>>>>>> From: <pgpool-general-request at pgpool.net>
>>>>>>>>>>>>>>>>>>>>>> Date: Mon, Oct 25, 2021 at 5:00 AM
>>>>>>>>>>>>>>>>>>>>>> Subject: pgpool-general Digest, Vol 120, Issue 45
>>>>>>>>>>>>>>>>>>>>>> To: <pgpool-general at pgpool.net>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Send pgpool-general mailing list submissions to
>>>>>>>>>>>>>>>>>>>>>>         pgpool-general at pgpool.net
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> To subscribe or unsubscribe via the World Wide Web,
>>>>>>>>>>>>>>>>>>>>>> visit
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> https://www.pgpool.net/mailman/listinfo/pgpool-general
>>>>>>>>>>>>>>>>>>>>>> or, via email, send a message with subject or body
>>>>>>>>>>>>>>>>>>>>>> 'help' to
>>>>>>>>>>>>>>>>>>>>>>         pgpool-general-request at pgpool.net
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> You can reach the person managing the list at
>>>>>>>>>>>>>>>>>>>>>>         pgpool-general-owner at pgpool.net
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> When replying, please edit your Subject line so it is
>>>>>>>>>>>>>>>>>>>>>> more specific
>>>>>>>>>>>>>>>>>>>>>> than "Re: Contents of pgpool-general digest..."
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Today's Topics:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    1. [pgpool-general: 7822] Pgpool2 example with
>>>>>>>>>>>>>>>>>>>>>> watchdog: Unable
>>>>>>>>>>>>>>>>>>>>>>       to test (Nirav Bhatt)
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Message: 1
>>>>>>>>>>>>>>>>>>>>>> Date: Sun, 24 Oct 2021 20:18:28 +0300
>>>>>>>>>>>>>>>>>>>>>> From: Nirav Bhatt <niravbhatt.cpp at gmail.com>
>>>>>>>>>>>>>>>>>>>>>> To: pgpool-general at pgpool.net
>>>>>>>>>>>>>>>>>>>>>> Subject: [pgpool-general: 7822] Pgpool2 example with
>>>>>>>>>>>>>>>>>>>>>> watchdog: Unable
>>>>>>>>>>>>>>>>>>>>>>         to test
>>>>>>>>>>>>>>>>>>>>>> Message-ID:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> <CABz_77PMu9ksuYM2QGBaAuf=m_edGoKcidmWqC=K7Rpr=
>>>>>>>>>>>>>>>>>>>>>> J_ZGw at mail.gmail.com>
>>>>>>>>>>>>>>>>>>>>>> Content-Type: text/plain; charset="utf-8"
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> After doing all the config (config files,
>>>>>>>>>>>>>>>>>>>>>> passwordless ssh, firewall-cmd
>>>>>>>>>>>>>>>>>>>>>> etc) as given in the following link, I am unable to
>>>>>>>>>>>>>>>>>>>>>> test my setup. My OS is
>>>>>>>>>>>>>>>>>>>>>> Ubuntu 20.04.3 LTS.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> https://www.pgpool.net/docs/41/en/html/example-cluster.html#EXAMPLE-CLUSTER-STRUCTURE
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> All the port numbers are as listed in the example.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The part in example section: 8.3.7.1 and onwards is
>>>>>>>>>>>>>>>>>>>>>> not working.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> When I run on server1 (my primary):
>>>>>>>>>>>>>>>>>>>>>> psql -h 20.20.20.20 -p 9999 -U pgpool postgres -c
>>>>>>>>>>>>>>>>>>>>>> "show pool_nodes"
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I got error:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> psql: error: connection to server at "20.20.20.20",
>>>>>>>>>>>>>>>>>>>>>> port 9999 failed:
>>>>>>>>>>>>>>>>>>>>>> Connection refused
>>>>>>>>>>>>>>>>>>>>>> Is the server running on that host and accepting
>>>>>>>>>>>>>>>>>>>>>> TCP/IP connections?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I restarted postgresql and pgpool2 services time and
>>>>>>>>>>>>>>>>>>>>>> again.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> pgpool2 service most recent logs:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> server1 (current primary):
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:01:41 ip-10-192-10-10 pgpool[104691]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:01:41: pid
>>>>>>>>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is
>>>>>>>>>>>>>>>>>>>>>> received from "<server1
>>>>>>>>>>>>>>>>>>>>>> host>:16553"
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:01:41 ip-10-192-10-10 pgpool[104691]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:01:41: pid
>>>>>>>>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is
>>>>>>>>>>>>>>>>>>>>>> received from "<server1
>>>>>>>>>>>>>>>>>>>>>> host>:17577"
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:04:02 ip-10-192-10-10 pgpool[104691]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:04:02: pid
>>>>>>>>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is
>>>>>>>>>>>>>>>>>>>>>> received from "<server2
>>>>>>>>>>>>>>>>>>>>>> host>:49304"
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:04:02 ip-10-192-10-10 pgpool[104691]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:04:02: pid
>>>>>>>>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is
>>>>>>>>>>>>>>>>>>>>>> received from "<server2
>>>>>>>>>>>>>>>>>>>>>> host>:50328"
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-10-10 pgpool[104691]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is
>>>>>>>>>>>>>>>>>>>>>> received from "<server3
>>>>>>>>>>>>>>>>>>>>>> host>:51864"
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-10-10 pgpool[104691]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 104691: LOG:  new watchdog node connection is
>>>>>>>>>>>>>>>>>>>>>> received from "<server3
>>>>>>>>>>>>>>>>>>>>>> host>:52888"
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> server2 and server3 (looks like pgpool2 service is
>>>>>>>>>>>>>>>>>>>>>> failing):
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125018: LOG:  setting the local watchdog node name to
>>>>>>>>>>>>>>>>>>>>>> "server3:5433 Linux
>>>>>>>>>>>>>>>>>>>>>> ip-<server 3 host>"
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125018: LOG:  watchdog cluster is configured with 2
>>>>>>>>>>>>>>>>>>>>>> remote nodes
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125018: LOG:  watchdog remote node:0 on server1:9000
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125018: LOG:  watchdog remote node:1 on server2:9000
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125018: LOG:  interface monitoring is disabled in
>>>>>>>>>>>>>>>>>>>>>> watchdog
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125018: LOG:  watchdog node state changed from [DEAD]
>>>>>>>>>>>>>>>>>>>>>> to [LOADING]
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125018: LOG:  new outbound connection to server1:9000
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125018: FATAL:  Add to watchdog cluster request is
>>>>>>>>>>>>>>>>>>>>>> rejected by node
>>>>>>>>>>>>>>>>>>>>>> "server1:9000"
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125018: HINT:  check the watchdog configurations.
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125018]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125018: LOG:  Watchdog is shutting down
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125014]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125014: LOG:  watchdog child process with pid: 125018
>>>>>>>>>>>>>>>>>>>>>> exits with status 768
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125014]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125014: FATAL:  watchdog child process exit with
>>>>>>>>>>>>>>>>>>>>>> fatal error. exiting
>>>>>>>>>>>>>>>>>>>>>> pgpool-II
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125019: LOG:  setting the local watchdog node name to
>>>>>>>>>>>>>>>>>>>>>> "server3:5433 Linux
>>>>>>>>>>>>>>>>>>>>>> ip-<server 3 host>"
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125019: LOG:  watchdog cluster is configured with 2
>>>>>>>>>>>>>>>>>>>>>> remote nodes
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125019: LOG:  watchdog remote node:0 on server1:9000
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125019: LOG:  watchdog remote node:1 on server2:9000
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125019: LOG:  interface monitoring is disabled in
>>>>>>>>>>>>>>>>>>>>>> watchdog
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125019: LOG:  watchdog node state changed from [DEAD]
>>>>>>>>>>>>>>>>>>>>>> to [LOADING]
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125019: LOG:  new outbound connection to server1:9000
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 pgpool[125019]:
>>>>>>>>>>>>>>>>>>>>>> 2021-10-24 17:11:12: pid
>>>>>>>>>>>>>>>>>>>>>> 125019: LOG:  Watchdog is shutting down
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 systemd[1]:
>>>>>>>>>>>>>>>>>>>>>> pgpool2.service: Main process
>>>>>>>>>>>>>>>>>>>>>> exited, code=exited, status=3/NOTIMPLEMENTED
>>>>>>>>>>>>>>>>>>>>>> Oct 24 17:11:12 ip-10-192-12-30 systemd[1]:
>>>>>>>>>>>>>>>>>>>>>> pgpool2.service: Failed with
>>>>>>>>>>>>>>>>>>>>>> result 'exit-code'.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> What steps am I missing?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thank you for the assistance,
>>>>>>>>>>>>>>>>>>>>>> -Nirav
>>>>>>>>>>>>>>>>>>>>>> -------------- next part --------------
>>>>>>>>>>>>>>>>>>>>>> An HTML attachment was scrubbed...
>>>>>>>>>>>>>>>>>>>>>> URL: <
>>>>>>>>>>>>>>>>>>>>>> http://www.pgpool.net/pipermail/pgpool-general/attachments/20211024/98ab6d68/attachment-0001.htm
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Subject: Digest Footer
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>>>> pgpool-general mailing list
>>>>>>>>>>>>>>>>>>>>>> pgpool-general at pgpool.net
>>>>>>>>>>>>>>>>>>>>>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> End of pgpool-general Digest, Vol 120, Issue 45
>>>>>>>>>>>>>>>>>>>>>> ***********************************************
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20211031/d78d35b6/attachment-0001.htm>

More information about the pgpool-general mailing list