[pgpool-general: 7016] Re: Pgpool-II-4.1.1 and Ubuntu 20.04

[Wolf Schwurack]

Still testing this I decided to start pgpool on the second node which started watchdog. I know you told me before about having an even number a nodes to set "enable_consensus_with_half_votes = on" Which is how this is set in pgpool.conf. But I have 3 nodes setup which is uneven number of nodes. 

It seems that the parameter "enable_consensus_with_half_votes" is not working. I can set it to 'on or off' The delegate IP is only successfully acquired if the second node is running pgpool 

Here's the log output once I start pgpool on the second node. 

2020-05-13 09:11:08: pid 4103: LOG:  new watchdog node connection is received from "10.11.0.206:17581"
2020-05-13 09:11:08: pid 4103: LOG:  new node joined the cluster hostname:"pgtest-02" port:9000 pgpool_port:9999
2020-05-13 09:11:08: pid 4103: DETAIL:  Pgpool-II version:"4.1.1" watchdog messaging version: 1.1
2020-05-13 09:11:08: pid 4103: LOG:  new outbound connection to pgtest-02:9000 
2020-05-13 09:11:14: pid 4103: LOG:  adding watchdog node "pgtest-02:9999 Linux pgtest-02" to the standby list
2020-05-13 09:11:14: pid 4103: LOG:  quorum found
2020-05-13 09:11:14: pid 4103: DETAIL:  starting escalation process
2020-05-13 09:11:14: pid 4103: LOG:  escalation process started with PID:4231
2020-05-13 09:11:14: pid 4099: LOG:  Pgpool-II parent process received watchdog quorum change signal from watchdog
2020-05-13 09:11:14: pid 4103: LOG:  new IPC connection received
2020-05-13 09:11:14: pid 4099: LOG:  watchdog cluster now holds the quorum
2020-05-13 09:11:14: pid 4099: DETAIL:  updating the state of quarantine backend nodes
2020-05-13 09:11:14: pid 4103: LOG:  new IPC connection received
2020-05-13 09:11:14: pid 4231: LOG:  watchdog: escalation started
2020-05-13 09:11:21: pid 4231: LOG:  successfully acquired the delegate IP:"10.11.0.204"
2020-05-13 09:11:21: pid 4231: DETAIL:  'if_up_cmd' returned with success
2020-05-13 09:11:21: pid 4103: LOG:  watchdog escalation process with pid: 4231 exit with SUCCESS.

Then I start pgpool on the third node
2020-05-13 09:26:52: pid 4906: LOG:  new watchdog node connection is received from "10.11.0.207:47278"
2020-05-13 09:26:52: pid 4906: LOG:  new node joined the cluster hostname:"pgtest-03" port:9000 pgpool_port:9999
2020-05-13 09:26:52: pid 4906: DETAIL:  Pgpool-II version:"4.1.1" watchdog messaging version: 1.1
2020-05-13 09:26:52: pid 4906: LOG:  new outbound connection to pgtest-03:9000 
2020-05-13 09:26:58: pid 4906: LOG:  adding watchdog node "pgtest-03:9999 Linux pgtest-03" to the standby list
 
Now when I stop pgpool on third node
2020-05-13 09:34:19: pid 4906: LOG:  client socket of pgtest-03:9999 Linux pgtest-03 is closed
2020-05-13 09:34:19: pid 4906: LOG:  remote node "pgtest-03:9999 Linux pgtest-03" is shutting down
2020-05-13 09:34:19: pid 4906: LOG:  removing watchdog node "pgtest-03:9999 Linux pgtest-03" from the standby list

Watchdog is still running
root at pgtest-01:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:50:56:b3:d3:58 brd ff:ff:ff:ff:ff:ff
    inet 10.11.0.205/24 brd 10.11.0.255 scope global ens160
       valid_lft forever preferred_lft forever
    inet 10.11.0.204/24 brd 10.11.0.255 scope global secondary ens160:1
       valid_lft forever preferred_lft forever

Now when I stop pgpool on the second node
2020-05-13 09:36:01: pid 4906: LOG:  We have lost the quorum
2020-05-13 09:36:01: pid 4902: LOG:  Pgpool-II parent process received watchdog quorum change signal from watchdog
2020-05-13 09:36:01: pid 4906: LOG:  new IPC connection received
2020-05-13 09:36:01: pid 5504: LOG:  watchdog: de-escalation started
2020-05-13 09:36:01: pid 5504: LOG:  successfully released the delegate IP:"10.11.0.204"
2020-05-13 09:36:01: pid 5504: DETAIL:  'if_down_cmd' returned with success
2020-05-13 09:36:01: pid 4906: LOG:  watchdog de-escalation process with pid: 5504 exit with SUCCESS.