View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000558 | Pgpool-II | Bug | public | 2019-11-18 20:51 | 2019-11-28 20:50 |
| Reporter | erkumarnitish | Assigned To | t-ishii | ||
| Priority | urgent | Severity | major | Reproducibility | always |
| Status | closed | Resolution | open | ||
| OS | centos | OS Version | 7 | ||
| Product Version | 4.1.0 | ||||
| Summary | 0000558: Watchdog Mode unable to attach Virtual IP after any PgPool Node goes down | ||||
| Description | Hi Team, We are using PgPool II 4.1.0 with 2 PostgreSQL (10.5) backend DBs. We recently upgraded our pgpool cluster from 4.0.4 to 4.1.0. We have a setup of 2 pgpool-II nodes in watchdog. Everything is working fine when both PgPool-II nodes are up. But as soon as any one of PgPool-II node (done manually by stopping service). The other PgPool-II node is unable to attach the virtual IP to itself. The status of Watchdog PgPool-II node changes from STANDBY to MASTER but it does not gets the VIP and whole traffic times out. As soon has we start the OLD MASTER it joins the cluster of watchdog its status changes from SHUTDOWN to STANDBY and then the new MASTER gets the VIP and whole traffic resumes. We don't why this is happening. This was not happening in the old builds. here are few outputs of pcp_watchdog_info --username=pgpool_admin --host=localhost --port=9898 --password -v command. Watchdog Cluster Information Total Nodes : 2 Remote Nodes : 1 Quorum state : QUORUM EXIST Alive Remote Nodes : 1 VIP up on local node : NO Master Node Name : 172.23.0.37:9999 Linux ngbpgpool1 Master Host Name : 172.23.0.37 Watchdog Node Information Node Name : 172.23.0.38:9999 Linux ngbpgpool2 Host Name : 172.23.0.38 Delegate IP : 172.23.0.65 Pgpool port : 9999 Watchdog port : 9000 Node priority : 1 Status : 7 Status Name : STANDBY Node Name : 172.23.0.37:9999 Linux ngbpgpool1 Host Name : 172.23.0.37 Delegate IP : 172.23.0.65 Pgpool port : 9999 Watchdog port : 9000 Node priority : 1 Status : 4 Status Name : MASTER ############################################################# Watchdog Cluster Information Total Nodes : 2 Remote Nodes : 1 Quorum state : NO MASTER NODE Alive Remote Nodes : 0 VIP up on local node : NO Master Node Name : Not Set Master Host Name : Not Set Watchdog Node Information Node Name : 172.23.0.38:9999 Linux ngbpgpool2 Host Name : 172.23.0.38 Delegate IP : 172.23.0.65 Pgpool port : 9999 Watchdog port : 9000 Node priority : 1 Status : 4 Status Name : MASTER Node Name : 172.23.0.37:9999 Linux ngbpgpool1 Host Name : 172.23.0.37 Delegate IP : 172.23.0.65 Pgpool port : 9999 Watchdog port : 9000 Node priority : 1 Status : 10 Status Name : SHUTDOWN ############################################################## Watchdog Cluster Information Total Nodes : 2 Remote Nodes : 1 Quorum state : QUORUM ABSENT Alive Remote Nodes : 0 VIP up on local node : NO Master Node Name : 172.23.0.38:9999 Linux ngbpgpool2 Master Host Name : 172.23.0.38 Watchdog Node Information Node Name : 172.23.0.38:9999 Linux ngbpgpool2 Host Name : 172.23.0.38 Delegate IP : 172.23.0.65 Pgpool port : 9999 Watchdog port : 9000 Node priority : 1 Status : 4 Status Name : MASTER Node Name : 172.23.0.37:9999 Linux ngbpgpool1 Host Name : 172.23.0.37 Delegate IP : 172.23.0.65 Pgpool port : 9999 Watchdog port : 9000 Node priority : 1 Status : 10 Status Name : SHUTDOWN | ||||
| Tags | pgpool-II, virtual ip, watchdog | ||||
|
|
From Pgpool-II 4.1, you do not get quorum consensus if the number of votes is half of the total nodes. This means that you will never get quorum unless both of 2 nodes are healthy. To fall back to pre 4.1 behavior (only 1 node up out of 2 nodes is enough to get quorum), you can turn on following parameter: enable_consensus_with_half_votes = on See the manual for more details. https://www.pgpool.net/docs/latest/en/html/runtime-watchdog-config.html#CONFIG-WATCHDOG-FAILOVER-BEHAVIOR |
|
|
Ohk ! That solved the issue correctly. Thanks for pointing out. I have missed that update. |
|
|
Just a small advice though. It will be great if we can mark new attributes in the latest version's documentation highlighting that this attribute is available since this version. It will really help while updating. |
|
|
> It will be great if we can mark new attributes in the latest version's documentation highlighting that > this attribute is available since this version. Thank you for the suggestion. Actually it is mentioned in the doc that the parameter is available since 4.1. (From https://www.pgpool.net/docs/latest/en/html/runtime-watchdog-config.html#CONFIG-WATCHDOG-FAILOVER-BEHAVIOR) "enable_consensus_with_half_votes is not available prior to Pgpool-II V4.1. The prior versions work as if the parameter is on. " |
|
|
May close this issue? |
|
|
Yes ! Thanks for the resolution though |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2019-11-18 20:51 | erkumarnitish | New Issue | |
| 2019-11-18 20:51 | erkumarnitish | Tag Attached: pgpool-II | |
| 2019-11-18 20:51 | erkumarnitish | Tag Attached: watchdog | |
| 2019-11-18 20:51 | erkumarnitish | Tag Attached: virtual ip | |
| 2019-11-18 21:51 | t-ishii | Note Added: 0002966 | |
| 2019-11-18 22:30 | erkumarnitish | Note Added: 0002967 | |
| 2019-11-18 22:32 | erkumarnitish | Note Added: 0002968 | |
| 2019-11-19 17:12 | t-ishii | Assigned To | => t-ishii |
| 2019-11-19 17:12 | t-ishii | Status | new => assigned |
| 2019-11-19 17:15 | t-ishii | Note Added: 0002969 | |
| 2019-11-19 17:16 | t-ishii | Note Edited: 0002969 | |
| 2019-11-19 17:17 | t-ishii | Status | assigned => feedback |
| 2019-11-28 09:33 | t-ishii | Note Added: 0002982 | |
| 2019-11-28 19:20 | erkumarnitish | Note Added: 0002983 | |
| 2019-11-28 19:20 | erkumarnitish | Status | feedback => assigned |
| 2019-11-28 20:50 | t-ishii | Status | assigned => closed |
