View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000597 | Pgpool-II | Bug | public | 2020-03-13 02:11 | 2022-02-03 16:41 |
| Reporter | dcvythoulkas | Assigned To | Muhammad Usama | ||
| Priority | high | Severity | major | Reproducibility | always |
| Status | assigned | Resolution | open | ||
| Platform | PostgreSQL 12 | OS | Debian | OS Version | 10.3 |
| Product Version | 4.1.0 | ||||
| Summary | 0000597: pgpool2 fails to run the if_down_cmd and arping_cmd | ||||
| Description | This a two node installation using watchdog with the gateway as the trusted server. When pgpool2 shuts down on the master node, the secondary succsessfully gets promoted to master and raises the delegated IP. However the former master never releases the IP, so both nodes hold the IP. The only choice is to use 'ip addr del DELEGATED_IP/NETMASK dev NIC' to release the IP. Also the new master node does not notify the local network about acquiring the delegated IP and therefore clients cannot connect. | ||||
| Steps To Reproduce | - Node1 master & Node2 standby - Shutdown pgpool2 process on node1. pgpool2 on node2 promotes to master and raises delagated IP - Node1 still has the delegated IP. Use ip addr dell command to remove - Node2 has not announced taking the delegated ip and client cannot connect | ||||
| Tags | virtual ip, watchdog | ||||
|
|
Can you please share the pgpool config and log files. |
|
|
pgpool.conf is from node ...cn1, node cn2 has an equivalent configuration. The setup is deployed by Ansible. Initially, cn2 was master and cn1 was standby. cn2 pgpool2 process was shut down and cn1 took over and raised the virtual IP. However, cn2 never released the virtual IP and the local network was never notified about the change. |
|
|
Might be wrong but in your pgpool.conf file, if_up_cmd is assigning delegate IP to eth0.153:0 but if_down_cmd is trying to delete eth0.153 instead. |
|
|
i take that back. your conf is right. but that tells me my conf was wrong. hmm |
|
|
Hi Despite my best efforts, I am not able to replicate the issue. I am using centos 7 docker environment. ^C2020-04-16 17:14:53: pid 29543: LOG: Watchdog is shutting down 2020-04-16 17:14:53: pid 29673: LOG: watchdog: de-escalation started 2020-04-16 17:14:53: pid 29673: LOG: successfully released the delegate IP:"172.17.0.5" 2020-04-16 17:14:53: pid 29673: DETAIL: 'if_down_cmd' returned with success Can you confirm did you get same behaviour every time you shutdown Pgpool. Also, can you provide information on how are you running and stoping Pgpool-II. (service or from terminal) |
|
|
@tito052879 thnx for your effort :) @Muhammad Usama, I am starting/stopping pgpool-II via systemd. I am attaching 4 files terminal commands/output and corresponding pgpool.logs. Where I failover the pgpool master. The VIP changes once, and but after that it stays attached on both nodes. |
|
|
I tried the same configuration (deployed via ansible for consistency) on Debian testing(bullseye) and noticed similar behavior. The attached log is from syslog with pgpool logs combined. The pg2 node is the only pgpool node configured with trusted server. So it achieves quorum by itself. If I am not mistaken, it seems as if the watchdog process fails to acknowledge the 'if_up_cmd' ran successfully (it always does) and then completely fails to run the 'if_down_cmd' (which fails 90% of the times). The pgpool process runs as the 'postgres' OS. The 'postgres' user also sudo permissions on the ip and arping commands. When I run the 'if_up_cmd' and 'if_down_cmd' command from a postgres session, the IP is acquired and released without any problems. I have attached the syslog and the sudoer file in case there is an issue with the sudo permissions given. |
|
|
Now running on PostgreSQL 14, with Debian 11 and latest pgpool 4.1.x series, the issue no longer appears. Unfortunately we have changed many parts of our infrastructure ( Virtualization solution, network stack both in and out of target nodes) that I can't really pinpoint where the problem could reside. I will close this issue as the VIP seems to be properly released. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2020-03-13 02:11 | dcvythoulkas | New Issue | |
| 2020-03-13 02:11 | dcvythoulkas | Tag Attached: virtual ip | |
| 2020-03-13 02:11 | dcvythoulkas | Tag Attached: watchdog | |
| 2020-03-13 23:31 | Muhammad Usama | Note Added: 0003269 | |
| 2020-03-16 20:22 | dcvythoulkas | File Added: pgpool-cn2.log | |
| 2020-03-16 20:22 | dcvythoulkas | File Added: pgpool-cn1.log | |
| 2020-03-16 20:22 | dcvythoulkas | File Added: pgpool.conf | |
| 2020-03-16 20:22 | dcvythoulkas | Note Added: 0003271 | |
| 2020-04-10 06:23 | tito052879 | Note Added: 0003312 | |
| 2020-04-10 14:15 | tito052879 | Note Added: 0003313 | |
| 2020-04-17 02:43 | Muhammad Usama | Note Added: 0003331 | |
| 2020-04-21 22:26 | dcvythoulkas | File Added: cn2-pgpool.log | |
| 2020-04-21 22:26 | dcvythoulkas | File Added: cn1-pgpool.log | |
| 2020-04-21 22:26 | dcvythoulkas | File Added: cn2-terminal.log | |
| 2020-04-21 22:26 | dcvythoulkas | File Added: cn1-terminal.log | |
| 2020-04-21 22:26 | dcvythoulkas | Note Added: 0003339 | |
| 2020-04-23 20:09 | dcvythoulkas | File Added: bullseye-pg2-syslog | |
| 2020-04-23 20:09 | dcvythoulkas | File Added: pg2-sudoers | |
| 2020-04-23 20:09 | dcvythoulkas | Note Added: 0003340 | |
| 2020-05-15 10:43 | administrator | Assigned To | => Muhammad Usama |
| 2020-05-15 10:43 | administrator | Status | new => assigned |
| 2022-02-03 16:41 | dcvythoulkas | Note Added: 0003991 |
