View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0000549Pgpool-IIBugpublic2019-10-02 01:092019-10-16 12:38
Reportercarlosmesan Assigned Topengbo  
PriorityhighSeveritymajorReproducibilityalways
Status closedResolutionopen 
Platformpgpool-2OSRed Hat Enterprise Linux ServerOS Version7.5
Product Version4.0.5 
Summary0000549: Pgpool doesn't release virtual IP when we stop Postgres service in one node
DescriptionHi,

We are trying to configure two servers with PostgreSQL 11.5 and Pgpool-II 4.0.5. We set it to delegate to a virtual IP, but when we stop Postgres in one of the instances the virtual IP is not released and the node that is promoted to primary is not attached with the virtual IP.

This is what appears when we execute 'psql -h 192.168.10.6 -p 9999 -U postgres postgres -c "show pool_nodes" -w', after stopping the node 0 (it was the primary node):

node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay | last_status_change
---------+---------------+------+--------+-----------+---------+------------+-------------------+-------------------+---------------------
0 | 192.168.10.5 | 5432 | down | 0.500000 | standby | 0 | false | 0 | 2019-10-01 17:18:40
1 | 192.168.10.12 | 5432 | up | 0.500000 | primary | 0 | true | 0 | 2019-10-01 17:18:40
(2 filas)

You can see the node 1 as primary because it has been promoted correctly, but the virtual IP is attached to the node 0 when we execute ifconfig, as you can see in the watchdog cluster info (VIP up on local node):

Watchdog Cluster Information
Total Nodes : 2
Remote Nodes : 1
Quorum state : QUORUM EXIST
Alive Remote Nodes : 1
VIP up on local node : YES
Master Node Name : 192.168.10.5:9999 Linux postgresql-0
Master Host Name : 192.168.10.5

Watchdog Node Information
Node Name : 192.168.10.5:9999 Linux postgresql-0
Host Name : 192.168.10.5
Delegate IP : 192.168.10.6
Pgpool port : 9999
Watchdog port : 9000
Node priority : 1
Status : 4
Status Name : MASTER

Node Name : 192.168.10.12:9999 Linux postgresql-1
Host Name : 192.168.10.12
Delegate IP : 192.168.10.6
Pgpool port : 9999
Watchdog port : 9000
Node priority : 1
Status : 7
Status Name : STANDBY


These are the logs of the node 0 when we stop Postgres:

Oct 1 17:18:43 postgresql-0 pgpool[12260]: [51-2] 2019-10-01 17:18:43: pid 12260: LOCATION: watchdog.c:2570
Oct 1 17:18:43 postgresql-0 pgpool[12260]: [52-1] 2019-10-01 17:18:43: pid 12260: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface
Oct 1 17:18:43 postgresql-0 pgpool[12260]: [52-2] 2019-10-01 17:18:43: pid 12260: LOCATION: watchdog.c:2491
Oct 1 17:18:43 postgresql-0 pgpool[12260]: [53-1] 2019-10-01 17:18:43: pid 12260: LOG: we have got the consensus to perform the failover
Oct 1 17:18:43 postgresql-0 pgpool[12260]: [53-2] 2019-10-01 17:18:43: pid 12260: DETAIL: 1 node(s) voted in the favor
Oct 1 17:18:43 postgresql-0 pgpool[12260]: [53-3] 2019-10-01 17:18:43: pid 12260: LOCATION: watchdog.c:2363
Oct 1 17:18:43 postgresql-0 pgpool[12259]: [98-1] 2019-10-01 17:18:43: pid 12259: LOG: Pgpool-II parent process has received failover request
Oct 1 17:18:43 postgresql-0 pgpool[12259]: [98-2] 2019-10-01 17:18:43: pid 12259: LOCATION: pgpool_main.c:1594
Oct 1 17:18:43 postgresql-0 pgpool[12260]: [54-1] 2019-10-01 17:18:43: pid 12260: LOG: new IPC connection received
Oct 1 17:18:43 postgresql-0 pgpool[12260]: [54-2] 2019-10-01 17:18:43: pid 12260: LOCATION: watchdog.c:3147
Oct 1 17:18:43 postgresql-0 pgpool[12260]: [55-1] 2019-10-01 17:18:43: pid 12260: LOG: received the failover indication from Pgpool-II on IPC interface
Oct 1 17:18:43 postgresql-0 pgpool[12260]: [55-2] 2019-10-01 17:18:43: pid 12260: LOCATION: watchdog.c:2716
Oct 1 17:18:43 postgresql-0 pgpool[12260]: [56-1] 2019-10-01 17:18:43: pid 12260: LOG: watchdog is informed of failover start by the main process
Oct 1 17:18:43 postgresql-0 pgpool[12260]: [56-2] 2019-10-01 17:18:43: pid 12260: LOCATION: watchdog.c:2784
Oct 1 17:18:43 postgresql-0 pgpool[12259]: [99-1] 2019-10-01 17:18:43: pid 12259: LOG: failover: no backends are degenerated
Oct 1 17:18:43 postgresql-0 pgpool[12259]: [99-2] 2019-10-01 17:18:43: pid 12259: LOCATION: pgpool_main.c:1913

And these are the logs of the node 1:

Oct 1 17:18:40 postgresql-1 pgpool[3063]: [25-2] 2019-10-01 17:18:40: pid 3063: LOCATION: watchdog.c:3147
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [19-1] 2019-10-01 17:18:40: pid 3062: LOG: master watchdog has performed failover
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [19-2] 2019-10-01 17:18:40: pid 3062: DETAIL: syncing the backend states from the MASTER watchdog node
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [19-3] 2019-10-01 17:18:40: pid 3062: LOCATION: pgpool_main.c:1572
Oct 1 17:18:40 postgresql-1 pgpool[3063]: [26-1] 2019-10-01 17:18:40: pid 3063: LOG: new IPC connection received
Oct 1 17:18:40 postgresql-1 pgpool[3063]: [26-2] 2019-10-01 17:18:40: pid 3063: LOCATION: watchdog.c:3147
Oct 1 17:18:40 postgresql-1 pgpool[3063]: [27-1] 2019-10-01 17:18:40: pid 3063: LOG: received the get data request from local pgpool-II on IPC interface
Oct 1 17:18:40 postgresql-1 pgpool[3063]: [27-2] 2019-10-01 17:18:40: pid 3063: LOCATION: watchdog.c:2657
Oct 1 17:18:40 postgresql-1 pgpool[3063]: [28-1] 2019-10-01 17:18:40: pid 3063: LOG: get data request from local pgpool-II node received on IPC interface is forwarded to master watchdog node "192.168.10.5:9999 Linux postgresql-0"
Oct 1 17:18:40 postgresql-1 pgpool[3063]: [28-2] 2019-10-01 17:18:40: pid 3063: DETAIL: waiting for the reply...
Oct 1 17:18:40 postgresql-1 pgpool[3063]: [28-3] 2019-10-01 17:18:40: pid 3063: LOCATION: watchdog.c:2684
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [20-1] 2019-10-01 17:18:40: pid 3062: LOG: master watchdog node "192.168.10.5:9999 Linux postgresql-0" returned status for 2 backend nodes
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [20-2] 2019-10-01 17:18:40: pid 3062: LOCATION: pgpool_main.c:4115
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [21-1] 2019-10-01 17:18:40: pid 3062: LOG: primary node:1 on master watchdog node "192.168.10.5:9999 Linux postgresql-0" is different from local primary node:0
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [21-2] 2019-10-01 17:18:40: pid 3062: LOCATION: pgpool_main.c:4126
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [22-1] 2019-10-01 17:18:40: pid 3062: LOG: backend:0 is set to down status
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [22-2] 2019-10-01 17:18:40: pid 3062: DETAIL: backend:0 is DOWN on cluster master "192.168.10.5:9999 Linux postgresql-0"
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [22-3] 2019-10-01 17:18:40: pid 3062: LOCATION: pgpool_main.c:4149
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [23-1] 2019-10-01 17:18:40: pid 3062: LOG: primary node was chenged after the sync from "192.168.10.5:9999 Linux postgresql-0"
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [23-2] 2019-10-01 17:18:40: pid 3062: DETAIL: all children needs to be restarted
Oct 1 17:18:40 postgresql-1 pgpool[3104]: [13-1] 2019-10-01 17:18:40: pid 3104: LOG: worker process received restart request
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [86-2] 2019-10-01 17:18:40: pid 3062: LOCATION: pgpool_main.c:2626
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [87-1] 2019-10-01 17:18:40: pid 3062: LOG: child process with pid: 3102 exited with success and will not be restarted
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [87-2] 2019-10-01 17:18:40: pid 3062: LOCATION: pgpool_main.c:2741
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [88-1] 2019-10-01 17:18:40: pid 3062: LOG: worker child process with pid: 3104 exits with status 256
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [88-2] 2019-10-01 17:18:40: pid 3062: LOCATION: pgpool_main.c:2626
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [89-1] 2019-10-01 17:18:40: pid 3062: LOG: fork a new worker child process with pid: 3208
Oct 1 17:18:40 postgresql-1 pgpool[3062]: [89-2] 2019-10-01 17:18:40: pid 3062: LOCATION: pgpool_main.c:2735
Oct 1 17:19:14 postgresql-1 pgpool[3064]: [9-1] 2019-10-01 17:19:14: pid 3064: LOG: watchdog: lifecheck started
Oct 1 17:19:14 postgresql-1 pgpool[3064]: [9-2] 2019-10-01 17:19:14: pid 3064: LOCATION: wd_lifecheck.c:431

Additional InformationThe configuration file of the primary and slave servers are the same and are attached.
TagsNo tags attached.

Activities

carlosmesan

2019-10-02 01:09

reporter  

pgpool.conf (38,895 bytes)

pengbo

2019-10-04 11:57

developer   ~0002909

VIP is attached to master pgpool, not to PostgreSQL.
So, If pgpool "master/standby switchover” doesn't happen even if PostgreSQL failover happens, VIP will not be reassigned.

carlosmesan

2019-10-08 18:46

reporter   ~0002914

Thanks for your help, pengbo. It seems we had a misunderstanding about Pgpool's way of operating.

Issue History

Date Modified Username Field Change
2019-10-02 01:09 carlosmesan New Issue
2019-10-02 01:09 carlosmesan File Added: pgpool.conf
2019-10-02 09:13 pengbo Assigned To => pengbo
2019-10-02 09:13 pengbo Status new => assigned
2019-10-04 11:57 pengbo Note Added: 0002909
2019-10-04 11:58 pengbo Status assigned => feedback
2019-10-08 18:46 carlosmesan Note Added: 0002914
2019-10-08 18:46 carlosmesan Status feedback => assigned
2019-10-16 12:38 pengbo Status assigned => closed