[pgpool-general: 7217] Node status "lost" not recognized by standby PgPool

[Anssi Kanninen]

Hi everyone!

I'm having a problem with information exchange between PgPool instances. I 
have 3 nodes, each containing one DB backend instance and one PgPool 
instance.

If I shut down one standby node cleanly, everything seems to go ok. The 
master PgPool notices that and informs the remaining standby PgPool about 
it.

But the situation changes if a standby node just vahishes from the network 
by powering it off without clean shutdown. The master PgPool marks the 
node as "lost" but the remaining standby PgPool still thinks we are having 
another standby PgPool. It doesn't get any information about a lost node.

Here it goes. In the example I'm checking the statuses by connecting each 
node with pcp_watchdog_info . I have sorted the results by node hostname.

Nodes are:
* ID 0 (centos8i1-int)
* ID 1 (centos8i2-int)
* ID 2 (centos8i3-int).

***** INITIAL SETUP *****

$ pcp_watchdog_info -w -h centos8i1-int
3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int

centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY

$ pcp_watchdog_info -w -h centos8i2-int
3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int

centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY

$ pcp_watchdog_info -w -h centos8i3-int
3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int

centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY

***** SHUTDOWN node ID 1 *****

$ pcp_watchdog_info -w -h centos8i1-int
3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int

centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 10 SHUTDOWN
centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY

$ pcp_watchdog_info -w -h centos8i3-int
3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int

centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 10 SHUTDOWN
centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY

***** RESTART node ID 1 *****

$ pcp_watchdog_info -w -h centos8i1-int
3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int

centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY

$ pcp_watchdog_info -w -h centos8i2-int
3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int

centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY

$ pcp_watchdog_info -w -h centos8i3-int
3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int

centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY

***** POWER OFF node ID 1 *****

$ pcp_watchdog_info -w -h centos8i1-int
3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int

centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 8 LOST
centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY

$ pcp_watchdog_info -w -h centos8i3-int
3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int

centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY


Best regards,
Anssi Kanninen




-- 
anssi at iki.fi