[pgpool-general: 7254] Re: Node status "lost" not recognized by standby PgPool

Tue Sep 1 18:04:35 JST 2020

Here are my configs and logs for nodes 0, 1 and 2 regarding the "power 
off" problem.

CLEAN SHUTDOWN LOGS: pgpool-shutdown.log.nodeX

***** Node status from node 0 after shutdown of node 1:

[node0]$ pcp_watchdog_info -w -h localhost
3 NO centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int

centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 7 STANDBY
centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 10 SHUTDOWN
centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 4 MASTER

***** Node status from node 2 after shutdown of node 1:

[node2]$ pcp_watchdog_info -w -h localhost
3 YES centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int

centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 7 STANDBY
centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 10 SHUTDOWN
centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 4 MASTER

=====> Everything is ok

NON-CLEAN ("power switch off") SHUTDOWN LOGS: pgpool-poweroff.log.nodeX

***** Node status from node 0 after power off of node 1:

[node0]$ pcp_watchdog_info -w -h localhost
3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int

centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 8 LOST
centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY

***** Node status from node 2 after power off of node 1:

[node2]$ pcp_watchdog_info -w -h localhost
3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int

centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY

=====> Node 2 is thinks node 1 is still in standby mode

Cheers!
   - Anssi Kanninen

On Mon, 31 Aug 2020, Bo Peng wrote:

> Hello,
>
> On Fri, 28 Aug 2020 12:27:48 +0300 (FLE Daylight Time)
> Anssi Kanninen <anssi at iki.fi> wrote:
>
>> Hi everyone!
>>
>> I'm having a problem with information exchange between PgPool instances. I
>> have 3 nodes, each containing one DB backend instance and one PgPool
>> instance.
>>
>> If I shut down one standby node cleanly, everything seems to go ok. The
>> master PgPool notices that and informs the remaining standby PgPool about
>> it.
>>
>> But the situation changes if a standby node just vahishes from the network
>> by powering it off without clean shutdown. The master PgPool marks the
>> node as "lost" but the remaining standby PgPool still thinks we are having
>> another standby PgPool. It doesn't get any information about a lost node.
>
> How did you shutdown pgpool node?
> Could you share the pgpool.log of each node?
>
>> Here it goes. In the example I'm checking the statuses by connecting each
>> node with pcp_watchdog_info . I have sorted the results by node hostname.
>>
>> Nodes are:
>> * ID 0 (centos8i1-int)
>> * ID 1 (centos8i2-int)
>> * ID 2 (centos8i3-int).
>>
>> ***** INITIAL SETUP *****
>>
>> $ pcp_watchdog_info -w -h centos8i1-int
>> 3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>
>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>
>> $ pcp_watchdog_info -w -h centos8i2-int
>> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>
>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>
>> $ pcp_watchdog_info -w -h centos8i3-int
>> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>
>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>
>> ***** SHUTDOWN node ID 1 *****
>>
>> $ pcp_watchdog_info -w -h centos8i1-int
>> 3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>
>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 10 SHUTDOWN
>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>
>> $ pcp_watchdog_info -w -h centos8i3-int
>> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>
>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 10 SHUTDOWN
>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>
>> ***** RESTART node ID 1 *****
>>
>> $ pcp_watchdog_info -w -h centos8i1-int
>> 3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>
>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>
>> $ pcp_watchdog_info -w -h centos8i2-int
>> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>
>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>
>> $ pcp_watchdog_info -w -h centos8i3-int
>> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>
>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>
>> ***** POWER OFF node ID 1 *****
>>
>> $ pcp_watchdog_info -w -h centos8i1-int
>> 3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>
>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 8 LOST
>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>
>> $ pcp_watchdog_info -w -h centos8i3-int
>> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>
>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>
>>
>> Best regards,
>> Anssi Kanninen
>>
>>
>>
>>
>> --
>> anssi at iki.fi
>> _______________________________________________
>> pgpool-general mailing list
>> pgpool-general at pgpool.net
>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>
>
> -- 
> Bo Peng <pengbo at sraoss.co.jp>
> SRA OSS, Inc. Japan
> _______________________________________________
> pgpool-general mailing list
> pgpool-general at pgpool.net
> http://www.pgpool.net/mailman/listinfo/pgpool-general
>

-- 
anssi at iki.fi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pgpool.conf.node0
Type: application/octet-stream
Size: 44432 bytes
Desc: 
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20200901/0b0f0aaf/attachment-0007.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pgpool.conf.node1
Type: application/octet-stream
Size: 44432 bytes
Desc: 
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20200901/0b0f0aaf/attachment-0008.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pgpool.conf.node2
Type: application/octet-stream
Size: 44432 bytes
Desc: 
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20200901/0b0f0aaf/attachment-0009.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pgpool-poweroff.log.node0
Type: application/octet-stream
Size: 31403 bytes
Desc: 
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20200901/0b0f0aaf/attachment-0010.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pgpool-poweroff.log.node2
Type: application/octet-stream
Size: 23058 bytes
Desc: 
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20200901/0b0f0aaf/attachment-0011.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pgpool-shutdown.log.node0
Type: application/octet-stream
Size: 26692 bytes
Desc: 
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20200901/0b0f0aaf/attachment-0012.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pgpool-shutdown.log.node2
Type: application/octet-stream
Size: 24254 bytes
Desc: 
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20200901/0b0f0aaf/attachment-0013.obj>