[pgpool-general: 7261] Re: Node status "lost" not recognized by standby PgPool

Wed Sep 2 19:16:59 JST 2020

Thanks Bo Peng,

Here are the "power off" logs with debugging enabled.

Cheers,
   - Anssi Kanninen

Sep  2 11:20:13

On Wed, 2 Sep 2020, Bo Peng wrote:

> Hi,
>
> Thank you for sharing log and connfig files.
>
> I can't see log like:
>
>   "LOG:  watchdog: lifecheck started"
>
> Watchdog lifecheck may not being performed properly.
> Sometimes watchdog lifecheck takes time,
> could you wait a while and check the watchdog status.
>
> To check watchdog lifecheck behaviour details you need to enable debug mode.
> Is possible to enable debug mode and try again?
>
> To enable debug mode (If you installed pgpool from rpm):
>
> vi /etc/sysconfig/pgpool
>
> OPTS=" -D -n" => OPTS="-d -D -n"
>
>
> On Tue, 1 Sep 2020 12:04:35 +0300 (FLE Daylight Time)
> Anssi Kanninen <anssi at iki.fi> wrote:
>
>> Here are my configs and logs for nodes 0, 1 and 2 regarding the "power
>> off" problem.
>>
>> CLEAN SHUTDOWN LOGS: pgpool-shutdown.log.nodeX
>>
>> ***** Node status from node 0 after shutdown of node 1:
>>
>> [node0]$ pcp_watchdog_info -w -h localhost
>> 3 NO centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int
>>
>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 7 STANDBY
>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 10 SHUTDOWN
>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 4 MASTER
>>
>> ***** Node status from node 2 after shutdown of node 1:
>>
>> [node2]$ pcp_watchdog_info -w -h localhost
>> 3 YES centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int
>>
>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 7 STANDBY
>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 10 SHUTDOWN
>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 4 MASTER
>>
>> =====> Everything is ok
>>
>>
>> NON-CLEAN ("power switch off") SHUTDOWN LOGS: pgpool-poweroff.log.nodeX
>>
>> ***** Node status from node 0 after power off of node 1:
>>
>> [node0]$ pcp_watchdog_info -w -h localhost
>> 3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>
>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 8 LOST
>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>
>>
>> ***** Node status from node 2 after power off of node 1:
>>
>> [node2]$ pcp_watchdog_info -w -h localhost
>> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>
>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>
>> =====> Node 2 is thinks node 1 is still in standby mode
>>
>> Cheers!
>>    - Anssi Kanninen
>>
>>
>> On Mon, 31 Aug 2020, Bo Peng wrote:
>>
>>> Hello,
>>>
>>> On Fri, 28 Aug 2020 12:27:48 +0300 (FLE Daylight Time)
>>> Anssi Kanninen <anssi at iki.fi> wrote:
>>>
>>>> Hi everyone!
>>>>
>>>> I'm having a problem with information exchange between PgPool instances. I
>>>> have 3 nodes, each containing one DB backend instance and one PgPool
>>>> instance.
>>>>
>>>> If I shut down one standby node cleanly, everything seems to go ok. The
>>>> master PgPool notices that and informs the remaining standby PgPool about
>>>> it.
>>>>
>>>> But the situation changes if a standby node just vahishes from the network
>>>> by powering it off without clean shutdown. The master PgPool marks the
>>>> node as "lost" but the remaining standby PgPool still thinks we are having
>>>> another standby PgPool. It doesn't get any information about a lost node.
>>>
>>> How did you shutdown pgpool node?
>>> Could you share the pgpool.log of each node?
>>>
>>>> Here it goes. In the example I'm checking the statuses by connecting each
>>>> node with pcp_watchdog_info . I have sorted the results by node hostname.
>>>>
>>>> Nodes are:
>>>> * ID 0 (centos8i1-int)
>>>> * ID 1 (centos8i2-int)
>>>> * ID 2 (centos8i3-int).
>>>>
>>>> ***** INITIAL SETUP *****
>>>>
>>>> $ pcp_watchdog_info -w -h centos8i1-int
>>>> 3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>>>
>>>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>>>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
>>>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>>>
>>>> $ pcp_watchdog_info -w -h centos8i2-int
>>>> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>>>
>>>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>>>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
>>>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>>>
>>>> $ pcp_watchdog_info -w -h centos8i3-int
>>>> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>>>
>>>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>>>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
>>>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>>>
>>>> ***** SHUTDOWN node ID 1 *****
>>>>
>>>> $ pcp_watchdog_info -w -h centos8i1-int
>>>> 3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>>>
>>>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>>>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 10 SHUTDOWN
>>>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>>>
>>>> $ pcp_watchdog_info -w -h centos8i3-int
>>>> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>>>
>>>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>>>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 10 SHUTDOWN
>>>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>>>
>>>> ***** RESTART node ID 1 *****
>>>>
>>>> $ pcp_watchdog_info -w -h centos8i1-int
>>>> 3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>>>
>>>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>>>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
>>>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>>>
>>>> $ pcp_watchdog_info -w -h centos8i2-int
>>>> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>>>
>>>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>>>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
>>>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>>>
>>>> $ pcp_watchdog_info -w -h centos8i3-int
>>>> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>>>
>>>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>>>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
>>>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>>>
>>>> ***** POWER OFF node ID 1 *****
>>>>
>>>> $ pcp_watchdog_info -w -h centos8i1-int
>>>> 3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>>>
>>>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>>>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 8 LOST
>>>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>>>
>>>> $ pcp_watchdog_info -w -h centos8i3-int
>>>> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int
>>>>
>>>> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER
>>>> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY
>>>> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY
>>>>
>>>>
>>>> Best regards,
>>>> Anssi Kanninen
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> anssi at iki.fi
>>>> _______________________________________________
>>>> pgpool-general mailing list
>>>> pgpool-general at pgpool.net
>>>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>>>
>>>
>>> --
>>> Bo Peng <pengbo at sraoss.co.jp>
>>> SRA OSS, Inc. Japan
>>> _______________________________________________
>>> pgpool-general mailing list
>>> pgpool-general at pgpool.net
>>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>>>
>>
>> --
>> anssi at iki.fi
>
>
> -- 
> Bo Peng <pengbo at sraoss.co.jp>
> SRA OSS, Inc. Japan
> _______________________________________________
> pgpool-general mailing list
> pgpool-general at pgpool.net
> http://www.pgpool.net/mailman/listinfo/pgpool-general
>

-- 
anssi at iki.fi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pgpool-poweroff_log_v2.zip
Type: application/x-zip-compressed
Size: 592042 bytes
Desc: 
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20200902/761af887/attachment-0001.bin>