[pgpool-general: 6919] Re: Inconsistency between DB nodes with native replication

[Láznička Vladimír]

> Have you seen any failover event of node 1 and then it failback while the test? If process 29684 copied the down status of node 1 when failover happened but failed to change the local status to "up" when node 1 failed back, then this could be an expiation.

Not as far as I remember. If node 1 went down, it would always stay down until I manually used some kind of recovery (as in - I never was anything in the log about failback).

> If your test environment's network is more unstable than the data center, it is likely to have more chance of failover. Of course this is just a guess.

I asked our sysadmin to move the second DB server (node 1) to the same physical machine in virtualization where is the first DB server (node 0), so I'll see if there is any difference.

Vladimír Láznička

-----Original Message-----
From: Tatsuo Ishii <ishii at sraoss.co.jp> 
Sent: Thursday, March 5, 2020 1:18 PM
To: Láznička Vladimír <Vladimir.Laznicka at cca.cz>
Cc: pengbo at sraoss.co.jp; pgpool-general at pgpool.net
Subject: Re: [pgpool-general: 6896] Re: Inconsistency between DB nodes with native replication

> Do you think there is a possibility that only one specific child 
> process of pgpool thinks that one of the DB nodes is dead?

Yes. Pgpool-II maintains backend status (up, down ,quarantine etc.) in shared memory and also each pgpool process has local copy of it (usually each pgpool looks into the local copy status). So if for some reason the local status becomes out of sync with the shared memory status, we might see the phenomena.

Have you seen any failover event of node 1 and then it failback while the test? If process 29684 copied the down status of node 1 when failover happened but failed to change the local status to "up" when node 1 failed back, then this could be an expiation.

> Is there a way to find about it? I guess I could set the logging to 
> some debug level, but I don't even want to imagine the log volume in 
> that case, since it is already hundreds of megabytes in size.

As far as I know there's no way to find out the phenomena without turning on debug log. But, yes, it would create huge amount of log.

> On the side note I also have the test running on other environment (in 
> customer's data center) and it did not fail so far (since 25th of 
> February) so it may come from some specific setting on those servers 
> (pgpool.conf is set the same apart from IP addresses and number of 
> child processes generated at startup).

If your test environment's network is more unstable than the data center, it is likely to have more chance of failover. Of course this is just a guess.

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp