[pgpool-general: 8546] Re: Issues taking a node out of a cluster

[Emond Papegaaij]

On Tue, Jan 17, 2023 at 1:30 AM Tatsuo Ishii <ishii at sraoss.co.jp> wrote:

> >> Have you removed pgpool_status file before restarting pgpool?  The
> >> file remembers the backend status along with node id hence you need to
> >> update the file. If the file does not exist upon pgpol startup, it
> >> will be automatically created.
> >>
> >
> > Yes, we remove the status file when we change the configuration of
> pgpool.
> >>From what we can see in the logs, the backend is set to down after
> synching
> > the status in the cluster. Are backends identified by their index in the
> > cluster?
>
> No, backend id is only idenfitied by pgpool.conf.
>

I meant, what is the primary identifier of backends when information about
a backend is being communicated over the cluster in messages like these
(taken from node 0):

pid 1: DETAIL:  syncing the backend states from the LEADER watchdog node
pid 11: LOG:  received the get data request from local pgpool-II on IPC
interface
pid 11: LOG:  get data request from local pgpool-II node received on IPC
interface is forwarded to leader watchdog node "172.29.30.3:5432 Linux
3b78c0e38b73"
pid 11: DETAIL:  waiting for the reply...
pid 1: LOG:  leader watchdog node "172.29.30.3:5432 Linux 3b78c0e38b73"
returned status for 3 backend nodes
pid 1: LOG:  backend:1 is set to down status
pid 1: DETAIL:  backend:1 is DOWN on cluster leader "172.29.30.3:5432 Linux
3b78c0e38b73"
pid 1: LOG:  backend:2 is set to UP status
pid 1: DETAIL:  backend:2 is UP on cluster leader "172.29.30.3:5432 Linux
3b78c0e38b73"

To me, it seems node 0 identifies the backends via their index. Also note,
that at this point, node 0 no longer knows about the former backend:1. At
this point, backend:1 is what used to be backend:2. There is no backend:2
anymore in the configuration, as there are only 2 nodes left. In the logs,
I also do not see any information about backend:0.

> After node 0 gets its new configuration, its backend1 will point
> > to node 2, while on node 2, backend1 still points to the former node 1.
> It
> > seems like this causes the backends to get mixed up and the wrong one is
> > marked down.
>
> I think so too. Each pgpool node tries to sync with leader watchdog node.
>
> I suspect there's something wrong in node 0's pgpool.conf. Can you
> share pgpool.conf of node 0? Only the "backend_*" part is enough.
>

The pgpool.conf for node 0 and 2 are different, as this all happens during
the process of updating the configuration. Node 0 gets the new
configuration (with only 2 backends) first. Notice how the configuration
for backend:0 matches, but backend:1 on node 0 matches with backend:2 on
node 2.
The configuration at node 0 is:
backend_hostname0 = '172.29.30.1'
backend_port0 = 5432
backend_weight0 = 1
backend_data_directory0 = '/var/lib/postgresql/data/'
backend_flag0 = 'ALLOW_TO_FAILOVER'
backend_application_name0 = 'tkh-server-f30c63b399'

backend_hostname1 = '172.29.30.3'
backend_port1 = 5432
backend_weight1 = 1
backend_data_directory1 = '/var/lib/postgresql/data/'
backend_flag1 = 'ALLOW_TO_FAILOVER'
backend_application_name1 = 'tkh-server-2871131ebe'


The configuration at node 2 is:
backend_hostname0 = '172.29.30.1'
backend_port0 = 5432
backend_weight0 = 1
backend_data_directory0 = '/var/lib/postgresql/data/'
backend_flag0 = 'ALLOW_TO_FAILOVER'
backend_application_name0 = 'tkh-server-f30c63b399'

backend_hostname1 = '172.29.30.2'
backend_port1 = 5432
backend_weight1 = 1
backend_data_directory1 = '/var/lib/postgresql/data/'
backend_flag1 = 'ALLOW_TO_FAILOVER'
backend_application_name1 = 'tkh-server-474c1139c7'

backend_hostname2 = '172.29.30.3'
backend_port2 = 5432
backend_weight2 = 1
backend_data_directory2 = '/var/lib/postgresql/data/'
backend_flag2 = 'ALLOW_TO_FAILOVER'
backend_application_name2 = 'tkh-server-2871131ebe'

Best regards,
Emond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20230117/e9d664ca/attachment.htm>