<div dir="ltr"><div dir="ltr">On Tue, Jan 17, 2023 at 1:30 AM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">>> Have you removed pgpool_status file before restarting pgpool? The<br>
>> file remembers the backend status along with node id hence you need to<br>
>> update the file. If the file does not exist upon pgpol startup, it<br>
>> will be automatically created.<br>
>><br>
> <br>
> Yes, we remove the status file when we change the configuration of pgpool.<br>
>>From what we can see in the logs, the backend is set to down after synching<br>
> the status in the cluster. Are backends identified by their index in the<br>
> cluster?<br>
<br>
No, backend id is only idenfitied by pgpool.conf.<br></blockquote><div><br></div><div>I meant, what is the primary identifier of backends when information about a backend is being communicated over the cluster in messages like these (taken from node 0):</div><div><br></div><div>pid 1: DETAIL: syncing the backend states from the LEADER watchdog node<br>pid 11: LOG: received the get data request from local pgpool-II on IPC interface<br>pid 11: LOG: get data request from local pgpool-II node received on IPC interface is forwarded to leader watchdog node "<a href="http://172.29.30.3:5432" target="_blank">172.29.30.3:5432</a> Linux 3b78c0e38b73"<br>pid 11: DETAIL: waiting for the reply...<br>pid 1: LOG: leader watchdog node "<a href="http://172.29.30.3:5432" target="_blank">172.29.30.3:5432</a> Linux 3b78c0e38b73" returned status for 3 backend nodes<br>pid 1: LOG: backend:1 is set to down status<br>pid 1: DETAIL: backend:1 is DOWN on cluster leader "<a href="http://172.29.30.3:5432" target="_blank">172.29.30.3:5432</a> Linux 3b78c0e38b73"<br>pid 1: LOG: backend:2 is set to UP status<br>pid 1: DETAIL: backend:2 is UP on cluster leader "<a href="http://172.29.30.3:5432" target="_blank">172.29.30.3:5432</a> Linux 3b78c0e38b73"<br></div><div><br></div><div>To me, it seems node 0 identifies the backends via their index. Also note, that at this point, node 0 no longer knows about the former backend:1. At this point, backend:1 is what used to be backend:2. There is no backend:2 anymore in the configuration, as there are only 2 nodes left. In the logs, I also do not see any information about backend:0.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> After node 0 gets its new configuration, its backend1 will point<br>
> to node 2, while on node 2, backend1 still points to the former node 1. It<br>
> seems like this causes the backends to get mixed up and the wrong one is<br>
> marked down.<br>
<br>
I think so too. Each pgpool node tries to sync with leader watchdog node.<br>
<br>
I suspect there's something wrong in node 0's pgpool.conf. Can you<br>
share pgpool.conf of node 0? Only the "backend_*" part is enough.<br></blockquote><div><br></div><div>The pgpool.conf for node 0 and 2 are different, as this all happens during the process of updating the configuration. Node 0 gets the new configuration (with only 2 backends) first. Notice how the configuration for backend:0 matches, but backend:1 on node 0 matches with backend:2 on node 2.</div><div>The configuration at node 0 is:</div><div>backend_hostname0 = '172.29.30.1'<br>backend_port0 = 5432<br>backend_weight0 = 1<br>backend_data_directory0 = '/var/lib/postgresql/data/'<br>backend_flag0 = 'ALLOW_TO_FAILOVER'<br>backend_application_name0 = 'tkh-server-f30c63b399'<br><br>backend_hostname1 = '172.29.30.3'<br>backend_port1 = 5432<br>backend_weight1 = 1<br>backend_data_directory1 = '/var/lib/postgresql/data/'<br>backend_flag1 = 'ALLOW_TO_FAILOVER'<br>backend_application_name1 = 'tkh-server-2871131ebe'<br></div><div><br></div><div><br></div><div>The configuration at node 2 is:</div><div>backend_hostname0 = '172.29.30.1'<br>backend_port0 = 5432<br>backend_weight0 = 1<br>backend_data_directory0 = '/var/lib/postgresql/data/'<br>backend_flag0 = 'ALLOW_TO_FAILOVER'<br>backend_application_name0 = 'tkh-server-f30c63b399'<br><br>backend_hostname1 = '172.29.30.2'<br>backend_port1 = 5432<br>backend_weight1 = 1<br>backend_data_directory1 = '/var/lib/postgresql/data/'<br>backend_flag1 = 'ALLOW_TO_FAILOVER'<br>backend_application_name1 = 'tkh-server-474c1139c7'<br><br>backend_hostname2 = '172.29.30.3'<br>backend_port2 = 5432<br>backend_weight2 = 1<br>backend_data_directory2 = '/var/lib/postgresql/data/'<br>backend_flag2 = 'ALLOW_TO_FAILOVER'<br>backend_application_name2 = 'tkh-server-2871131ebe'<br></div><div> <br>
</div><div>Best regards,</div><div>Emond</div></div></div>