<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div>Hello, I'm testing watchdog with 2 instances of pgpool 4.2.2 , package pgpool-II-pg12 from pgpool repository, centos 7.</div><div>I have problems in starting pgpool, in both nodes, even if I start only one, it starts and it becomes leader but then it dies. I see some errors in log, here is the complete log of one node.</div><div><br data-mce-bogus="1"></div><div><span style="font-size: 12pt;">systemd: Started Pgpool-II.</span><br></div><div><div>pgpool: 2021-03-01 11:54:14: pid 1868: LOG: health_check_stats_shared_memory_size: requested size: 12288</div><div>pgpool: 2021-03-01 11:54:14: pid 1868: LOG: memory cache initialized</div><div>pgpool: 2021-03-01 11:54:14: pid 1868: DETAIL: memcache blocks :64</div><div>pgpool: 2021-03-01 11:54:14: pid 1868: LOG: allocating (136555376) bytes of shared memory segment</div><div>pgpool: 2021-03-01 11:54:14: pid 1868: LOG: allocating shared memory segment of size: 136555376</div><div>pgpool: 2021-03-01 11:54:14: pid 1868: LOG: health_check_stats_shared_memory_size: requested size: 12288</div><div>pgpool: 2021-03-01 11:54:14: pid 1868: LOG: health_check_stats_shared_memory_size: requested size: 12288</div><div>pgpool: 2021-03-01 11:54:14: pid 1868: LOG: memory cache initialized</div><div>pgpool: 2021-03-01 11:54:14: pid 1868: DETAIL: memcache blocks :64</div><div>pgpool: 2021-03-01 11:54:14: pid 1868: LOG: pool_discard_oid_maps: discarded memqcache oid maps</div><div>pgpool: 2021-03-01 11:54:14: pid 1868: LOG: waiting for watchdog to initialize</div><div>pgpool: 2021-03-01 11:54:14: pid 1873: LOG: setting the local watchdog node name to "vm-lxpgproxy1:5432 Linux vm-lxpgproxy1"</div><div>pgpool: 2021-03-01 11:54:14: pid 1873: LOG: watchdog cluster is configured with 1 remote nodes</div><div>pgpool: 2021-03-01 11:54:14: pid 1873: LOG: watchdog remote node:0 on vm-lxpgproxy2:9000</div><div>pgpool: 2021-03-01 11:54:14: pid 1873: LOG: watchdog node state changed from [DEAD] to [LOADING]</div><div>pgpool: 2021-03-01 11:54:19: pid 1873: LOG: watchdog node state changed from [LOADING] to [JOINING]</div><div>pgpool: 2021-03-01 11:54:23: pid 1873: LOG: watchdog node state changed from [JOINING] to [INITIALIZING]</div><div>pgpool: 2021-03-01 11:54:24: pid 1873: LOG: I am the only alive node in the watchdog cluster</div><div>pgpool: 2021-03-01 11:54:24: pid 1873: HINT: skipping stand for coordinator state</div><div>pgpool: 2021-03-01 11:54:24: pid 1873: LOG: watchdog node state changed from [INITIALIZING] to [LEADER]</div><div>pgpool: 2021-03-01 11:54:24: pid 1873: LOG: I am announcing my self as leader/coordinator watchdog node</div><div>pgpool: 2021-03-01 11:54:28: pid 1873: LOG: I am the cluster leader node</div><div>pgpool: 2021-03-01 11:54:28: pid 1873: DETAIL: our declare coordinator message is accepted by all nodes</div><div>pgpool: 2021-03-01 11:54:28: pid 1873: LOG: setting the local node "vm-lxpgproxy1:5432 Linux vm-lxpgproxy1" as watchdog cluste</div><div>r leader</div><div>pgpool: 2021-03-01 11:54:28: pid 1873: LOG: I am the cluster leader node. Starting escalation process</div><div>pgpool: 2021-03-01 11:54:28: pid 1873: LOG: escalation process started with PID:1888</div><div>pgpool: 2021-03-01 11:54:28: pid 1868: LOG: watchdog process is initialized</div><div>pgpool: 2021-03-01 11:54:28: pid 1868: DETAIL: watchdog messaging data version: 1.2</div><div>pgpool: 2021-03-01 11:54:28: pid 1873: LOG: new IPC connection received</div><div>pgpool: 2021-03-01 11:54:28: pid 1873: LOG: authentication failed</div><div>pgpool: 2021-03-01 11:54:28: pid 1873: DETAIL: IPC command contains an invalid data</div><div>pgpool: 2021-03-01 11:54:28: pid 1868: WARNING: get runtime variable value from watchdog failed</div><div>pgpool: 2021-03-01 11:54:28: pid 1868: LOG: failed to get current state of local watchdog node</div><div>pgpool: 2021-03-01 11:54:28: pid 1868: DETAIL: get runtime variable value from watchdog returned no data</div><div>pgpool: 2021-03-01 11:54:28: pid 1868: LOG: Setting up socket for 0.0.0.0:5432</div><div>pgpool: 2021-03-01 11:54:28: pid 1868: LOG: Setting up socket for :::5432</div><div>pgpool: 2021-03-01 11:54:28: pid 1868: LOG: find_primary_node_repeatedly: waiting for finding a primary node</div><div>pgpool: 2021-03-01 11:54:28: pid 1888: LOG: watchdog: escalation started</div><div>pgpool: 2021-03-01 11:54:28: pid 1873: LOG: new IPC connection received</div><div>pgpool: 2021-03-01 11:54:28: pid 1873: LOG: authentication failed</div><div>pgpool: 2021-03-01 11:54:28: pid 1873: DETAIL: IPC command contains an invalid data</div><div>pgpool: 2021-03-01 11:54:28: pid 1889: FATAL: get node list command reply contains no data</div><div>pgpool: + PGPOOLS=(vm-lxpgproxy2)</div><div>pgpool: + VIP=192.168.153.230</div><div>pgpool: + DEVICE=ens32</div><div>pgpool: + for pgpool in '"${PGPOOLS[@]}"'</div><div>pgpool: + '[' vm-lxpgproxy1 = vm-lxpgproxy2 ']'</div><div>pgpool: + ssh -T -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null postgres@vm-lxpgproxy2 -i /var/lib/pgsql/.ssh/id_rsa '</div><div>pgpool: /usr/bin/sudo /sbin/ip addr del 192.168.153.230/22 dev ens32</div><div>pgpool: '</div><div>pgpool: 2021-03-01 11:54:28: pid 1868: LOG: find_primary_node: standby node is 0</div><div>pgpool: 2021-03-01 11:54:28: pid 1868: LOG: find_primary_node: primary node is 1</div><div>pgpool: 2021-03-01 11:54:28: pid 1868: LOG: find_primary_node: standby node is 2</div><div>pgpool: 2021-03-01 11:54:28: pid 1928: LOG: process started</div><div>pgpool: 2021-03-01 11:54:28: pid 1925: LOG: process started</div><div>pgpool: 2021-03-01 11:54:28: pid 1926: LOG: process started</div><div>pgpool: 2021-03-01 11:54:28: pid 1927: LOG: process started</div><div>pgpool: 2021-03-01 11:54:28: pid 1924: LOG: PCP process: 1924 started</div><div>pgpool: Warning: Permanently added 'vm-lxpgproxy2,192.168.153.237' (ECDSA) to the list of known hosts.</div><div>pgpool: 2021-03-01 11:54:28: pid 1868: LOG: pgpool-II successfully started. version 4.2.2 (chichiriboshi)</div><div>pgpool: 2021-03-01 11:54:28: pid 1868: LOG: node status[0]: 2</div><div>pgpool: 2021-03-01 11:54:28: pid 1868: LOG: node status[1]: 1</div><div>pgpool: 2021-03-01 11:54:28: pid 1868: LOG: node status[2]: 2</div><div>pgpool: 2021-03-01 11:54:28: pid 1868: LOG: watchdog lifecheck process with pid: 1889 exits with status 768</div><div>pgpool: 2021-03-01 11:54:28: pid 1868: FATAL: watchdog lifecheck process exit with fatal error. exiting pgpool-II</div><div>pgpool: 2021-03-01 11:54:28: pid 1873: LOG: new IPC connection received</div><div>pgpool: 2021-03-01 11:54:28: pid 1873: LOG: authentication failed</div><div>pgpool: 2021-03-01 11:54:28: pid 1873: DETAIL: IPC command contains an invalid data</div><div>pgpool: 2021-03-01 11:54:28: pid 1929: ERROR: get node list command reply contains no data</div><div>pgpool: 2021-03-01 11:54:28: pid 1868: LOG: shutting down</div><div>pgpool: 2021-03-01 11:54:28: pid 1873: LOG: Watchdog is shutting down</div><div>pgpool: 2021-03-01 11:54:28: pid 1873: LOG: waiting for escalation process to exit before starting de-escalation</div><div>pgpool: RTNETLINK answers: Cannot assign requested address</div><div>pgpool: + exit 0</div><div>pgpool: 2021-03-01 11:54:28: pid 1888: LOG: watchdog escalation successful</div><div>pgpool: 2021-03-01 11:54:29: pid 1873: LOG: waiting for escalation process to exit before starting de-escalation</div><div>pgpool: 2021-03-01 11:54:30: pid 1873: LOG: waiting for escalation process to exit before starting de-escalation</div><div>pgpool: 2021-03-01 11:54:31: pid 1873: LOG: waiting for escalation process to exit before starting de-escalation</div><div>pgpool: 2021-03-01 11:54:32: pid 1873: LOG: waiting for escalation process to exit before starting de-escalation</div><div>pgpool: 2021-03-01 11:54:33: pid 1888: LOG: successfully acquired the delegate IP:"192.168.153.230"</div><div>pgpool: 2021-03-01 11:54:33: pid 1888: DETAIL: 'if_up_cmd' returned with success</div><div>pgpool: 2021-03-01 11:54:33: pid 1873: LOG: escalation process does not exited in time</div><div>pgpool: 2021-03-01 11:54:33: pid 1873: DETAIL: starting the de-escalation anyway</div><div>pgpool: 2021-03-01 11:54:33: pid 1942: LOG: watchdog: de-escalation started</div><div>pgpool: 2021-03-01 11:54:33: pid 1942: LOG: successfully released the delegate IP:"192.168.153.230"</div><div>pgpool: 2021-03-01 11:54:33: pid 1942: DETAIL: 'if_down_cmd' returned with success</div><div>systemd: pgpool.service: main process exited, code=exited, status=1/FAILURE</div><div>systemd: Unit pgpool.service entered failed state.</div><div>systemd: pgpool.service failed.</div><div><br data-mce-bogus="1"></div><div>Any ideas?</div></div><div data-marker="__SIG_PRE__"><div><br data-mce-bogus="1"></div></div></div></body></html>