<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title></title>
</head>
<body>
<div name="messageBodySection">
<div dir="auto">Just FYI, I also asked one of my colleagues to test it. They set it up without Prometheus or alertmanager. Just ran the commands given in the tutorial. They observed the same behaviour.<br />
<br />
<span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">Starting Pgpool-II...</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: LOG: Backend status file /tmp/pgpool_status does not exist</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: LOG: health_check_stats_shared_memory_size: requested size: 12288</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: LOG: memory cache initialized</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: DETAIL: memcache blocks :64</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: LOG: allocating (136555320) bytes of shared memory segment</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: LOG: allocating shared memory segment of size: 136555320 </span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: LOG: health_check_stats_shared_memory_size: requested size: 12288</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: LOG: health_check_stats_shared_memory_size: requested size: 12288</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: LOG: memory cache initialized</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: DETAIL: memcache blocks :64</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: LOG: pool_discard_oid_maps: discarded memqcache oid maps</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: LOG: Setting up socket for 0.0.0.0:9999</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: LOG: Setting up socket for :::9999</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: LOG: find_primary_node_repeatedly: waiting for finding a primary node</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: LOG: find_primary_node: primary node is 0</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: LOG: find_primary_node: standby node is 1</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 54: LOG: PCP process: 54 started</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 56: LOG: process started</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 55: LOG: process started</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 57: LOG: process started</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: LOG: pgpool-II successfully started. version 4.2.3 (chichiriboshi)</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: LOG: node status[0]: 1</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:39: pid 20: LOG: node status[1]: 2</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:42: pid 52: LOG: pool_reuse_block: blockid: 0</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:18:42: pid 52: CONTEXT: while searching system catalog, When relcache is missed</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:19:49: pid 53: LOG: status_changed_time 1633591122</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:19:49: pid 53: LOG: status_changed_time 1633591122</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:19:59: pid 56: ERROR: authentication failed</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:19:59: pid 56: DETAIL: password authentication failed for user "postgres"</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:19:59: pid 56: LOG: health check failed on node 0 (timeout:0)</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:19:59: pid 56: LOG: received degenerate backend request for node_id: 0 from pid [56]</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:19:59: pid 20: LOG: Pgpool-II parent process has received failover request</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:19:59: pid 20: LOG: starting degeneration. shutdown host primary(5432)</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:19:59: pid 20: LOG: Restart all children</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:19:59: pid 20: LOG: find_primary_node_repeatedly: waiting for finding a primary node</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:19:59: pid 20: LOG: find_primary_node: standby node is 1</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:20:00: pid 20: LOG: find_primary_node: standby node is 1</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:20:01: pid 20: LOG: find_primary_node: standby node is 1</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:20:02: pid 20: LOG: find_primary_node: standby node is 1</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:20:03: pid 20: LOG: find_primary_node: standby node is 1</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:20:05: pid 20: LOG: find_primary_node: standby node is 1</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:20:06: pid 20: LOG: find_primary_node: standby node is 1</span><span style="font-size: 13px"><br /></span><span style="color:#24292e;font-family:Menlo, Monaco, Consolas, Courier New, monospace;font-size: 13px">2021-10-07 07:20:07: pid 20: LOG: find_primary_node: standby node is 1</span></div>
</div>
<div name="messageSignatureSection"><br />
Cheers
<div dir="auto">Anirudh</div>
</div>
<div name="messageReplySection">On 7 Oct 2021, 9:27 AM +0200, anirudh.goel@cern.ch, wrote:<br />
<blockquote type="cite" style="border-left-color: grey; border-left-width: thin; border-left-style: solid; margin: 5px 5px;padding-left: 10px;">
<div name="messageBodySection">
<div dir="auto">Hello Bo<br />
<br />
Thank you for your quick response.<br />
<br />
I am not using Kubernetes for the setup. I just followed this tutorial and ran the docker commands without any additional / modified configuration - <a href="https://b-peng.blogspot.com/2021/07/deploying-pgpool2-exporter-with-docker.html" target="_blank">https://b-peng.blogspot.com/2021/07/deploying-pgpool2-exporter-with-docker.html</a>.<br />
<br />
I am doing the whole setup using docker-compose and have attached the docker-compose.yml file herewith.</div>
</div>
<div name="messageSignatureSection"><br />
Cheers
<div dir="auto">Anirudh</div>
</div>
<div name="messageReplySection">On 7 Oct 2021, 7:23 AM +0200, Bo Peng <pengbo@sraoss.co.jp>, wrote:<br />
<blockquote type="cite" style="border-left-color: grey; border-left-width: thin; border-left-style: solid; margin: 5px 5px;padding-left: 10px;">Hello,<br />
<br />
<blockquote type="cite">I was following this amazing tutorial by Bo Peng to setup a pgpool cluster with pgpool exporter. The setup went fine and I was able to run the exporter successfully, but on further testing, I found something strange. The setup works fine for a few mins, but after that, even without any activity on the primary or standby, pgpool throws an "authentication failed, password authentication failed for user “postgres”” error and performs a failover. Here are the logs-<br />
<br />
Configuring Pgpool-II...<br />
No custom pgpool.conf detected. Use environment variables and default config.<br />
Starting Pgpool-II...<br />
2021-10-06 17:19:27: pid 20: LOG: Backend status file /tmp/pgpool_status does not exist<br />
2021-10-06 17:19:27: pid 20: LOG: health_check_stats_shared_memory_size: requested size: 12288<br />
2021-10-06 17:19:27: pid 20: LOG: memory cache initialized<br />
2021-10-06 17:19:27: pid 20: DETAIL: memcache blocks :64<br />
2021-10-06 17:19:27: pid 20: LOG: allocating (136555320) bytes of shared memory segment<br />
2021-10-06 17:19:27: pid 20: LOG: allocating shared memory segment of size: 136555320<br />
2021-10-06 17:19:27: pid 20: LOG: health_check_stats_shared_memory_size: requested size: 12288<br />
2021-10-06 17:19:27: pid 20: LOG: health_check_stats_shared_memory_size: requested size: 12288<br />
2021-10-06 17:19:27: pid 20: LOG: memory cache initialized<br />
2021-10-06 17:19:27: pid 20: DETAIL: memcache blocks :64<br />
2021-10-06 17:19:27: pid 20: LOG: pool_discard_oid_maps: discarded memqcache oid maps<br />
2021-10-06 17:19:27: pid 20: LOG: Setting up socket for 0.0.0.0:9999<br />
2021-10-06 17:19:27: pid 20: LOG: Setting up socket for :::9999<br />
2021-10-06 17:19:27: pid 20: LOG: find_primary_node_repeatedly: waiting for finding a primary node<br />
2021-10-06 17:19:28: pid 20: LOG: find_primary_node: primary node is 0<br />
2021-10-06 17:19:28: pid 20: LOG: find_primary_node: standby node is 1<br />
2021-10-06 17:19:28: pid 54: LOG: PCP process: 54 started<br />
2021-10-06 17:19:28: pid 57: LOG: process started<br />
2021-10-06 17:19:28: pid 56: LOG: process started<br />
2021-10-06 17:19:28: pid 55: LOG: process started<br />
2021-10-06 17:19:28: pid 20: LOG: pgpool-II successfully started. version 4.2.3 (chichiriboshi)<br />
2021-10-06 17:19:28: pid 20: LOG: node status[0]: 1<br />
2021-10-06 17:19:28: pid 20: LOG: node status[1]: 2<br />
2021-10-06 17:19:28: pid 47: LOG: pool_reuse_block: blockid: 0<br />
2021-10-06 17:19:28: pid 47: CONTEXT: while searching system catalog, When relcache is missed<br />
2021-10-06 17:19:43: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:19:43: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:19:53: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:19:53: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:20:03: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:20:03: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:20:13: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:20:13: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:20:23: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:20:23: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:20:33: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:20:33: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:20:43: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:20:43: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:20:53: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:20:53: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:21:03: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:21:03: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:21:13: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:21:13: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:21:23: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:21:23: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:21:33: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:21:33: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:21:43: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:21:43: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:21:48: pid 57: ERROR: authentication failed<br />
2021-10-06 17:21:48: pid 57: DETAIL: password authentication failed for user "postgres"<br />
2021-10-06 17:21:48: pid 57: LOG: health check failed on node 1 (timeout:0)<br /></blockquote>
<br />
Are you following this configuration example?<br />
<br />
https://www.pgpool.net/docs/latest/en/html/example-kubernetes.html<br />
<br />
It seems that the health check process failed.<br />
If you are running your cluster on K8s, you need to<br />
disable Pgpool-II's health check:<br />
<br />
- name: PGPOOL_PARAMS_HEALTH_CHECK_PERIOD<br />
value: "0"<br />
<br />
By default health check is disabled.<br />
How did you set the environment variables?<br />
<br />
<blockquote type="cite">2021-10-06 17:21:48: pid 57: LOG: received degenerate backend request for node_id: 1 from pid [57]<br />
2021-10-06 17:21:48: pid 20: LOG: Pgpool-II parent process has received failover request<br />
2021-10-06 17:21:48: pid 20: LOG: starting degeneration. shutdown host postgres_replica(5432)<br />
2021-10-06 17:21:48: pid 20: LOG: Do not restart children because we are switching over node id 1 host: postgres_replica port: 5432 and we are in streaming replication mode<br />
2021-10-06 17:21:48: pid 20: LOG: failover: set new primary node: 0<br />
2021-10-06 17:21:48: pid 20: LOG: failover: set new main node: 0<br />
failover done. shutdown host postgres_replica(5432)2021-10-06 17:21:48: pid 20: LOG: failover done. shutdown host postgres_replica(5432)<br />
2021-10-06 17:21:48: pid 55: LOG: worker process received restart request<br />
2021-10-06 17:21:49: pid 54: LOG: restart request received in pcp child process<br />
2021-10-06 17:21:49: pid 20: LOG: PCP child 54 exits with status 0 in failover()<br />
2021-10-06 17:21:49: pid 20: LOG: fork a new PCP child pid 58 in failover()<br />
2021-10-06 17:21:49: pid 20: LOG: worker child process with pid: 55 exits with status 256<br />
2021-10-06 17:21:49: pid 58: LOG: PCP process: 58 started<br />
2021-10-06 17:21:49: pid 20: LOG: fork a new worker child process with pid: 59<br />
2021-10-06 17:21:49: pid 59: LOG: process started<br />
2021-10-06 17:21:53: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:21:53: pid 47: LOG: status_changed_time 1633540908<br />
2021-10-06 17:22:03: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:22:03: pid 47: LOG: status_changed_time 1633540908<br />
2021-10-06 17:22:13: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:22:13: pid 47: LOG: status_changed_time 1633540908<br />
2021-10-06 17:22:23: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:22:23: pid 47: LOG: status_changed_time 1633540908<br />
2021-10-06 17:22:33: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:22:33: pid 47: LOG: status_changed_time 1633540908<br />
2021-10-06 17:22:43: pid 47: LOG: status_changed_time 1633540768<br />
2021-10-06 17:22:43: pid 47: LOG: status_changed_time 1633540908<br />
2021-10-06 17:22:53: pid 47: LOG: status_changed_time 1633540768<br />
<br />
As you will see in the logs, there’s no activity on my side and I have tried this multiple times and it always shows the same behaviour. I didn’t change anything in the configuration from Bo’s tutorial. Though I have added an additional Prometheus and Alertmanager in front of it. Here’s the configuration for those-<br />
<br />
prometheus:<br />
restart: unless-stopped<br />
ports:<br />
- "9090:9090"<br />
volumes:<br />
- ./prometheus/config.yml:/etc/prometheus/prometheus.yml<br />
- ./prometheus/alert_rules.yml:/etc/prometheus/alert_rules/alert_rules.yml<br />
depends_on:<br />
- pgpool2_exporter<br />
image: "prom/prometheus"<br />
<br />
alertmanager:<br />
restart: unless-stopped<br />
volumes:<br />
- ./prometheus-alertmanager/config.yml:/etc/alertmanager/alertmanager.yml<br />
depends_on:<br />
- prometheus<br />
image: "quay.io/prometheus/alertmanager"<br />
<br />
<br />
I don’t think prometheus or the alertmanager are causing any problems but I am unable to figure out why pgpool is doing a failover after several minutes of running fine.<br />
<br />
Thanks in advance!<br />
<br />
Anirudh<br /></blockquote>
<br />
<br />
--<br />
Bo Peng <pengbo@sraoss.co.jp><br />
SRA OSS, Inc. Japan<br />
http://www.sraoss.co.jp/<br /></blockquote>
</div>
_______________________________________________<br />
pgpool-hackers mailing list<br />
pgpool-hackers@pgpool.net<br />
http://www.pgpool.net/mailman/listinfo/pgpool-hackers<br /></blockquote>
</div>
</body>
</html>