[pgpool-general-jp: 1694] Re: pgpool-iiの3.7系のフェイルオーバーについて
nakamura
nakamura ＠ ichi-holdings.com
2022年 4月 7日 (木) 14:42:38 JST
中村です
お世話になっております
彭様
返信いただき、ありがとうございます。
> 設定ファイルより、Pgpool-II は2台構成と認識しています。
> ap-001、ap-002 のいずれかが 172.20.8.10 でしょうか。
現在、ap-001のpgpool一台のみを起動させた状態での仮想環境を作成し
ap-001,db-001~003の4台で投入前のテストを行なっている状況です。
上の環境にて、use_watchdogをoffにし、テストを行なってみたのですが
状況変わらず旧プライマリがfollow_masterに来てしまう状態です…
debugを有効にしたログと、failover.shを添付いたします。
pcp_promoteの部分については、コメントアウトした状態と機能させた状態でテストしましたが
旧プライマリがfollow_masterに来てしまう状態は変化せずの
よろしくお願いいたします。
On 2022/04/07 11:46, Bo Peng wrote:
> 彭です。
>
> ご報告ありがとうございます。
>
>> この構成で、DB1を落とした時
>> follow_masterがうまく動かずにはまってしまっています…
> 事象について、了解しました。
>
>> DB1 0 172.20.8.7 master
>> DB2 1 172.20.8.8 slave
>> DB3 2 172.20.8.9 slave
>>
>> pgpool 172.20.8.10
> 設定ファイルより、Pgpool-II は2台構成と認識しています。
> ap-001、ap-002 のいずれかが 172.20.8.10 でしょうか。
>
> ----
> wd_hostname = 'ap-001'
> heartbeat_destination0 = 'ap-002'
> other_pgpool_hostname0 = 'ap-002'
> ----
>
> 可能でしたら、Watchdog を無効にして頂き、再度お試して頂けますでしょうか。
>
> use_watchdog = off
>
>
> 上記設定変更で改善されない場合には、以下の情報を共有していただけますと幸いです。
>
> - すべての Pgpool-II のログ (起動後から follow master 完了までのログ)
> - すべての Pgpool-II の pgpool.conf
> - failover.sh
>
> 以上、よろしくお願いします。
>
> On Wed, 6 Apr 2022 18:17:01 +0900
> nakamura <nakamura ＠ ichi-holdings.com> wrote:
>
>> 中村と申します。
>>
>> お世話になります
>>
>> pgpool-II
>> 3.7.18、PostgreSQLは13.5でのストリーミングレプリケーションを使用しています
>>
>> DB1 0 172.20.8.7 master
>> DB2 1 172.20.8.8 slave
>> DB3 2 172.20.8.9 slave
>>
>> pgpool 172.20.8.10
>>
>> この構成で、DB1を落とした時
>> follow_masterがうまく動かずにはまってしまっています…
>>
>> failover.shでは
>> new_primaryが1 , 172.20.8.8になったと検知され
>> dbのpromoteと、pcpでのpromoteが共に動く状態です
>> #念の為、follow_masterの部分を''にしてfailbackのみでテストしたところここまでは問題がなく動作しておりました
>>
>> ただ、follow_master側で受けた時の引数が、ダウンしたはずの
>> 172.20.8.7がプライマリとして呼び出されてしまい、follow_masterが失敗してしまう状態です
>>
>>
>> failoverの後すぐ呼び出されるのが原因かと思い、sleepなども入れたのですが
>> やはり落ちたはずのDBをprimaryとして呼び出されてしまう状態です
>>
>> failover.shについては
>> https://www.pgpool.net/docs/37/ja/html/example-cluster.html
>> を参考にpromoteをかけ、念の為pcp_promoteなどもつけるなどの処理も付け加えたり戻したりをしている状態です
>>
>>
>> 何かお気づきの点などありましたら、ご指摘いただけませんでしょうか？
>> よろしくお願いいたします。
>>
>> 以下設定内容になります
>>
>> pgpool.conf
>> -------------------------
>> follow_master_command = '/etc/pgpool/follow_master.sh %d %h %p %D %m %H
>> %M %P %r %R'
>> -------------------------
>>
>> follow_master.sh
>> -------------------------
>> NODE_ID=$1
>> NODE_HOST=$2
>> NODE_PORT=$3
>> NODE_PGDATA=$4
>> .
>> .
>> .
>>
>>
>> echo "count= $# DUMP >>> $@" >> $log
>> -------------------------
>>
>> *log
>> -------------------------
>> failed_node_id=0 new_primary=172.20.8.8 new_primary_id=1
>> count= 10 DUMP >>> 0 172.20.8.7 5432 /var/postgresql/data/ 0 172.20.8.7 0 0
>> OLD PRIMARY = 0 NEW PRIMARY = 0
>> count= 10 DUMP >>> 1 172.20.8.8 5432 /var/postgresql/data/ 0 172.20.8.7 0 0
>> count= 10 DUMP >>> 2 172.20.8.9 5432 /var/postgresql/data/ 0 172.20.8.7 0 0
>>
>> -------------------------
-------------- next part --------------
$B%F%-%9%H7A<00J30$NE:IU%U%!%$%k$rJ]4I$7$^$7$?(B...
$B%U%!%$%kL>(B: failover.sh
$B7?(B: application/x-sh
$B%5%$%:(B: 1450 $B%P%$%H(B
$B ＠ bL@(B: $BL5$7(B
URL: <http://www.pgpool.net/pipermail/pgpool-general-jp/attachments/20220407/0d81af90/attachment-0001.sh>
-------------- next part --------------
2022-04-05 23:30:02: pid 55704: DEBUG: initializing pool configuration
2022-04-05 23:30:02: pid 55704: DETAIL: num_backends: 3 total_weight: 10.000000
2022-04-05 23:30:02: pid 55704: DEBUG: initializing pool configuration
2022-04-05 23:30:02: pid 55704: DETAIL: backend 0 weight: 429496729.400000 flag: 0002
2022-04-05 23:30:02: pid 55704: DEBUG: initializing pool configuration
2022-04-05 23:30:02: pid 55704: DETAIL: backend 1 weight: 858993458.800000 flag: 0000
2022-04-05 23:30:02: pid 55704: DEBUG: initializing pool configuration
2022-04-05 23:30:02: pid 55704: DETAIL: backend 2 weight: 858993458.800000 flag: 0000
2022-04-05 23:30:02: pid 55704: LOG: Backend status file /tmp/pgpool_status does not exist
2022-04-05 23:30:02: pid 55704: DEBUG: pool_coninfo_size: num_init_children (32) * max_pool (4) * MAX_NUM_BACKENDS (128) * sizeof(ConnectionInfo) (136) = 2228224 bytes requested for shared memory
2022-04-05 23:30:02: pid 55704: DEBUG: ProcessInfo: num_init_children (32) * sizeof(ProcessInfo) (32) = 1024 bytes requested for shared memory
2022-04-05 23:30:02: pid 55704: DEBUG: Request info are: sizeof(POOL_REQUEST_INFO) 5264 bytes requested for shared memory
2022-04-05 23:30:02: pid 55704: DEBUG: Recovery management area: sizeof(int) 4 bytes requested for shared memory
2022-04-05 23:30:02: pid 55704: LOG: Setting up socket for 127.0.0.1:9999
2022-04-05 23:30:02: pid 55704: LOG: Setting up socket for ::1:9999
2022-04-05 23:30:02: pid 55704: LOG: find_primary_node_repeatedly: waiting for finding a primary node
2022-04-05 23:30:02: pid 77033: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 24713: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 27238: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 92886: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 9367: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 71424: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 34448: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 9248: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 19251: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 91789: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 72898: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 34669: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 68738: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 65556: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 79441: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 58747: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 46699: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 5604: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 36608: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 90427: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 53778: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 29937: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 30400: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 39645: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 28858: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 42332: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 810: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 29142: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 74976: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 73507: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 51291: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 12787: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 55704: DEBUG: find_primary_node: ALWAYS_MASTER flag found. Returns node id: 0
2022-04-05 23:30:02: pid 55704: LOG: pgpool-II successfully started. version 3.7.18 (amefuriboshi)
2022-04-05 23:30:02: pid 31877: DEBUG: I am health check process pid:31877 DB node id:2
2022-04-05 23:30:02: pid 4153: DEBUG: I am PCP child with pid:4153
2022-04-05 23:30:02: pid 51494: DEBUG: I am 51494
2022-04-05 23:30:02: pid 2191: DEBUG: I am health check process pid:2191 DB node id:0
2022-04-05 23:30:02: pid 36687: DEBUG: I am health check process pid:36687 DB node id:1
2022-04-05 23:30:02: pid 31877: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 51494: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 36687: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:02: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:02: pid 51494: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:02: pid 2191: DEBUG: initializing backend status
2022-04-05 23:30:02: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:02: pid 36687: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:02: pid 31877: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:02: pid 2191: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:02: pid 2191: DEBUG: pool_read: read 315 bytes from backend 0
2022-04-05 23:30:02: pid 2191: DEBUG: authenticate kind = 0
2022-04-05 23:30:02: pid 2191: DEBUG: authenticate backend: key data received
2022-04-05 23:30:02: pid 2191: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:02: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:02: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:02: pid 2191: DEBUG: pool_write: to backend: 0 kind:X
2022-04-05 23:30:02: pid 2191: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:02: pid 51494: DEBUG: pool_read: read 315 bytes from backend 0
2022-04-05 23:30:02: pid 51494: DEBUG: authenticate kind = 0
2022-04-05 23:30:02: pid 51494: DEBUG: authenticate backend: key data received
2022-04-05 23:30:02: pid 51494: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:02: pid 51494: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:02: pid 31877: DEBUG: pool_read: read 315 bytes from backend 2
2022-04-05 23:30:02: pid 31877: DEBUG: authenticate kind = 0
2022-04-05 23:30:02: pid 31877: DEBUG: authenticate backend: key data received
2022-04-05 23:30:02: pid 31877: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:02: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:02: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:02: pid 31877: DEBUG: pool_write: to backend: 2 kind:X
2022-04-05 23:30:02: pid 31877: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:02: pid 51494: DEBUG: pool_read: read 315 bytes from backend 1
2022-04-05 23:30:02: pid 51494: DEBUG: authenticate kind = 0
2022-04-05 23:30:02: pid 51494: DEBUG: authenticate backend: key data received
2022-04-05 23:30:02: pid 51494: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:02: pid 36687: DEBUG: pool_read: read 315 bytes from backend 1
2022-04-05 23:30:02: pid 36687: DEBUG: authenticate kind = 0
2022-04-05 23:30:02: pid 36687: DEBUG: authenticate backend: key data received
2022-04-05 23:30:02: pid 36687: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:02: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:02: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:02: pid 36687: DEBUG: pool_write: to backend: 1 kind:X
2022-04-05 23:30:02: pid 36687: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:02: pid 51494: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:02: pid 51494: DEBUG: pool_read: read 315 bytes from backend 2
2022-04-05 23:30:02: pid 51494: DEBUG: authenticate kind = 0
2022-04-05 23:30:02: pid 51494: DEBUG: authenticate backend: key data received
2022-04-05 23:30:02: pid 51494: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: extended:0 query:"SELECT current_setting('server_version_num')"
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_write: to backend: 0 kind:Q
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_flush_it: flush size: 50
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_read: read 78 bytes from backend 0
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'T'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received ROW DESCRIPTION ('T')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: row description: num_fileds: 1
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'D'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received DATA ROW ('D')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'C'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received COMMAND COMPLETE ('C')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'Z'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received READY FOR QUERY ('Z')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: backend 0 server version: 130006
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: extended:0 query:"SELECT pg_current_wal_lsn()"
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_write: to backend: 0 kind:Q
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_flush_it: flush size: 33
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_read: read 85 bytes from backend 0
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'T'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received ROW DESCRIPTION ('T')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: row description: num_fileds: 1
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'D'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received DATA ROW ('D')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'C'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received COMMAND COMPLETE ('C')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'Z'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received READY FOR QUERY ('Z')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: extended:0 query:"SELECT current_setting('server_version_num')"
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_write: to backend: 1 kind:Q
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_flush_it: flush size: 50
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_read: read 78 bytes from backend 1
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'T'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received ROW DESCRIPTION ('T')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: row description: num_fileds: 1
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'D'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received DATA ROW ('D')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'C'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received COMMAND COMPLETE ('C')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'Z'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received READY FOR QUERY ('Z')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: backend 1 server version: 130006
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: extended:0 query:"SELECT pg_last_wal_replay_lsn()"
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_write: to backend: 1 kind:Q
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_flush_it: flush size: 37
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_read: read 89 bytes from backend 1
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'T'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received ROW DESCRIPTION ('T')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: row description: num_fileds: 1
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'D'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received DATA ROW ('D')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'C'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received COMMAND COMPLETE ('C')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'Z'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received READY FOR QUERY ('Z')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: extended:0 query:"SELECT current_setting('server_version_num')"
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_write: to backend: 2 kind:Q
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_flush_it: flush size: 50
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_read: read 78 bytes from backend 2
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'T'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received ROW DESCRIPTION ('T')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: row description: num_fileds: 1
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'D'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received DATA ROW ('D')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'C'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received COMMAND COMPLETE ('C')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'Z'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received READY FOR QUERY ('Z')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: backend 2 server version: 130006
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: extended:0 query:"SELECT pg_last_wal_replay_lsn()"
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_write: to backend: 2 kind:Q
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_flush_it: flush size: 37
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_read: read 89 bytes from backend 2
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'T'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received ROW DESCRIPTION ('T')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: row description: num_fileds: 1
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'D'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received DATA ROW ('D')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'C'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received COMMAND COMPLETE ('C')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: kind: 'Z'
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: do_query: received READY FOR QUERY ('Z')
2022-04-05 23:30:02: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:02: pid 51494: DEBUG: pool_write: to backend: 0 kind:X
2022-04-05 23:30:02: pid 51494: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:02: pid 51494: DEBUG: pool_write: to backend: 1 kind:X
2022-04-05 23:30:02: pid 51494: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:02: pid 51494: DEBUG: pool_write: to backend: 2 kind:X
2022-04-05 23:30:02: pid 51494: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:07: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:07: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:07: pid 2191: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:07: pid 31877: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:07: pid 2191: DEBUG: pool_read: read 315 bytes from backend 0
2022-04-05 23:30:07: pid 2191: DEBUG: authenticate kind = 0
2022-04-05 23:30:07: pid 2191: DEBUG: authenticate backend: key data received
2022-04-05 23:30:07: pid 2191: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:07: pid 31877: DEBUG: pool_read: read 315 bytes from backend 2
2022-04-05 23:30:07: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:07: pid 31877: DEBUG: authenticate kind = 0
2022-04-05 23:30:07: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:07: pid 31877: DEBUG: authenticate backend: key data received
2022-04-05 23:30:07: pid 2191: DEBUG: pool_write: to backend: 0 kind:X
2022-04-05 23:30:07: pid 31877: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:07: pid 2191: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:07: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:07: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:07: pid 31877: DEBUG: pool_write: to backend: 2 kind:X
2022-04-05 23:30:07: pid 31877: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:07: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:07: pid 36687: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:07: pid 36687: DEBUG: pool_read: read 315 bytes from backend 1
2022-04-05 23:30:07: pid 36687: DEBUG: authenticate kind = 0
2022-04-05 23:30:07: pid 36687: DEBUG: authenticate backend: key data received
2022-04-05 23:30:07: pid 36687: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:07: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:07: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:07: pid 36687: DEBUG: pool_write: to backend: 1 kind:X
2022-04-05 23:30:07: pid 36687: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:12: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:12: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:12: pid 2191: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:12: pid 31877: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:12: pid 31877: DEBUG: pool_read: read 315 bytes from backend 2
2022-04-05 23:30:12: pid 2191: DEBUG: pool_read: read 315 bytes from backend 0
2022-04-05 23:30:12: pid 2191: DEBUG: authenticate kind = 0
2022-04-05 23:30:12: pid 2191: DEBUG: authenticate backend: key data received
2022-04-05 23:30:12: pid 2191: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:12: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:12: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:12: pid 2191: DEBUG: pool_write: to backend: 0 kind:X
2022-04-05 23:30:12: pid 2191: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:12: pid 31877: DEBUG: authenticate kind = 0
2022-04-05 23:30:12: pid 31877: DEBUG: authenticate backend: key data received
2022-04-05 23:30:12: pid 31877: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:12: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:12: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:12: pid 31877: DEBUG: pool_write: to backend: 2 kind:X
2022-04-05 23:30:12: pid 31877: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:12: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:12: pid 36687: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:12: pid 36687: DEBUG: pool_read: read 315 bytes from backend 1
2022-04-05 23:30:12: pid 36687: DEBUG: authenticate kind = 0
2022-04-05 23:30:12: pid 36687: DEBUG: authenticate backend: key data received
2022-04-05 23:30:12: pid 36687: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:12: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:12: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:12: pid 36687: DEBUG: pool_write: to backend: 1 kind:X
2022-04-05 23:30:12: pid 36687: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:12: pid 51494: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:12: pid 51494: DEBUG: pool_read: read 315 bytes from backend 0
2022-04-05 23:30:12: pid 51494: DEBUG: authenticate kind = 0
2022-04-05 23:30:12: pid 51494: DEBUG: authenticate backend: key data received
2022-04-05 23:30:12: pid 51494: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:12: pid 51494: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:12: pid 51494: DEBUG: pool_read: read 315 bytes from backend 1
2022-04-05 23:30:12: pid 51494: DEBUG: authenticate kind = 0
2022-04-05 23:30:12: pid 51494: DEBUG: authenticate backend: key data received
2022-04-05 23:30:12: pid 51494: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:12: pid 51494: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:12: pid 51494: DEBUG: pool_read: read 315 bytes from backend 2
2022-04-05 23:30:12: pid 51494: DEBUG: authenticate kind = 0
2022-04-05 23:30:12: pid 51494: DEBUG: authenticate backend: key data received
2022-04-05 23:30:12: pid 51494: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: extended:0 query:"SELECT pg_current_wal_lsn()"
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: pool_write: to backend: 0 kind:Q
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: pool_flush_it: flush size: 33
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: pool_read: read 85 bytes from backend 0
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: kind: 'T'
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: received ROW DESCRIPTION ('T')
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: row description: num_fileds: 1
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: kind: 'D'
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: received DATA ROW ('D')
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: kind: 'C'
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: received COMMAND COMPLETE ('C')
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: kind: 'Z'
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: received READY FOR QUERY ('Z')
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: extended:0 query:"SELECT pg_last_wal_replay_lsn()"
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: pool_write: to backend: 1 kind:Q
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: pool_flush_it: flush size: 37
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: pool_read: read 89 bytes from backend 1
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: kind: 'T'
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: received ROW DESCRIPTION ('T')
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: row description: num_fileds: 1
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: kind: 'D'
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: received DATA ROW ('D')
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: kind: 'C'
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: received COMMAND COMPLETE ('C')
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: kind: 'Z'
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: received READY FOR QUERY ('Z')
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: extended:0 query:"SELECT pg_last_wal_replay_lsn()"
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: pool_write: to backend: 2 kind:Q
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: pool_flush_it: flush size: 37
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: pool_read: read 89 bytes from backend 2
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: kind: 'T'
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: received ROW DESCRIPTION ('T')
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: row description: num_fileds: 1
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: kind: 'D'
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: received DATA ROW ('D')
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: kind: 'C'
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: received COMMAND COMPLETE ('C')
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: kind: 'Z'
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: do_query: received READY FOR QUERY ('Z')
2022-04-05 23:30:12: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:12: pid 51494: DEBUG: pool_write: to backend: 0 kind:X
2022-04-05 23:30:12: pid 51494: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:12: pid 51494: DEBUG: pool_write: to backend: 1 kind:X
2022-04-05 23:30:12: pid 51494: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:12: pid 51494: DEBUG: pool_write: to backend: 2 kind:X
2022-04-05 23:30:12: pid 51494: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:17: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:17: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:17: pid 31877: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:17: pid 2191: LOG: failed to connect to PostgreSQL server on "172.20.8.7:5432", getsockopt() detected error "Connection refused"
2022-04-05 23:30:17: pid 2191: ERROR: failed to make persistent db connection
2022-04-05 23:30:17: pid 2191: DETAIL: connection to host:"172.20.8.7:5432" failed
2022-04-05 23:30:17: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:17: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:17: pid 2191: LOG: health check retrying on DB node: 0 (round:1)
2022-04-05 23:30:17: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:17: pid 36687: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:17: pid 31877: DEBUG: pool_read: read 315 bytes from backend 2
2022-04-05 23:30:17: pid 31877: DEBUG: authenticate kind = 0
2022-04-05 23:30:17: pid 31877: DEBUG: authenticate backend: key data received
2022-04-05 23:30:17: pid 31877: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:17: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:17: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:17: pid 31877: DEBUG: pool_write: to backend: 2 kind:X
2022-04-05 23:30:17: pid 31877: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:17: pid 36687: DEBUG: pool_read: read 315 bytes from backend 1
2022-04-05 23:30:17: pid 36687: DEBUG: authenticate kind = 0
2022-04-05 23:30:17: pid 36687: DEBUG: authenticate backend: key data received
2022-04-05 23:30:17: pid 36687: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:17: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:17: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:17: pid 36687: DEBUG: pool_write: to backend: 1 kind:X
2022-04-05 23:30:17: pid 36687: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:18: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:18: pid 2191: LOG: failed to connect to PostgreSQL server on "172.20.8.7:5432", getsockopt() detected error "Connection refused"
2022-04-05 23:30:18: pid 2191: ERROR: failed to make persistent db connection
2022-04-05 23:30:18: pid 2191: DETAIL: connection to host:"172.20.8.7:5432" failed
2022-04-05 23:30:18: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:18: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:18: pid 2191: LOG: health check retrying on DB node: 0 (round:2)
2022-04-05 23:30:19: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:19: pid 2191: LOG: failed to connect to PostgreSQL server on "172.20.8.7:5432", getsockopt() detected error "Connection refused"
2022-04-05 23:30:19: pid 2191: ERROR: failed to make persistent db connection
2022-04-05 23:30:19: pid 2191: DETAIL: connection to host:"172.20.8.7:5432" failed
2022-04-05 23:30:19: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:19: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:19: pid 2191: LOG: health check retrying on DB node: 0 (round:3)
2022-04-05 23:30:20: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:20: pid 2191: LOG: failed to connect to PostgreSQL server on "172.20.8.7:5432", getsockopt() detected error "Connection refused"
2022-04-05 23:30:20: pid 2191: ERROR: failed to make persistent db connection
2022-04-05 23:30:20: pid 2191: DETAIL: connection to host:"172.20.8.7:5432" failed
2022-04-05 23:30:20: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:20: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:20: pid 2191: LOG: health check retrying on DB node: 0 (round:4)
2022-04-05 23:30:22: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:22: pid 2191: LOG: failed to connect to PostgreSQL server on "172.20.8.7:5432", getsockopt() detected error "Connection refused"
2022-04-05 23:30:22: pid 2191: ERROR: failed to make persistent db connection
2022-04-05 23:30:22: pid 2191: DETAIL: connection to host:"172.20.8.7:5432" failed
2022-04-05 23:30:22: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:22: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:22: pid 2191: LOG: health check retrying on DB node: 0 (round:5)
2022-04-05 23:30:23: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:23: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:23: pid 31877: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:23: pid 36687: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:23: pid 36687: DEBUG: pool_read: read 315 bytes from backend 1
2022-04-05 23:30:23: pid 36687: DEBUG: authenticate kind = 0
2022-04-05 23:30:23: pid 36687: DEBUG: authenticate backend: key data received
2022-04-05 23:30:23: pid 36687: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:23: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:23: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:23: pid 36687: DEBUG: pool_write: to backend: 1 kind:X
2022-04-05 23:30:23: pid 31877: DEBUG: pool_read: read 315 bytes from backend 2
2022-04-05 23:30:23: pid 31877: DEBUG: authenticate kind = 0
2022-04-05 23:30:23: pid 31877: DEBUG: authenticate backend: key data received
2022-04-05 23:30:23: pid 31877: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:23: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:23: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:23: pid 31877: DEBUG: pool_write: to backend: 2 kind:X
2022-04-05 23:30:23: pid 31877: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:23: pid 36687: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:23: pid 51494: LOG: failed to connect to PostgreSQL server on "172.20.8.7:5432", getsockopt() detected error "Connection refused"
2022-04-05 23:30:23: pid 51494: ERROR: failed to make persistent db connection
2022-04-05 23:30:23: pid 51494: DETAIL: connection to host:"172.20.8.7:5432" failed
2022-04-05 23:30:23: pid 51494: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:23: pid 51494: DEBUG: pool_read: read 315 bytes from backend 1
2022-04-05 23:30:23: pid 51494: DEBUG: authenticate kind = 0
2022-04-05 23:30:23: pid 51494: DEBUG: authenticate backend: key data received
2022-04-05 23:30:23: pid 51494: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:23: pid 51494: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:23: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:23: pid 2191: LOG: failed to connect to PostgreSQL server on "172.20.8.7:5432", getsockopt() detected error "Connection refused"
2022-04-05 23:30:23: pid 2191: ERROR: failed to make persistent db connection
2022-04-05 23:30:23: pid 2191: DETAIL: connection to host:"172.20.8.7:5432" failed
2022-04-05 23:30:23: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:23: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:23: pid 2191: LOG: health check retrying on DB node: 0 (round:6)
2022-04-05 23:30:23: pid 51494: DEBUG: pool_read: read 315 bytes from backend 2
2022-04-05 23:30:23: pid 51494: DEBUG: authenticate kind = 0
2022-04-05 23:30:23: pid 51494: DEBUG: authenticate backend: key data received
2022-04-05 23:30:23: pid 51494: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:23: pid 51494: DEBUG: pool_write: to backend: 1 kind:X
2022-04-05 23:30:23: pid 51494: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:23: pid 51494: DEBUG: pool_write: to backend: 2 kind:X
2022-04-05 23:30:23: pid 51494: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:24: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:24: pid 2191: LOG: failed to connect to PostgreSQL server on "172.20.8.7:5432", getsockopt() detected error "Connection refused"
2022-04-05 23:30:24: pid 2191: ERROR: failed to make persistent db connection
2022-04-05 23:30:24: pid 2191: DETAIL: connection to host:"172.20.8.7:5432" failed
2022-04-05 23:30:24: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:24: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:24: pid 2191: LOG: health check retrying on DB node: 0 (round:7)
2022-04-05 23:30:25: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:25: pid 2191: LOG: failed to connect to PostgreSQL server on "172.20.8.7:5432", getsockopt() detected error "Connection refused"
2022-04-05 23:30:25: pid 2191: ERROR: failed to make persistent db connection
2022-04-05 23:30:25: pid 2191: DETAIL: connection to host:"172.20.8.7:5432" failed
2022-04-05 23:30:25: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:25: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:25: pid 2191: LOG: health check retrying on DB node: 0 (round:8)
2022-04-05 23:30:26: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:26: pid 2191: LOG: failed to connect to PostgreSQL server on "172.20.8.7:5432", getsockopt() detected error "Connection refused"
2022-04-05 23:30:26: pid 2191: ERROR: failed to make persistent db connection
2022-04-05 23:30:26: pid 2191: DETAIL: connection to host:"172.20.8.7:5432" failed
2022-04-05 23:30:26: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:26: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:26: pid 2191: LOG: health check retrying on DB node: 0 (round:9)
2022-04-05 23:30:27: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:27: pid 2191: LOG: failed to connect to PostgreSQL server on "172.20.8.7:5432", getsockopt() detected error "Connection refused"
2022-04-05 23:30:27: pid 2191: ERROR: failed to make persistent db connection
2022-04-05 23:30:27: pid 2191: DETAIL: connection to host:"172.20.8.7:5432" failed
2022-04-05 23:30:27: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:27: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:27: pid 2191: LOG: health check retrying on DB node: 0 (round:10)
2022-04-05 23:30:28: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:28: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:28: pid 36687: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:28: pid 31877: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:28: pid 31877: DEBUG: pool_read: read 315 bytes from backend 2
2022-04-05 23:30:28: pid 31877: DEBUG: authenticate kind = 0
2022-04-05 23:30:28: pid 31877: DEBUG: authenticate backend: key data received
2022-04-05 23:30:28: pid 31877: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:28: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:28: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:28: pid 31877: DEBUG: pool_write: to backend: 2 kind:X
2022-04-05 23:30:28: pid 31877: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:28: pid 36687: DEBUG: pool_read: read 315 bytes from backend 1
2022-04-05 23:30:28: pid 36687: DEBUG: authenticate kind = 0
2022-04-05 23:30:28: pid 36687: DEBUG: authenticate backend: key data received
2022-04-05 23:30:28: pid 36687: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:28: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:28: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:28: pid 36687: DEBUG: pool_write: to backend: 1 kind:X
2022-04-05 23:30:28: pid 36687: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:28: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:28: pid 2191: LOG: failed to connect to PostgreSQL server on "172.20.8.7:5432", getsockopt() detected error "Connection refused"
2022-04-05 23:30:28: pid 2191: ERROR: failed to make persistent db connection
2022-04-05 23:30:28: pid 2191: DETAIL: connection to host:"172.20.8.7:5432" failed
2022-04-05 23:30:28: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:28: pid 2191: DEBUG: health check: clearing alarm
2022-04-05 23:30:28: pid 2191: LOG: health check failed on node 0 (timeout:0)
2022-04-05 23:30:28: pid 2191: LOG: received degenerate backend request for node_id: 0 from pid [2191]
2022-04-05 23:30:28: pid 2191: DEBUG: sending signal:30 to the parent process with PID:55704
2022-04-05 23:30:28: pid 55704: DEBUG: Pgpool-II parent process received SIGUSR1
2022-04-05 23:30:28: pid 55704: LOG: Pgpool-II parent process has received failover request
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler called
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kind: 1 flags: 1 node_count: 1 index:0
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: starting to select new master node
2022-04-05 23:30:28: pid 55704: LOG: starting degeneration. shutdown host 172.20.8.7(5432)
2022-04-05 23:30:28: pid 55704: DEBUG: failover/failback request details: STREAM: 1 reqkind: 1 detail: 1 node_id: 0
2022-04-05 23:30:28: pid 55704: LOG: Restart all children
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:77033
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:24713
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:27238
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:92886
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:9367
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:71424
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:34448
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:9248
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:19251
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:91789
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:72898
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:34669
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:68738
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:65556
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:79441
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:58747
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:46699
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:5604
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:36608
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:90427
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:53778
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:29937
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:30400
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:39645
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:28858
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:42332
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:810
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:29142
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:74976
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:73507
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:51291
2022-04-05 23:30:28: pid 55704: DEBUG: failover handler
2022-04-05 23:30:28: pid 55704: DETAIL: kill process with PID:12787
2022-04-05 23:30:28: pid 55704: LOG: execute command: /etc/pgpool/failover.sh 0 0 172.20.8.8 /var/postgresql/data/ 0 1
+ falling_node=0
+ old_primary=0
+ new_primary=172.20.8.8
+ pgdata=/var/postgresql/data/
+ old_node_id=0
+ new_node_id=1
+ PGHOME=/opt/postgresql-13.0
+ log=/var/log/pgpool/failover.log
+ /usr/bin/whoami
+ USER=_pgpool
+ date
+ >> /var/log/pgpool/failover.log
+ echo failed_node_id=0 new_primary=172.20.8.8 new_primary_id=1
+ >> /var/log/pgpool/failover.log
+ [ 0 = 0 ]
+ [ _pgpool = root ]
+ doas -u nakamura ssh -T nakamura ＠ 172.20.8.8 doas -u _postgresql /opt/postgresql-13.0/bin/pg_ctl promote -D /var/postgresql/data/
+ >> /var/log/pgpool/failover.log
+ 2>&1
+ pcp_promote_node -U pgpool 1 -w
+ >> /var/log/pgpool/failover.log
+ 2>&1
2022-04-05 23:30:29: pid 4153: LOG: forked new pcp worker, pid=57531 socket=6
2022-04-05 23:30:29: pid 57531: DEBUG: I am PCP worker child with pid:57531
2022-04-05 23:30:29: pid 57531: DEBUG: initializing backend status
2022-04-05 23:30:29: pid 57531: DEBUG: received PCP packet
2022-04-05 23:30:29: pid 57531: DETAIL: PCP packet type of service 'M'
2022-04-05 23:30:29: pid 57531: DEBUG: PCP: sending md5 salt to client
2022-04-05 23:30:29: pid 57531: DEBUG: received PCP packet
2022-04-05 23:30:29: pid 57531: DETAIL: PCP packet type of service 'R'
2022-04-05 23:30:29: pid 57531: DEBUG: PCP: processing authentication request
2022-04-05 23:30:29: pid 57531: DETAIL: authentication OK
2022-04-05 23:30:29: pid 57531: DEBUG: received PCP packet
2022-04-05 23:30:29: pid 57531: DETAIL: PCP packet type of service 'J'
2022-04-05 23:30:29: pid 57531: DEBUG: PCP: processing promote node
2022-04-05 23:30:29: pid 57531: DETAIL: promoting Node ID 1
2022-04-05 23:30:29: pid 57531: LOG: received promote backend request for node_id: 1 from pid [57531]
+ sleep 5
2022-04-05 23:30:29: pid 57531: FATAL: unable to read from client
2022-04-05 23:30:29: pid 57531: DETAIL: pcp_read failed with error : "Undefined error: 0"
2022-04-05 23:30:29: pid 4153: DEBUG: PCP child reaper handler
2022-04-05 23:30:29: pid 4153: LOG: PCP process with pid: 57531 exit with SUCCESS.
2022-04-05 23:30:29: pid 4153: LOG: PCP process with pid: 57531 exits with status 256
2022-04-05 23:30:33: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:33: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:33: pid 31877: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:33: pid 36687: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:33: pid 36687: DEBUG: pool_read: read 315 bytes from backend 1
2022-04-05 23:30:33: pid 36687: DEBUG: authenticate kind = 0
2022-04-05 23:30:33: pid 36687: DEBUG: authenticate backend: key data received
2022-04-05 23:30:33: pid 36687: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:33: pid 31877: DEBUG: pool_read: read 315 bytes from backend 2
2022-04-05 23:30:33: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:33: pid 31877: DEBUG: authenticate kind = 0
2022-04-05 23:30:33: pid 36687: DEBUG: health check: clearing alarm
2022-04-05 23:30:33: pid 31877: DEBUG: authenticate backend: key data received
2022-04-05 23:30:33: pid 36687: DEBUG: pool_write: to backend: 1 kind:X
2022-04-05 23:30:33: pid 31877: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:33: pid 36687: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:33: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:33: pid 31877: DEBUG: health check: clearing alarm
2022-04-05 23:30:33: pid 31877: DEBUG: pool_write: to backend: 2 kind:X
2022-04-05 23:30:33: pid 31877: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:33: pid 51494: ERROR: Failed to check replication time lag
2022-04-05 23:30:33: pid 51494: DETAIL: No persistent db connection for the node 0
2022-04-05 23:30:33: pid 51494: HINT: check sr_check_user and sr_check_password
2022-04-05 23:30:33: pid 51494: CONTEXT: while checking replication time lag
2022-04-05 23:30:33: pid 51494: LOG: failed to connect to PostgreSQL server on "172.20.8.7:5432", getsockopt() detected error "Connection refused"
2022-04-05 23:30:33: pid 51494: ERROR: failed to make persistent db connection
2022-04-05 23:30:33: pid 51494: DETAIL: connection to host:"172.20.8.7:5432" failed
2022-04-05 23:30:33: pid 51494: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:33: pid 51494: DEBUG: pool_read: read 315 bytes from backend 1
2022-04-05 23:30:33: pid 51494: DEBUG: authenticate kind = 0
2022-04-05 23:30:33: pid 51494: DEBUG: authenticate backend: key data received
2022-04-05 23:30:33: pid 51494: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:33: pid 51494: DEBUG: pool_flush_it: flush size: 35
2022-04-05 23:30:33: pid 51494: DEBUG: pool_read: read 315 bytes from backend 2
2022-04-05 23:30:33: pid 51494: DEBUG: authenticate kind = 0
2022-04-05 23:30:33: pid 51494: DEBUG: authenticate backend: key data received
2022-04-05 23:30:33: pid 51494: DEBUG: authenticate backend: transaction state: I
2022-04-05 23:30:33: pid 51494: DEBUG: pool_write: to backend: 1 kind:X
2022-04-05 23:30:33: pid 51494: DEBUG: pool_flush_it: flush size: 5
2022-04-05 23:30:33: pid 51494: DEBUG: pool_write: to backend: 2 kind:X
2022-04-05 23:30:33: pid 51494: DEBUG: pool_flush_it: flush size: 5
+ exit 0
2022-04-05 23:30:34: pid 55704: LOG: find_primary_node_repeatedly: waiting for finding a primary node
2022-04-05 23:30:34: pid 55704: DEBUG: find_primary_node: ALWAYS_MASTER flag found. Returns node id: 0
2022-04-05 23:30:34: pid 55704: LOG: starting follow degeneration. shutdown host 172.20.8.8(5432)
2022-04-05 23:30:34: pid 55704: LOG: starting follow degeneration. shutdown host 172.20.8.9(5432)
2022-04-05 23:30:34: pid 55704: WARNING: All the DB nodes are in down status and skip writing status file.
2022-04-05 23:30:34: pid 55704: LOG: failover: 2 follow backends have been degenerated
2022-04-05 23:30:34: pid 55704: LOG: failover: set new primary node: 0
2022-04-05 23:30:34: pid 85021: LOG: start triggering follow command.
failover done. shutdown host 172.20.8.7(5432)2022-04-05 23:30:34: pid 85021: LOG: execute command: /etc/pgpool/follow_master.sh 0 172.20.8.7 5432 /var/postgresql/data/ 0 172.20.8.7 0 0 "" ""
2022-04-05 23:30:34: pid 66313: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 77466: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 27783: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 9519: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 31359: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 29323: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 81080: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 5154: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 68484: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 55830: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 61785: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 39663: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 8963: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 87729: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 10561: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 62002: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 4545: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 33983: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 25651: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 62767: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 49555: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 94257: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 91105: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 21400: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 72453: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 46923: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 57491: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 77512: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 23051: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 72639: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 52585: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 75707: DEBUG: initializing backend status
2022-04-05 23:30:34: pid 55704: LOG: failover done. shutdown host 172.20.8.7(5432)
2022-04-05 23:30:34: pid 55704: DEBUG: failover handler
2022-04-05 23:30:34: pid 55704: DETAIL: kind: 4 flags: 4 node_count: 1 index:1
2022-04-05 23:30:34: pid 51494: ERROR: Failed to check replication time lag
2022-04-05 23:30:34: pid 51494: DETAIL: No persistent db connection for the node 0
2022-04-05 23:30:34: pid 51494: HINT: check sr_check_user and sr_check_password
2022-04-05 23:30:34: pid 51494: CONTEXT: while checking replication time lag
+ NODE_ID=0
+ NODE_HOST=172.20.8.7
+ NODE_PORT=5432
+ NODE_PGDATA=/var/postgresql/data/
+ NEW_PRIMARY_NODE_ID=0
+ NEW_PRIMARY_NODE_HOST=172.20.8.7
+ OLD_MAIN_NODE_ID=0
+ OLD_PRIMARY_NODE_ID=0
+ NEW_PRIMARY_NODE_PORT=
+ NEW_PRIMARY_NODE_PGDATA=
+ OLD_PRIMARY_NODE_HOSTNAME=
+ OLD_PRIMARY_NODE_PORT=
+ PGHOME=/opt/postgresql-13.0
+ log=/var/log/pgpool/failover.log
+ echo NODE_ID = 0
+ >> /var/log/pgpool/failover.log
2022-04-05 23:30:34: pid 51494: LOG: worker process received restart request
+ echo NODE_HOST = 172.20.8.7
+ >> /var/log/pgpool/failover.log
+ echo NODE_PORT = 5432
+ >> /var/log/pgpool/failover.log
+ echo NODE_PGDATA = /var/postgresql/data/
+ >> /var/log/pgpool/failover.log
+ echo NEW_PRIMARY_NODE_ID = 0
+ >> /var/log/pgpool/failover.log
+ echo NEW_PRIMARY_NODE_HOST = 172.20.8.7
+ >> /var/log/pgpool/failover.log
+ echo OLD_MAIN_NODE_ID = 0
+ >> /var/log/pgpool/failover.log
+ echo OLD_PRIMARY_NODE_ID = 0
+ >> /var/log/pgpool/failover.log
+ echo NEW_PRIMARY_NODE_PORT =
+ >> /var/log/pgpool/failover.log
+ echo NEW_PRIMARY_NODE_PGDATA =
+ >> /var/log/pgpool/failover.log
+ echo OLD_PRIMARY_NODE_HOSTNAME =
+ >> /var/log/pgpool/failover.log
+ echo OLD_PRIMARY_NODE_PORT =
+ >> /var/log/pgpool/failover.log
+ echo FOLLOW MASTER START NEW_PRIMARY=172.20.8.7 NODE=0
+ >> /var/log/pgpool/failover.log
+ echo count= 10 DUMP >>> 0 172.20.8.7 5432 /var/postgresql/data/ 0 172.20.8.7 0 0
+ >> /var/log/pgpool/failover.log
+ echo OLD PRIMARY = 0 NEW PRIMARY = 0
+ >> /var/log/pgpool/failover.log
+ exit 0
2022-04-05 23:30:34: pid 55704: DEBUG: failover handler
2022-04-05 23:30:34: pid 55704: DETAIL: starting to select new master node
2022-04-05 23:30:34: pid 85021: LOG: execute command: /etc/pgpool/follow_master.sh 1 172.20.8.8 5432 /var/postgresql/data/ 0 172.20.8.7 0 0 "" ""
2022-04-05 23:30:34: pid 55704: LOG: failover: no backends are promoted
+ NODE_ID=1
+ NODE_HOST=172.20.8.8
+ NODE_PORT=5432
+ NODE_PGDATA=/var/postgresql/data/
+ NEW_PRIMARY_NODE_ID=0
+ NEW_PRIMARY_NODE_HOST=172.20.8.7
+ OLD_MAIN_NODE_ID=0
+ OLD_PRIMARY_NODE_ID=0
+ NEW_PRIMARY_NODE_PORT=
+ NEW_PRIMARY_NODE_PGDATA=
+ OLD_PRIMARY_NODE_HOSTNAME=
+ OLD_PRIMARY_NODE_PORT=
+ PGHOME=/opt/postgresql-13.0
+ log=/var/log/pgpool/failover.log
+ echo NODE_ID = 1
+ >> /var/log/pgpool/failover.log
+ echo NODE_HOST = 172.20.8.8
+ >> /var/log/pgpool/failover.log
+ echo NODE_PORT = 5432
+ >> /var/log/pgpool/failover.log
+ echo NODE_PGDATA = /var/postgresql/data/
+ >> /var/log/pgpool/failover.log
+ echo NEW_PRIMARY_NODE_ID = 0
+ >> /var/log/pgpool/failover.log
+ echo NEW_PRIMARY_NODE_HOST = 172.20.8.7
+ >> /var/log/pgpool/failover.log
+ echo OLD_MAIN_NODE_ID = 0
+ >> /var/log/pgpool/failover.log
+ echo OLD_PRIMARY_NODE_ID = 0
+ >> /var/log/pgpool/failover.log
+ echo NEW_PRIMARY_NODE_PORT =
+ >> /var/log/pgpool/failover.log
+ echo NEW_PRIMARY_NODE_PGDATA =
+ >> /var/log/pgpool/failover.log
+ echo OLD_PRIMARY_NODE_HOSTNAME =
+ >> /var/log/pgpool/failover.log
+ echo OLD_PRIMARY_NODE_PORT =
+ >> /var/log/pgpool/failover.log
+ echo FOLLOW MASTER START NEW_PRIMARY=172.20.8.7 NODE=1
+ >> /var/log/pgpool/failover.log
+ echo count= 10 DUMP >>> 1 172.20.8.8 5432 /var/postgresql/data/ 0 172.20.8.7 0 0
+ >> /var/log/pgpool/failover.log
+ echo OLD PRIMARY = 0 NEW PRIMARY = 0
+ >> /var/log/pgpool/failover.log
+ exit 0
2022-04-05 23:30:34: pid 85021: LOG: execute command: /etc/pgpool/follow_master.sh 2 172.20.8.9 5432 /var/postgresql/data/ 0 172.20.8.7 0 0 "" ""
+ NODE_ID=2
+ NODE_HOST=172.20.8.9
+ NODE_PORT=5432
+ NODE_PGDATA=/var/postgresql/data/
+ NEW_PRIMARY_NODE_ID=0
+ NEW_PRIMARY_NODE_HOST=172.20.8.7
+ OLD_MAIN_NODE_ID=0
+ OLD_PRIMARY_NODE_ID=0
+ NEW_PRIMARY_NODE_PORT=
+ NEW_PRIMARY_NODE_PGDATA=
+ OLD_PRIMARY_NODE_HOSTNAME=
+ OLD_PRIMARY_NODE_PORT=
+ PGHOME=/opt/postgresql-13.0
+ log=/var/log/pgpool/failover.log
+ echo NODE_ID = 2
+ >> /var/log/pgpool/failover.log
+ echo NODE_HOST = 172.20.8.9
+ >> /var/log/pgpool/failover.log
+ echo NODE_PORT = 5432
+ >> /var/log/pgpool/failover.log
+ echo NODE_PGDATA = /var/postgresql/data/
+ >> /var/log/pgpool/failover.log
+ echo NEW_PRIMARY_NODE_ID = 0
+ >> /var/log/pgpool/failover.log
+ echo NEW_PRIMARY_NODE_HOST = 172.20.8.7
+ >> /var/log/pgpool/failover.log
+ echo OLD_MAIN_NODE_ID = 0
+ >> /var/log/pgpool/failover.log
+ echo OLD_PRIMARY_NODE_ID = 0
+ >> /var/log/pgpool/failover.log
+ echo NEW_PRIMARY_NODE_PORT =
+ >> /var/log/pgpool/failover.log
+ echo NEW_PRIMARY_NODE_PGDATA =
+ >> /var/log/pgpool/failover.log
+ echo OLD_PRIMARY_NODE_HOSTNAME =
+ >> /var/log/pgpool/failover.log
+ echo OLD_PRIMARY_NODE_PORT =
+ >> /var/log/pgpool/failover.log
+ echo FOLLOW MASTER START NEW_PRIMARY=172.20.8.7 NODE=2
+ >> /var/log/pgpool/failover.log
+ echo count= 10 DUMP >>> 2 172.20.8.9 5432 /var/postgresql/data/ 0 172.20.8.7 0 0
+ >> /var/log/pgpool/failover.log
+ echo OLD PRIMARY = 0 NEW PRIMARY = 0
+ >> /var/log/pgpool/failover.log
+ exit 0
2022-04-05 23:30:35: pid 4153: LOG: restart request received in pcp child process
2022-04-05 23:30:35: pid 55704: LOG: PCP child 4153 exits with status 0 in failover()
2022-04-05 23:30:35: pid 55704: LOG: fork a new PCP child pid 48235 in failover()
2022-04-05 23:30:35: pid 48235: DEBUG: I am PCP child with pid:48235
2022-04-05 23:30:35: pid 55704: DEBUG: reaper handler
2022-04-05 23:30:35: pid 55704: DEBUG: child process with pid: 85021 exit with SUCCESS. child will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 85021 exits with status 0
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 85021 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: worker child process with pid: 51494 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: fork a new worker child process with pid: 94527
2022-04-05 23:30:35: pid 94527: DEBUG: I am 94527
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 12787 exits with status 256
2022-04-05 23:30:35: pid 94527: DEBUG: initializing backend status
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 12787 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 51291 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 51291 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 73507 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 73507 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 74976 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 74976 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 29142 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 29142 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 810 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 810 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 42332 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 42332 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 28858 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 28858 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 39645 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 39645 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 30400 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 30400 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 29937 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 29937 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 53778 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 53778 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 90427 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 90427 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 36608 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 36608 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 5604 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 5604 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 46699 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 46699 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 58747 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 58747 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 79441 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 79441 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 65556 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 65556 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 68738 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 68738 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 34669 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 34669 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 72898 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 72898 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 91789 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 91789 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 19251 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 19251 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 9248 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 9248 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 34448 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 34448 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 71424 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 71424 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 9367 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 9367 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 92886 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 92886 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 27238 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 27238 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 24713 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 24713 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 77033 exits with status 256
2022-04-05 23:30:35: pid 55704: LOG: child process with pid: 77033 exited with success and will not be restarted
2022-04-05 23:30:35: pid 55704: DEBUG: reaper handler: exiting normally
2022-04-05 23:30:47: pid 55704: WARNING: All the DB nodes are in down status and skip writing status file.
-------------- next part --------------
# ----------------------------
# pgPool-II configuration file
# ----------------------------
#
# This file consists of lines of the form:
#
# name = value
#
# Whitespace may be used. Comments are introduced with "#" anywhere on a line.
# The complete list of parameter names and allowed values can be found in the
# pgPool-II documentation.
#
# This file is read on server startup and when the server receives a SIGHUP
# signal. If you edit the file on a running system, you have to SIGHUP the
# server for the changes to take effect, or use "pgpool reload". Some
# parameters, which are marked below, require a server shutdown and restart to
# take effect.
#
#------------------------------------------------------------------------------
# CONNECTIONS
#------------------------------------------------------------------------------
# - pgpool Connection Settings -
listen_addresses = 'localhost'
# Host name or IP address to listen on:
# '*' for all, '' for no TCP/IP connections
# (change requires restart)
port = 9999
# Port number
# (change requires restart)
socket_dir = '/tmp'
# Unix domain socket path
# The Debian package defaults to
# /var/run/postgresql
# (change requires restart)
reserved_connections = 0
# Number of reserved connections.
# Pgpool-II does not accept connections if over
# num_init_chidlren - reserved_connections.
# - pgpool Communication Manager Connection Settings -
pcp_listen_addresses = '*'
# Host name or IP address for pcp process to listen on:
# '*' for all, '' for no TCP/IP connections
# (change requires restart)
pcp_port = 9898
# Port number for pcp
# (change requires restart)3
pcp_socket_dir = '/tmp'
# Unix domain socket path for pcp
# The Debian package defaults to
# /var/run/postgresql
# (change requires restart)
listen_backlog_multiplier = 2
# Set the backlog parameter of listen(2) to
# num_init_children * listen_backlog_multiplier.
# (change requires restart)
serialize_accept = off
# whether to serialize accept() call to avoid thundering herd problem
# (change requires restart)
# - Backend Connection Settings -
# backend_hostname0 = '172.16.8.3'
# # Host name or IP address to connect to for backend 0
# backend_port0 = 5432
# # Port number for backend 0
# backend_weight0 = 1
# # Weight for backend 0 (only in load balancing mode)
# #backend_data_directory0 = '/data'
# # Data directory for backend 0
# backend_flag0 = 'ALLOW_TO_FAILOVER'
# # Controls various backend behavior
# # ALLOW_TO_FAILOVER, DISALLOW_TO_FAILOVER
# # or ALWAYS_MASTER
# backend_application_name0 = 'server0'
# walsender's application_name, used for "show pool_nodes" command
backend_hostname0 = '172.20.8.7'
backend_port0 = 5432
backend_weight0 = 2
backend_data_directory0 = '/var/postgresql/data/'
backend_flag0 = 'ALLOW_TO_FAILOVER|ALWAYS_MASTER'
backend_application_name0 = 's0'
backend_hostname1 = '172.20.8.8'
backend_port1 = 5432
backend_weight1 = 4
backend_data_directory1 = '/var/postgresql/data/'
backend_flag1 = 'ALLOW_TO_FAILOVER'
backend_application_name1 = 's1'
backend_hostname2 = '172.20.8.9'
backend_port2 = 5432
backend_weight2 = 4
backend_data_directory2 = '/var/postgresql/data/'
backend_flag2 = 'ALLOW_TO_FAILOVER'
backend_application_name2 = 's2'
# - Authentication -
enable_pool_hba = off
# Use pool_hba.conf for client authentication
pool_passwd = ''
# File name of pool_passwd for md5 authentication.
# "" disables pool_passwd.
# (change requires restart)
authentication_timeout = 60
# Delay in seconds to complete client authentication
# 0 means no timeout.
allow_clear_text_frontend_auth = on
# Allow Pgpool-II to use clear text password authentication
# with clients, when pool_passwd does not
# contain the user password
# - SSL Connections -
ssl = off
# Enable SSL support
# (change requires restart)
#ssl_key = './server.key'
# Path to the SSL private key file
# (change requires restart)
#ssl_cert = './server.cert'
# Path to the SSL public certificate file
# (change requires restart)
#ssl_ca_cert = ''
# Path to a single PEM format file
# containing CA root certificate(s)
# (change requires restart)
#ssl_ca_cert_dir = ''
# Directory containing CA root certificate(s)
# (change requires restart)
ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL'
# Allowed SSL ciphers
# (change requires restart)
ssl_prefer_server_ciphers = off
# Use server's SSL cipher preferences,
# rather than the client's
# (change requires restart)
ssl_ecdh_curve = 'prime256v1'
# Name of the curve to use in ECDH key exchange
ssl_dh_params_file = ''
# Name of the file containing Diffie-Hellman parameters used
# for so-called ephemeral DH family of SSL cipher.
#------------------------------------------------------------------------------
# POOLS
#------------------------------------------------------------------------------
# - Concurrent session and pool size -
num_init_children = 32
# Number of concurrent sessions allowed
# (change requires restart)
max_pool = 4
# Number of connection pool caches per connection
# (change requires restart)
# - Life time -
child_life_time = 300
# Pool exits after being idle for this many seconds
child_max_connections = 0
# Pool exits after receiving that many connections
# 0 means no exit
connection_life_time = 0
# Connection to backend closes after being idle for this many seconds
# 0 means no close
client_idle_limit = 0
# Client is disconnected after being idle for that many seconds
# (even inside an explicit transactions!)
# 0 means no disconnection
#------------------------------------------------------------------------------
# LOGS
#------------------------------------------------------------------------------
# - Where to log -
#log_destination = 'stderr'
log_destination = 'syslog,stderr'
# Where to log
# Valid values are combinations of stderr,
# and syslog. Default to stderr.
# - What to log -
log_line_prefix = '%t: pid %p: ' # printf-style string to output at beginning of each log line.
log_connections = off
# Log connections
log_hostname = off
# Hostname will be shown in ps status
# and in logs if connections are logged
log_statement = off
# Log all statements
log_per_node_statement = off
# Log all statements
# with node and backend informations
log_client_messages = off
# Log any client messages
log_standby_delay = 'if_over_threshold'
# Log standby delay
# Valid values are combinations of always,
# if_over_threshold, none
# - Syslog specific -
syslog_facility = 'LOCAL0'
# Syslog local facility. Default to LOCAL0
syslog_ident = 'pgpool'
# Syslog program identification string
# Default to 'pgpool'
# - Debug -
#log_error_verbosity = default # terse, default, or verbose messages
#client_min_messages = debug5 # values in order of decreasing detail:
#client_min_messages = notice # values in order of decreasing detail:
# debug5
# debug4
# debug3
# debug2
# debug1
# log
# notice
# warning
# error
#log_min_messages = debug5
#log_min_messages = warning # values in order of decreasing detail:
# debug5
# debug4
# debug3
# debug2
# debug1
# info
# notice
# warning
# error
# log
# fatal
# panic
#------------------------------------------------------------------------------
# FILE LOCATIONS
#------------------------------------------------------------------------------
pid_file_name = '/var/run/pgpool/pgpool.pid'
# PID file name
# Can be specified as relative to the"
# location of pgpool.conf file or
# as an absolute path
# (change requires restart)
logdir = '/tmp'
# Directory of pgPool status file
# (change requires restart)
#------------------------------------------------------------------------------
# CONNECTION POOLING
#------------------------------------------------------------------------------
connection_cache = on
# Activate connection pools
# (change requires restart)
# Semicolon separated list of queries
# to be issued at the end of a session
# The default is for 8.3 and later
reset_query_list = 'ABORT; DISCARD ALL'
# The following one is for 8.2 and before
#reset_query_list = 'ABORT; RESET ALL; SET SESSION AUTHORIZATION DEFAULT'
#------------------------------------------------------------------------------
# REPLICATION MODE
#------------------------------------------------------------------------------
replication_mode = off
# Activate replication mode
# (change requires restart)
replicate_select = off
# Replicate SELECT statements
# when in replication mode
# replicate_select is higher priority than
# load_balance_mode.
insert_lock = off
# Automatically locks a dummy row or a table
# with INSERT statements to keep SERIAL data
# consistency
# Without SERIAL, no lock will be issued
lobj_lock_table = ''
# When rewriting lo_creat command in
# replication mode, specify table name to
# lock
# - Degenerate handling -
replication_stop_on_mismatch = off
# On disagreement with the packet kind
# sent from backend, degenerate the node
# which is most likely "minority"
# If off, just force to exit this session
failover_if_affected_tuples_mismatch = off
# On disagreement with the number of affected
# tuples in UPDATE/DELETE queries, then
# degenerate the node which is most likely
# "minority".
# If off, just abort the transaction to
# keep the consistency
#------------------------------------------------------------------------------
# LOAD BALANCING MODE
#------------------------------------------------------------------------------
load_balance_mode = on
# Activate load balancing mode
# (change requires restart)
ignore_leading_white_space = on
# Ignore leading white spaces of each query
white_function_list = ''
# Comma separated list of function names
# that don't write to database
# Regexp are accepted
black_function_list = 'currval,lastval,nextval,setval'
# Comma separated list of function names
# that write to database
# Regexp are accepted
black_query_pattern_list = ''
# Semicolon separated list of query patterns
# that should be sent to primary node
# Regexp are accepted
# valid for streaming replicaton mode only.
database_redirect_preference_list = ''
# comma separated list of pairs of database and node id.
# example: postgres:primary,mydb[0-4]:1,mydb[5-9]:2'
# valid for streaming replicaton mode only.
app_name_redirect_preference_list = ''
# comma separated list of pairs of app name and node id.
# example: 'psql:primary,myapp[0-4]:1,myapp[5-9]:standby'
# valid for streaming replicaton mode only.
allow_sql_comments = off
# if on, ignore SQL comments when judging if load balance or
# query cache is possible.
# If off, SQL comments effectively prevent the judgment
# (pre 3.4 behavior).
statement_level_load_balance = off
# Enables statement level load balancing
#------------------------------------------------------------------------------
# MASTER/SLAVE MODE
#------------------------------------------------------------------------------
master_slave_mode = on
# Activate master/slave mode
# (change requires restart)
master_slave_sub_mode = 'stream'
# Master/slave sub mode
# Valid values are combinations stream, slony
# or logical. Default is stream.
# (change requires restart)
# - Streaming -
sr_check_period = 10
# Streaming replication check period
# Disabled (0) by default
sr_check_user = 'pgpool'
# Streaming replication check user
# This is neccessary even if you disable streaming
# replication delay check by sr_check_period = 0
sr_check_password = ''
# Password for streaming replication check user.
# Leaving it empty will make Pgpool-II to first look for the
# Password in pool_passwd file before using the empty password
sr_check_database = 'test'
# Database name for streaming replication check
delay_threshold = 10000000
# Threshold before not dispatching query to standby node
# Unit is in bytes
# Disabled (0) by default
# - Special commands -
#follow_master_command = ''
follow_master_command = '/etc/pgpool/follow_master.sh %d %h %p %D %m %H %M %P %r %R'
# Executes this command after master failover
# Special values:
# %d = node id
# %h = host name
# %p = port number
# %D = database cluster path
# %m = new master node id
# %H = hostname of the new master node
# %M = old master node id
# %P = old primary node id
# %r = new master port number
# %R = new master database cluster path
# %% = '%' character
#------------------------------------------------------------------------------
# HEALTH CHECK GLOBAL PARAMETERS
#------------------------------------------------------------------------------
health_check_period = 5
# Health check period
# Disabled (0) by default
health_check_timeout = 20
# Health check timeout
# 0 means no timeout
health_check_user = 'pgpool'
# Health check user
health_check_password = ''
# Password for health check user
# Leaving it empty will make Pgpool-II to first look for the
# Password in pool_passwd file before using the empty password
health_check_database = 'test'
# Database name for health check. If '', tries 'postgres' frist,
health_check_max_retries = 10
# Maximum number of times to retry a failed health check before giving up.
health_check_retry_delay = 1
# Amount of time to wait (in seconds) between retries.
connect_timeout = 10000
# Timeout value in milliseconds before giving up to connect to backend.
# Default is 10000 ms (10 second). Flaky network user may want to increase
# the value. 0 means no timeout.
# Note that this value is not only used for health check,
# but also for ordinary conection to backend.
#------------------------------------------------------------------------------
# HEALTH CHECK PER NODE PARAMETERS (OPTIONAL)
#------------------------------------------------------------------------------
#health_check_period0 = 0
#health_check_timeout0 = 20
#health_check_user0 = 'nobody'
#health_check_password0 = ''
#health_check_database0 = ''
#health_check_max_retries0 = 0
#health_check_retry_delay0 = 1
#connect_timeout0 = 10000
#------------------------------------------------------------------------------
# FAILOVER AND FAILBACK
#------------------------------------------------------------------------------
failover_command = '/etc/pgpool/failover.sh %d %P %H %R %M %m'
# Executes this command at failover
# Special values:
# %d = failed node id
# %h = failed node host name
# %p = failed node port number
# %D = failed node database cluster path
# %m = new master node id
# %H = new master node hostname
# %M = old master node id
# %P = old primary node id
# %r = new master port number
# %R = new master database cluster path
# %N = old primary node hostname
# %S = old primary node port number
# %% = '%' character
failback_command = ''
# Executes this command at failback.
# Special values:
# %d = failed node id
# %h = failed node host name
# %p = failed node port number
# %D = failed node database cluster path
# %m = new master node id
# %H = new master node hostname
# %M = old master node id
# %P = old primary node id
# %r = new master port number
# %R = new master database cluster path
# %N = old primary node hostname
# %S = old primary node port number
# %% = '%' character
failover_on_backend_error = on
# Initiates failover when reading/writing to the
# backend communication socket fails
# If set to off, pgpool will report an
# error and disconnect the session.
detach_false_primary = off
# Detach false primary if on. Only
# valid in streaming replicaton
# mode and with PostgreSQL 9.6 or
# after.
search_primary_node_timeout = 300
# Timeout in seconds to search for the
# primary node when a failover occurs.
# 0 means no timeout, keep searching
# for a primary node forever.
auto_failback = on
# Dettached backend node reattach automatically
# if replication_state is 'streaming'.
auto_failback_interval = 60
# Min interval of executing auto_failback in
# seconds.
#------------------------------------------------------------------------------
# ONLINE RECOVERY
#------------------------------------------------------------------------------
recovery_user = 'pgpool'
# Online recovery user
recovery_password = ''
# Online recovery password
# Leaving it empty will make Pgpool-II to first look for the
# Password in pool_passwd file before using the empty password
recovery_1st_stage_command = ''
# Executes a command in first stage
recovery_2nd_stage_command = ''
# Executes a command in second stage
recovery_timeout = 90
# Timeout in seconds to wait for the
# recovering node's postmaster to start up
# 0 means no wait
client_idle_limit_in_recovery = 0
# Client is disconnected after being idle
# for that many seconds in the second stage
# of online recovery
# 0 means no disconnection
# -1 means immediate disconnection
#------------------------------------------------------------------------------
# WATCHDOG
#------------------------------------------------------------------------------
# - Enabling -
use_watchdog = on
# Activates watchdog
# (change requires restart)
# -Connection to up stream servers -
trusted_servers = 'ap-001,ap-002'
# trusted server list which are used
# to confirm network connection
# (hostA,hostB,hostC,...)
# (change requires restart)
ping_path = '/sbin'
# ping command path
# (change requires restart)
# - Watchdog communication Settings -
wd_hostname = 'ap-001'
# Host name or IP address of this watchdog
# (change requires restart)
wd_port = 9000
# port number for watchdog service
# (change requires restart)
wd_priority = 1
# priority of this watchdog in leader election
# (change requires restart)
wd_authkey = ''
# Authentication key for watchdog communication
# (change requires restart)
wd_ipc_socket_dir = '/tmp'
# Unix domain socket path for watchdog IPC socket
# The Debian package defaults to
# /var/run/postgresql
# (change requires restart)
# - Virtual IP control Setting -
delegate_IP = ''
# delegate IP address
# If this is empty, virtual IP never bring up.
# (change requires restart)
if_cmd_path = ''
# path to the directory where if_up/down_cmd exists
# If if_up/down_cmd starts with "/", if_cmd_path will be ignored.
# (change requires restart)
if_up_cmd = '/usr/bin/doas ifconfig vio1 inet alias $_IP_$ netmask 255.255.255.0'
# startup delegate IP command
# (change requires restart)
if_down_cmd = '/usr/bin/doas ifconfig vio1 $_IP_$ delete'
# shutdown delegate IP command
# (change requires restart)
# arping_path = '/usr/sbin'
# arping command path
# If arping_cmd starts with "/", if_cmd_path will be ignored.
# (change requires restart)
# arping_cmd = '/usr/bin/sudo /usr/sbin/arping -U $_IP_$ -w 1 -I eth0'
# arping command
# (change requires restart)
# - Behaivor on escalation Setting -
clear_memqcache_on_escalation = on
# Clear all the query cache on shared memory
# when standby pgpool escalate to active pgpool
# (= virtual IP holder).
# This should be off if client connects to pgpool
# not using virtual IP.
# (change requires restart)
wd_escalation_command = ''
# Executes this command at escalation on new active pgpool.
# (change requires restart)
wd_de_escalation_command = ''
# Executes this command when master pgpool resigns from being master.
# (change requires restart)
# - Watchdog consensus settings for failover -
failover_when_quorum_exists = on
# Only perform backend node failover
# when the watchdog cluster holds the quorum
failover_require_consensus = on
# Perform failover when majority of Pgpool-II nodes
# aggrees on the backend node status change
enable_multiple_failover_requests_from_node = off
# A Pgpool-II node can cast multiple votes
# for building the consensus on failover
enable_consensus_with_half_votes = off
# apply majority rule for consensus and quorum computation
# at 50% of votes in a cluster with even number of nodes.
# when enabled the existence of quorum and consensus
# on failover is resolved after receiving half of the
# total votes in the cluster, otherwise both these
# decisions require at least one more vote than
# half of the total votes.
# (change requires restart)
# - Lifecheck Setting -
# -- common --
wd_monitoring_interfaces_list = '' # Comma separated list of interfaces names to monitor.
# if any interface from the list is active the watchdog will
# consider the network is fine
# 'any' to enable monitoring on all interfaces except loopback
# '' to disable monitoring
wd_lifecheck_method = 'query'
# Method of watchdog lifecheck ('heartbeat' or 'query' or 'external')
# (change requires restart)
wd_interval = 10
# lifecheck interval (sec) > 0
# (change requires restart)
# -- heartbeat mode --
wd_heartbeat_port = 9694
# Port number for receiving heartbeat signal
# (change requires restart)
wd_heartbeat_keepalive = 2
# Interval time of sending heartbeat signal (sec)
# (change requires restart)
wd_heartbeat_deadtime = 30
# Deadtime interval for heartbeat signal (sec)
# (change requires restart)
heartbeat_destination0 = 'ap-002'
# Host name or IP address of destination 0
# for sending heartbeat signal.
# (change requires restart)
heartbeat_destination_port0 = 9694
# Port number of destination 0 for sending
# heartbeat signal. Usually this is the
# same as wd_heartbeat_port.
# (change requires restart)
heartbeat_device0 = 'vio1'
# Name of NIC device (such like 'eth0')
# used for sending/receiving heartbeat
# signal to/from destination 0.
# This works only when this is not empty
# and pgpool has root privilege.
# (change requires restart)
#heartbeat_destination1 = 'host0_ip2'
#heartbeat_destination_port1 = 9694
#heartbeat_device1 = ''
# -- query mode --
wd_life_point = 3
# lifecheck retry times
# (change requires restart)
wd_lifecheck_query = 'SELECT 1'
# lifecheck query to pgpool from watchdog
# (change requires restart)
wd_lifecheck_dbname = 'template1'
# Database name connected for lifecheck
# (change requires restart)
wd_lifecheck_user = 'pgpool'
# watchdog user monitoring pgpools in lifecheck
# (change requires restart)
wd_lifecheck_password = ''
# Password for watchdog user in lifecheck
# Leaving it empty will make Pgpool-II to first look for the
# Password in pool_passwd file before using the empty password
# (change requires restart)
# - Other pgpool Connection Settings -
other_pgpool_hostname0 = 'ap-002'
other_pgpool_port0 = 9999
other_wd_port0 = 9000
#other_pgpool_hostname0 = 'host0'
# Host name or IP address to connect to for other pgpool 0
# (change requires restart)
#other_pgpool_port0 = 5432
# Port number for other pgpool 0
# (change requires restart)
#other_wd_port0 = 9000
# Port number for other watchdog 0
# (change requires restart)
#other_pgpool_hostname1 = 'host1'
#other_pgpool_port1 = 5432
#other_wd_port1 = 9000
#------------------------------------------------------------------------------
# OTHERS
#------------------------------------------------------------------------------
relcache_expire = 0
# Life time of relation cache in seconds.
# 0 means no cache expiration(the default).
# The relation cache is used for cache the
# query result against PostgreSQL system
# catalog to obtain various information
# including table structures or if it's a
# temporary table or not. The cache is
# maintained in a pgpool child local memory
# and being kept as long as it survives.
# If someone modify the table by using
# ALTER TABLE or some such, the relcache is
# not consistent anymore.
# For this purpose, cache_expiration
# controls the life time of the cache.
relcache_size = 256
# Number of relation cache
# entry. If you see frequently:
# "pool_search_relcache: cache replacement happend"
# in the pgpool log, you might want to increate this number.
check_temp_table = catalog
# Temporary table check method. catalog, trace or none.
# Default is catalog.
check_unlogged_table = on
# If on, enable unlogged table check in SELECT statements.
# This initiates queries against system catalog of primary/master
# thus increases load of master.
# If you are absolutely sure that your system never uses unlogged tables
# and you want to save access to primary/master, you could turn this off.
# Default is on.
enable_shared_relcache = on
# If on, relation cache stored in memory cache,
# the cache is shared among child process.
# Default is on.
# (change requires restart)
relcache_query_target = master # Target node to send relcache queries. Default is master (primary) node.
# If load_balance_node is specified, queries will be sent to load balance node.
#------------------------------------------------------------------------------
# IN MEMORY QUERY MEMORY CACHE
#------------------------------------------------------------------------------
memory_cache_enabled = off
# If on, use the memory cache functionality, off by default
# (change requires restart)
memqcache_method = 'shmem'
# Cache storage method. either 'shmem'(shared memory) or
# 'memcached'. 'shmem' by default
# (change requires restart)
memqcache_memcached_host = 'localhost'
# Memcached host name or IP address. Mandatory if
# memqcache_method = 'memcached'.
# Defaults to localhost.
# (change requires restart)
memqcache_memcached_port = 11211
# Memcached port number. Mondatory if memqcache_method = 'memcached'.
# Defaults to 11211.
# (change requires restart)
memqcache_total_size = 67108864
# Total memory size in bytes for storing memory cache.
# Mandatory if memqcache_method = 'shmem'.
# Defaults to 64MB.
# (change requires restart)
memqcache_max_num_cache = 1000000
# Total number of cache entries. Mandatory
# if memqcache_method = 'shmem'.
# Each cache entry consumes 48 bytes on shared memory.
# Defaults to 1,000,000(45.8MB).
# (change requires restart)
memqcache_expire = 0
# Memory cache entry life time specified in seconds.
# 0 means infinite life time. 0 by default.
# (change requires restart)
memqcache_auto_cache_invalidation = on
# If on, invalidation of query cache is triggered by corresponding
# DDL/DML/DCL(and memqcache_expire). If off, it is only triggered
# by memqcache_expire. on by default.
# (change requires restart)
memqcache_maxcache = 409600
# Maximum SELECT result size in bytes.
# Must be smaller than memqcache_cache_block_size. Defaults to 400KB.
# (change requires restart)
memqcache_cache_block_size = 1048576
# Cache block size in bytes. Mandatory if memqcache_method = 'shmem'.
# Defaults to 1MB.
# (change requires restart)
memqcache_oiddir = '/var/log/pgpool/oiddir'
# Temporary work directory to record table oids
# (change requires restart)
white_memqcache_table_list = ''
# Comma separated list of table names to memcache
# that don't write to database
# Regexp are accepted
black_memqcache_table_list = ''
# Comma separated list of table names not to memcache
# that don't write to database
# Regexp are accepted
pgpool-general-jp メーリングリストの案内