May 16 08:53:34 debpgstreama pgpool[456]: 2023-05-16 08:53:34.037: main pid 456: LOG: health_check_stats_shared_memory_size: requested size: 12288 May 16 08:53:34 debpgstreama pgpool[456]: 2023-05-16 08:53:34.037: main pid 456: LOG: health_check_stats_shared_memory_size: requested size: 12288 May 16 08:53:34 debpgstreama pgpool[456]: 2023-05-16 08:53:34.037: main pid 456: LOG: memory cache initialized May 16 08:53:34 debpgstreama pgpool[456]: 2023-05-16 08:53:34.037: main pid 456: DETAIL: memcache blocks :64 May 16 08:53:34 debpgstreama pgpool[456]: 2023-05-16 08:53:34.039: main pid 456: LOG: pool_discard_oid_maps: discarded memqcache oid maps May 16 08:53:34 debpgstreama pgpool[456]: 2023-05-16 08:53:34.043: main pid 456: LOG: Setting up socket for 0.0.0.0:9999 May 16 08:53:34 debpgstreama pgpool[456]: 2023-05-16 08:53:34.043: main pid 456: LOG: Setting up socket for :::9999 May 16 08:53:34 debpgstreama pgpool[456]: 2023-05-16 08:53:34.048: main pid 456: LOG: find_primary_node_repeatedly: waiting for finding a primary node May 16 08:53:34 debpgstreama pgpool[456]: 2023-05-16 08:53:34.093: main pid 456: LOG: find_primary_node: primary node is 0 May 16 08:53:34 debpgstreama pgpool[456]: 2023-05-16 08:53:34.093: main pid 456: LOG: find_primary_node: standby node is 1 May 16 08:53:34 debpgstreama pgpool[556]: 2023-05-16 08:53:34.094: pcp_main pid 556: LOG: PCP process: 556 started May 16 08:53:34 debpgstreama pgpool[557]: 2023-05-16 08:53:34.094: sr_check_worker pid 557: LOG: process started May 16 08:53:34 debpgstreama pgpool[558]: 2023-05-16 08:53:34.095: health_check pid 558: LOG: process started May 16 08:53:34 debpgstreama pgpool[559]: 2023-05-16 08:53:34.095: health_check pid 559: LOG: process started May 16 08:53:34 debpgstreama pgpool[456]: 2023-05-16 08:53:34.096: main pid 456: LOG: pgpool-II successfully started. version 4.3.5 (tamahomeboshi) May 16 08:53:34 debpgstreama pgpool[456]: 2023-05-16 08:53:34.096: main pid 456: LOG: node status[0]: 1 May 16 08:53:34 debpgstreama pgpool[456]: 2023-05-16 08:53:34.096: main pid 456: LOG: node status[1]: 2 May 16 08:54:43 debpgstreama pgpool[556]: 2023-05-16 08:54:43.994: pcp_main pid 556: LOG: forked new pcp worker, pid=641 socket=7 May 16 08:54:44 debpgstreama pgpool[556]: 2023-05-16 08:54:44.023: pcp_main pid 556: LOG: PCP process with pid: 641 exit with SUCCESS. May 16 08:54:44 debpgstreama pgpool[556]: 2023-05-16 08:54:44.023: pcp_main pid 556: LOG: PCP process with pid: 641 exits with status 0 May 16 08:56:22 debpgstreama pgpool[558]: 2023-05-16 08:56:22.050: health_check0 pid 558: LOG: failed to connect to PostgreSQL server on "10.0.0.224:5432", timed out May 16 08:56:22 debpgstreama pgpool[558]: 2023-05-16 08:56:22.050: health_check0 pid 558: LOG: health check retrying on DB node: 0 (round:1) May 16 08:56:22 debpgstreama pgpool[557]: 2023-05-16 08:56:22.197: sr_check_worker pid 557: LOG: failed to connect to PostgreSQL server on "10.0.0.224:5432", timed out May 16 08:56:32 debpgstreama pgpool[557]: 2023-05-16 08:56:32.210: sr_check_worker pid 557: ERROR: Failed to check replication time lag May 16 08:56:32 debpgstreama pgpool[557]: 2023-05-16 08:56:32.210: sr_check_worker pid 557: DETAIL: No persistent db connection for the node 0 May 16 08:56:32 debpgstreama pgpool[557]: 2023-05-16 08:56:32.210: sr_check_worker pid 557: HINT: check sr_check_user and sr_check_password May 16 08:56:32 debpgstreama pgpool[557]: 2023-05-16 08:56:32.210: sr_check_worker pid 557: CONTEXT: while checking replication time lag May 16 08:56:33 debpgstreama pgpool[558]: 2023-05-16 08:56:33.061: health_check0 pid 558: LOG: failed to connect to PostgreSQL server on "10.0.0.224:5432", timed out May 16 08:56:33 debpgstreama pgpool[558]: 2023-05-16 08:56:33.062: health_check0 pid 558: LOG: health check retrying on DB node: 0 (round:2) May 16 08:56:42 debpgstreama pgpool[557]: 2023-05-16 08:56:42.220: sr_check_worker pid 557: LOG: failed to connect to PostgreSQL server on "10.0.0.224:5432", timed out May 16 08:56:44 debpgstreama pgpool[558]: 2023-05-16 08:56:44.072: health_check0 pid 558: LOG: failed to connect to PostgreSQL server on "10.0.0.224:5432", timed out May 16 08:56:44 debpgstreama pgpool[558]: 2023-05-16 08:56:44.072: health_check0 pid 558: LOG: health check retrying on DB node: 0 (round:3) May 16 08:56:52 debpgstreama pgpool[557]: 2023-05-16 08:56:52.226: sr_check_worker pid 557: ERROR: Failed to check replication time lag May 16 08:56:52 debpgstreama pgpool[557]: 2023-05-16 08:56:52.226: sr_check_worker pid 557: DETAIL: No persistent db connection for the node 0 May 16 08:56:52 debpgstreama pgpool[557]: 2023-05-16 08:56:52.226: sr_check_worker pid 557: HINT: check sr_check_user and sr_check_password May 16 08:56:52 debpgstreama pgpool[557]: 2023-05-16 08:56:52.226: sr_check_worker pid 557: CONTEXT: while checking replication time lag May 16 08:56:55 debpgstreama pgpool[558]: 2023-05-16 08:56:55.083: health_check0 pid 558: LOG: failed to connect to PostgreSQL server on "10.0.0.224:5432", timed out May 16 08:56:55 debpgstreama pgpool[558]: 2023-05-16 08:56:55.083: health_check0 pid 558: LOG: health check retrying on DB node: 0 (round:4) May 16 08:57:02 debpgstreama pgpool[557]: 2023-05-16 08:57:02.236: sr_check_worker pid 557: LOG: failed to connect to PostgreSQL server on "10.0.0.224:5432", timed out May 16 08:57:06 debpgstreama pgpool[558]: 2023-05-16 08:57:06.094: health_check0 pid 558: LOG: failed to connect to PostgreSQL server on "10.0.0.224:5432", timed out May 16 08:57:06 debpgstreama pgpool[558]: 2023-05-16 08:57:06.094: health_check0 pid 558: LOG: health check retrying on DB node: 0 (round:5) May 16 08:57:12 debpgstreama pgpool[557]: 2023-05-16 08:57:12.251: sr_check_worker pid 557: ERROR: Failed to check replication time lag May 16 08:57:12 debpgstreama pgpool[557]: 2023-05-16 08:57:12.251: sr_check_worker pid 557: DETAIL: No persistent db connection for the node 0 May 16 08:57:12 debpgstreama pgpool[557]: 2023-05-16 08:57:12.251: sr_check_worker pid 557: HINT: check sr_check_user and sr_check_password May 16 08:57:12 debpgstreama pgpool[557]: 2023-05-16 08:57:12.251: sr_check_worker pid 557: CONTEXT: while checking replication time lag May 16 08:57:17 debpgstreama pgpool[558]: 2023-05-16 08:57:17.105: health_check0 pid 558: LOG: failed to connect to PostgreSQL server on "10.0.0.224:5432", timed out May 16 08:57:17 debpgstreama pgpool[558]: 2023-05-16 08:57:17.105: health_check0 pid 558: LOG: health check retrying on DB node: 0 (round:6) May 16 08:57:22 debpgstreama pgpool[557]: 2023-05-16 08:57:22.262: sr_check_worker pid 557: LOG: failed to connect to PostgreSQL server on "10.0.0.224:5432", timed out May 16 08:57:28 debpgstreama pgpool[558]: 2023-05-16 08:57:28.116: health_check0 pid 558: LOG: failed to connect to PostgreSQL server on "10.0.0.224:5432", timed out May 16 08:57:28 debpgstreama pgpool[558]: 2023-05-16 08:57:28.116: health_check0 pid 558: LOG: health check failed on node 0 (timeout:0) May 16 08:57:28 debpgstreama pgpool[558]: 2023-05-16 08:57:28.116: health_check0 pid 558: LOG: received degenerate backend request for node_id: 0 from pid [558] May 16 08:57:28 debpgstreama pgpool[558]: 2023-05-16 08:57:28.116: health_check0 pid 558: LOG: signal_user1_to_parent_with_reason(0) May 16 08:57:28 debpgstreama pgpool[456]: 2023-05-16 08:57:28.117: main pid 456: LOG: Pgpool-II parent process received SIGUSR1 May 16 08:57:28 debpgstreama pgpool[456]: 2023-05-16 08:57:28.117: main pid 456: LOG: Pgpool-II parent process has received failover request May 16 08:57:28 debpgstreama pgpool[456]: 2023-05-16 08:57:28.117: main pid 456: LOG: starting degeneration. shutdown host 10.0.0.224(5432) May 16 08:57:28 debpgstreama pgpool[456]: 2023-05-16 08:57:28.125: main pid 456: LOG: Restart all children May 16 08:57:28 debpgstreama pgpool[456]: 2023-05-16 08:57:28.125: main pid 456: LOG: execute command: /etc/pgpool2/scripts/failover.sh 0 10.0.0.224 5432 /data/pgcluster01 1 10.0.0.225 0 0 5432 /data/pgcluster01 10.0.0.224 5432 > /etc/pgpool2/scripts/failover.sh.log 2>&1 May 16 08:57:32 debpgstreama pgpool[557]: 2023-05-16 08:57:32.268: sr_check_worker pid 557: ERROR: Failed to check replication time lag May 16 08:57:32 debpgstreama pgpool[557]: 2023-05-16 08:57:32.268: sr_check_worker pid 557: DETAIL: No persistent db connection for the node 0 May 16 08:57:32 debpgstreama pgpool[557]: 2023-05-16 08:57:32.268: sr_check_worker pid 557: HINT: check sr_check_user and sr_check_password May 16 08:57:32 debpgstreama pgpool[557]: 2023-05-16 08:57:32.268: sr_check_worker pid 557: CONTEXT: while checking replication time lag May 16 08:58:00 debpgstreama pgpool[456]: 2023-05-16 08:58:00.193: main pid 456: LOG: find_primary_node_repeatedly: waiting for finding a primary node May 16 08:58:00 debpgstreama pgpool[456]: 2023-05-16 08:58:00.199: main pid 456: LOG: find_primary_node: primary node is 1 May 16 08:58:00 debpgstreama pgpool[456]: 2023-05-16 08:58:00.199: main pid 456: LOG: failover: set new primary node: 1 May 16 08:58:00 debpgstreama pgpool[456]: 2023-05-16 08:58:00.199: main pid 456: LOG: failover: set new main node: 1 May 16 08:58:00 debpgstreama pgpool[557]: 2023-05-16 08:58:00.205: sr_check_worker pid 557: LOG: worker process received restart request May 16 08:58:00 debpgstreama pgpool[456]: failover done. shutdown host 10.0.0.224(5432)2023-05-16 08:58:00.205: main pid 456: LOG: failover done. shutdown host 10.0.0.224(5432) May 16 08:58:01 debpgstreama pgpool[556]: 2023-05-16 08:58:01.206: pcp_main pid 556: LOG: restart request received in pcp child process May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: PCP child 556 exits with status 0 in failover() May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: fork a new PCP child pid 757 in failover() May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 480 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 481 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 482 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 483 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 484 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 485 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 486 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 487 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 488 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 489 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 490 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 491 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 492 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 493 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 494 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 495 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 496 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 497 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 498 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 499 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 500 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 501 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 502 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 503 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 504 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 505 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 506 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 507 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 508 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 509 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 510 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: child process with pid: 511 exits with status 256 May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.207: main pid 456: LOG: worker child process with pid: 557 exits with status 256 May 16 08:58:01 debpgstreama pgpool[757]: 2023-05-16 08:58:01.207: pcp_main pid 757: LOG: PCP process: 757 started May 16 08:58:01 debpgstreama pgpool[456]: 2023-05-16 08:58:01.208: main pid 456: LOG: fork a new worker child process with pid: 758 May 16 08:58:01 debpgstreama pgpool[758]: 2023-05-16 08:58:01.208: sr_check_worker pid 758: LOG: process started