View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000692 | Pgpool-II | General | public | 2021-02-16 19:11 | 2021-02-18 17:06 |
| Reporter | thesn@hotmail.com | Assigned To | t-ishii | ||
| Priority | urgent | Severity | major | Reproducibility | N/A |
| Status | feedback | Resolution | open | ||
| Product Version | 4.1.5 | ||||
| Summary | 0000692: Data is not replicated after bring back master to life | ||||
| Description | Hi, My settings: 2-node postgresql 12.5 (n0 & n1) - primary / hot-standby 2-node pgpool 4.1.5 (g0 & g1) my simulation: 1. initial state a. show pool_nodes: n0 - up - primary n1 - up - standby b. client can perform insert into n0 and replicated to n1 2. stop n0 (systemctl stop postgresql-12): a. show pool_nodes: n0 - down - standby n1 - up - primary b. client can still perform insert into n1 3. start n0 (systemctl start postgresql-12) a. show pool_nodes: n0 - down - standby n1 - up - primary b. client can still perform insert into n1 c. BUT data from 2b and 3b are not replicated to n0 d. run pcp_attach_node n0 e. show pool_nodes: n0 - up - primary n1 - up - standby f. the record inserted in step 2b and 3b are not available in n0 Please help, is this expected behavior? How to make n0 (which have become back to primary) received the data inserted in step 2b and 3b? Thanks. Satria | ||||
| Tags | No tags attached. | ||||
|
|
here the the log from n1 (/var/log/pgpool/pgpool.log) Feb 16 16:49:33 bjbmdwdbdev pgpool[68941]: [16-1] 2021-02-16 16:49:33: pid 68941: LOG: child process with pid: 68999 exits with status 256 Feb 16 16:49:33 bjbmdwdbdev pgpool[68941]: [17-1] 2021-02-16 16:49:33: pid 68941: LOG: fork a new child process with pid: 69851 Feb 16 16:49:34 bjbmdwdbdev pgpool[69011]: [11-1] 2021-02-16 16:49:34: pid 69011: LOG: failed to connect to PostgreSQL server on "bjbmwpostgre2:5432", getsockopt() detected error "Connection refused" Feb 16 16:49:34 bjbmdwdbdev pgpool[69011]: [12-1] 2021-02-16 16:49:34: pid 69011: ERROR: failed to make persistent db connection Feb 16 16:49:34 bjbmdwdbdev pgpool[69011]: [12-2] 2021-02-16 16:49:34: pid 69011: DETAIL: connection to host:"bjbmwpostgre2:5432" failed Feb 16 16:49:34 bjbmdwdbdev pgpool[69011]: [13-1] 2021-02-16 16:49:34: pid 69011: LOG: health check retrying on DB node: 0 (round:1) Feb 16 16:49:34 bjbmdwdbdev pgpool[69010]: [11-1] 2021-02-16 16:49:34: pid 69010: LOG: failed to connect to PostgreSQL server on "bjbmwpostgre2:5432", getsockopt() detected error "Connection refused" Feb 16 16:49:34 bjbmdwdbdev pgpool[69010]: [12-1] 2021-02-16 16:49:34: pid 69010: ERROR: failed to make persistent db connection Feb 16 16:49:34 bjbmdwdbdev pgpool[69010]: [12-2] 2021-02-16 16:49:34: pid 69010: DETAIL: connection to host:"bjbmwpostgre2:5432" failed Feb 16 16:49:35 bjbmdwdbdev pgpool[69011]: [14-1] 2021-02-16 16:49:35: pid 69011: LOG: failed to connect to PostgreSQL server on "bjbmwpostgre2:5432", getsockopt() detected error "Connection refused" Feb 16 16:49:35 bjbmdwdbdev pgpool[69011]: [15-1] 2021-02-16 16:49:35: pid 69011: ERROR: failed to make persistent db connection Feb 16 16:49:35 bjbmdwdbdev pgpool[69011]: [15-2] 2021-02-16 16:49:35: pid 69011: DETAIL: connection to host:"bjbmwpostgre2:5432" failed Feb 16 16:49:35 bjbmdwdbdev pgpool[69011]: [16-1] 2021-02-16 16:49:35: pid 69011: LOG: health check retrying on DB node: 0 (round:2) Feb 16 16:49:35 bjbmdwdbdev pgpool[68950]: [27-1] 2021-02-16 16:49:35: pid 68950: LOG: watchdog received the failover command from remote pgpool-II node "bjbmwpostgre2:9999 Linux BJBMWPOSTGRE2" Feb 16 16:49:35 bjbmdwdbdev pgpool[68950]: [28-1] 2021-02-16 16:49:35: pid 68950: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from bjbmwpostgre2:9999 Linux BJBMWPOSTGRE2 Feb 16 16:49:35 bjbmdwdbdev pgpool[68950]: [29-1] 2021-02-16 16:49:35: pid 68950: LOG: we have got the consensus to perform the failover Feb 16 16:49:35 bjbmdwdbdev pgpool[68950]: [29-2] 2021-02-16 16:49:35: pid 68950: DETAIL: 1 node(s) voted in the favor Feb 16 16:49:35 bjbmdwdbdev pgpool[68950]: [30-1] 2021-02-16 16:49:35: pid 68950: LOG: received degenerate backend request for node_id: 0 from pid [68950] Feb 16 16:49:35 bjbmdwdbdev pgpool[68941]: [18-1] 2021-02-16 16:49:35: pid 68941: LOG: Pgpool-II parent process has received failover request Feb 16 16:49:35 bjbmdwdbdev pgpool[68950]: [31-1] 2021-02-16 16:49:35: pid 68950: LOG: new IPC connection received Feb 16 16:49:35 bjbmdwdbdev pgpool[68950]: [32-1] 2021-02-16 16:49:35: pid 68950: LOG: received the failover indication from Pgpool-II on IPC interface Feb 16 16:49:35 bjbmdwdbdev pgpool[68950]: [33-1] 2021-02-16 16:49:35: pid 68950: LOG: watchdog is informed of failover start by the main process Feb 16 16:49:35 bjbmdwdbdev pgpool[68941]: [19-1] 2021-02-16 16:49:35: pid 68941: LOG: starting degeneration. shutdown host bjbmwpostgre2(5432) Feb 16 16:49:35 bjbmdwdbdev pgpool[68941]: [20-1] 2021-02-16 16:49:35: pid 68941: LOG: Restart all children Feb 16 16:49:35 bjbmdwdbdev pgpool[68941]: [21-1] 2021-02-16 16:49:35: pid 68941: LOG: execute command: /etc/pgpool-II/failover.sh 0 bjbmwpostgre2 5432 /var/lib/pgsql/12/data 1 bjbmdwdbdev 0 0 5432 /var/lib/pgsql/12/data bjbmwpostgre2 5432 Feb 16 16:49:35 bjbmdwdbdev pgpool: + exec Feb 16 16:49:35 bjbmdwdbdev pgpool: ++ logger -i -p local1.info Feb 16 16:49:36 bjbmdwdbdev pgpool[69011]: [17-1] 2021-02-16 16:49:36: pid 69011: LOG: failed to connect to PostgreSQL server on "bjbmwpostgre2:5432", getsockopt() detected error "Connection refused" Feb 16 16:49:36 bjbmdwdbdev pgpool[69011]: [18-1] 2021-02-16 16:49:36: pid 69011: ERROR: failed to make persistent db connection Feb 16 16:49:36 bjbmdwdbdev pgpool[69011]: [18-2] 2021-02-16 16:49:36: pid 69011: DETAIL: connection to host:"bjbmwpostgre2:5432" failed Feb 16 16:49:36 bjbmdwdbdev pgpool[69011]: [19-1] 2021-02-16 16:49:36: pid 69011: LOG: health check retrying on DB node: 0 (round:3) Feb 16 16:49:37 bjbmdwdbdev pgpool[68941]: [22-1] 2021-02-16 16:49:37: pid 68941: LOG: find_primary_node_repeatedly: waiting for finding a primary node Feb 16 16:49:37 bjbmdwdbdev pgpool[68941]: [23-1] 2021-02-16 16:49:37: pid 68941: LOG: find_primary_node: primary node is 1 Feb 16 16:49:37 bjbmdwdbdev pgpool[68941]: [24-1] 2021-02-16 16:49:37: pid 68941: LOG: starting follow degeneration. shutdown host bjbmwpostgre2(5432) Feb 16 16:49:37 bjbmdwdbdev pgpool[68941]: [25-1] 2021-02-16 16:49:37: pid 68941: LOG: failover: 1 follow backends have been degenerated Feb 16 16:49:37 bjbmdwdbdev pgpool[68941]: [26-1] 2021-02-16 16:49:37: pid 68941: LOG: failover: set new primary node: 1 Feb 16 16:49:37 bjbmdwdbdev pgpool[68941]: [27-1] 2021-02-16 16:49:37: pid 68941: LOG: failover: set new master node: 1 Feb 16 16:49:37 bjbmdwdbdev pgpool[69888]: [26-1] 2021-02-16 16:49:37: pid 69888: LOG: start triggering follow command. Feb 16 16:49:37 bjbmdwdbdev pgpool[69888]: [27-1] 2021-02-16 16:49:37: pid 69888: LOG: execute command: /etc/pgpool-II/follow_primary.sh 0 bjbmwpostgre2 5432 /var/lib/pgsql/12/data 1 bjbmdwdbdev 0 0 5432 /var/lib/pgsql/12/data Feb 16 16:49:37 bjbmdwdbdev pgpool: + NODE_ID=0 Feb 16 16:49:37 bjbmdwdbdev pgpool: + NODE_HOST=bjbmwpostgre2 Feb 16 16:49:37 bjbmdwdbdev pgpool: + NODE_PORT=5432 Feb 16 16:49:37 bjbmdwdbdev pgpool: + NODE_PGDATA=/var/lib/pgsql/12/data Feb 16 16:49:37 bjbmdwdbdev pgpool: + NEW_PRIMARY_NODE_ID=1 Feb 16 16:49:37 bjbmdwdbdev pgpool: + NEW_PRIMARY_NODE_HOST=bjbmdwdbdev Feb 16 16:49:37 bjbmdwdbdev pgpool: + OLD_MAIN_NODE_ID=0 Feb 16 16:49:37 bjbmdwdbdev pgpool: + OLD_PRIMARY_NODE_ID=0 Feb 16 16:49:37 bjbmdwdbdev pgpool: + NEW_PRIMARY_NODE_PORT=5432 Feb 16 16:49:37 bjbmdwdbdev pgpool: + NEW_PRIMARY_NODE_PGDATA=/var/lib/pgsql/12/data Feb 16 16:49:37 bjbmdwdbdev pgpool: + PGHOME=/usr/pgsql-12 Feb 16 16:49:37 bjbmdwdbdev pgpool: + ARCHIVEDIR=/var/lib/pgsql/archivedir Feb 16 16:49:37 bjbmdwdbdev pgpool: + REPLUSER=repl Feb 16 16:49:37 bjbmdwdbdev pgpool: + PCP_USER=pgpool Feb 16 16:49:37 bjbmdwdbdev pgpool: + PGPOOL_PATH=/usr/bin Feb 16 16:49:37 bjbmdwdbdev pgpool: + PCP_PORT=9898 Feb 16 16:49:37 bjbmdwdbdev pgpool: + echo follow_primary.sh: start: Standby node 0 Feb 16 16:49:37 bjbmdwdbdev pgpool: follow_primary.sh: start: Standby node 0 Feb 16 16:49:37 bjbmdwdbdev pgpool: + ssh -T -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null postgres@bjbmdwdbdev -i /var/lib/pgsql/.ssh/id_rsa_pgpool ls /tmp Feb 16 16:49:37 bjbmdwdbdev pgpool[69010]: [13-1] 2021-02-16 16:49:37: pid 69010: ERROR: Failed to check replication time lag Feb 16 16:49:37 bjbmdwdbdev pgpool[69010]: [13-2] 2021-02-16 16:49:37: pid 69010: DETAIL: No persistent db connection for the node 0 Feb 16 16:49:37 bjbmdwdbdev pgpool[69010]: [13-3] 2021-02-16 16:49:37: pid 69010: HINT: check sr_check_user and sr_check_password Feb 16 16:49:37 bjbmdwdbdev pgpool[69010]: [13-4] 2021-02-16 16:49:37: pid 69010: CONTEXT: while checking replication time lag Feb 16 16:49:37 bjbmdwdbdev pgpool[69010]: [14-1] 2021-02-16 16:49:37: pid 69010: LOG: worker process received restart request Feb 16 16:49:37 bjbmdwdbdev pgpool[68950]: [34-1] 2021-02-16 16:49:37: pid 68950: LOG: new IPC connection received Feb 16 16:49:37 bjbmdwdbdev pgpool[68950]: [35-1] 2021-02-16 16:49:37: pid 68950: LOG: received the failover indication from Pgpool-II on IPC interface Feb 16 16:49:37 bjbmdwdbdev pgpool[68950]: [36-1] 2021-02-16 16:49:37: pid 68950: LOG: watchdog is informed of failover end by the main process Feb 16 16:49:37 bjbmdwdbdev pgpool[68941]: [28-1] 2021-02-16 16:49:37: pid 68941: LOG: failover done. shutdown host bjbmwpostgre2(5432) Feb 16 16:49:37 bjbmdwdbdev pgpool: failover done. shutdown host bjbmwpostgre2(5432)Warning: Permanently added 'bjbmdwdbdev,10.6.226.104' (ECDSA) to the list of known hosts. Feb 16 16:49:37 bjbmdwdbdev pgpool[69011]: [20-1] 2021-02-16 16:49:37: pid 69011: LOG: failed to connect to PostgreSQL server on "bjbmwpostgre2:5432", getsockopt() detected error "Connection refused" Feb 16 16:49:37 bjbmdwdbdev pgpool[69011]: [21-1] 2021-02-16 16:49:37: pid 69011: ERROR: failed to make persistent db connection Feb 16 16:49:37 bjbmdwdbdev pgpool[69011]: [21-2] 2021-02-16 16:49:37: pid 69011: DETAIL: connection to host:"bjbmwpostgre2:5432" failed Feb 16 16:49:37 bjbmdwdbdev pgpool[69011]: [22-1] 2021-02-16 16:49:37: pid 69011: LOG: health check failed on node 0 (timeout:0) Feb 16 16:49:37 bjbmdwdbdev pgpool[69011]: [23-1] 2021-02-16 16:49:37: pid 69011: LOG: received degenerate backend request for node_id: 0 from pid [69011] Feb 16 16:49:37 bjbmdwdbdev pgpool[68950]: [37-1] 2021-02-16 16:49:37: pid 68950: LOG: new IPC connection received Feb 16 16:49:37 bjbmdwdbdev pgpool[68950]: [38-1] 2021-02-16 16:49:37: pid 68950: LOG: watchdog received the failover command from local pgpool-II on IPC interface Feb 16 16:49:37 bjbmdwdbdev pgpool[68950]: [39-1] 2021-02-16 16:49:37: pid 68950: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface Feb 16 16:49:37 bjbmdwdbdev pgpool[68950]: [40-1] 2021-02-16 16:49:37: pid 68950: LOG: we have got the consensus to perform the failover Feb 16 16:49:37 bjbmdwdbdev pgpool[68950]: [40-2] 2021-02-16 16:49:37: pid 68950: DETAIL: 1 node(s) voted in the favor Feb 16 16:49:37 bjbmdwdbdev pgpool: + '[' 0 -ne 0 ']' Feb 16 16:49:37 bjbmdwdbdev pgpool: ++ awk '{print $3}' Feb 16 16:49:37 bjbmdwdbdev pgpool: ++ /usr/pgsql-12/bin/initdb -V Feb 16 16:49:37 bjbmdwdbdev pgpool: ++ sed 's/\..*//' Feb 16 16:49:37 bjbmdwdbdev pgpool: ++ sed 's/\([0-9]*\)[a-zA-Z].*/\1/' Feb 16 16:49:37 bjbmdwdbdev pgpool: + PGVERSION=12 Feb 16 16:49:37 bjbmdwdbdev pgpool: + '[' 12 -ge 12 ']' Feb 16 16:49:37 bjbmdwdbdev pgpool: + echo Feb 16 16:49:37 bjbmdwdbdev pgpool: + RECOVERYCONF=/var/lib/pgsql/12/data/postgresql.auto.conf Feb 16 16:49:37 bjbmdwdbdev pgpool: + ssh -T -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null postgres@bjbmwpostgre2 -i /var/lib/pgsql/.ssh/id_rsa_pgpool /usr/pgsql-12/bin/pg_ctl -w -D /var/lib/pgsql/12/data status Feb 16 16:49:37 bjbmdwdbdev pgpool: Warning: Permanently added 'bjbmwpostgre2,10.6.226.111' (ECDSA) to the list of known hosts. Feb 16 16:49:38 bjbmdwdbdev pgpool[69009]: [12-1] 2021-02-16 16:49:38: pid 69009: LOG: restart request received in pcp child process Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [29-1] 2021-02-16 16:49:38: pid 68941: LOG: PCP child 69009 exits with status 0 in failover() Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [30-1] 2021-02-16 16:49:38: pid 68941: LOG: fork a new PCP child pid 69940 in failover() Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [31-1] 2021-02-16 16:49:38: pid 68941: LOG: Pgpool-II parent process has received failover request Feb 16 16:49:38 bjbmdwdbdev pgpool[69940]: [30-1] 2021-02-16 16:49:38: pid 69940: LOG: PCP process: 69940 started Feb 16 16:49:38 bjbmdwdbdev pgpool[68950]: [41-1] 2021-02-16 16:49:38: pid 68950: LOG: new IPC connection received Feb 16 16:49:38 bjbmdwdbdev pgpool[68950]: [42-1] 2021-02-16 16:49:38: pid 68950: LOG: received the failover indication from Pgpool-II on IPC interface Feb 16 16:49:38 bjbmdwdbdev pgpool[68950]: [43-1] 2021-02-16 16:49:38: pid 68950: LOG: watchdog is informed of failover start by the main process Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [32-1] 2021-02-16 16:49:38: pid 68941: LOG: failover: no backends are degenerated Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [33-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68974 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [34-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68974 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [35-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68976 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [36-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68976 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [37-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68977 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [38-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68977 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [39-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68978 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [40-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68978 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [41-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68979 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [42-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68979 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [43-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68980 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [44-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68980 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [45-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68981 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [46-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68981 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [47-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68982 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [48-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68982 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [49-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68983 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [50-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68983 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [51-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68984 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [52-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68984 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [53-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68985 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [54-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68985 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [55-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68986 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [56-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68986 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [57-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68987 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [58-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68987 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [59-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68988 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [60-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68988 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [61-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68989 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [62-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68989 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [63-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68991 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [64-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68991 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [65-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68992 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [66-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68992 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [67-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68993 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [68-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68993 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [69-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68994 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [70-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68994 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [71-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68995 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [72-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68995 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [73-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68996 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [74-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68996 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [75-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68997 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [76-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68997 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [77-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68998 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [78-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 68998 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [79-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 69000 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [80-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 69000 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [81-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 69001 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [82-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 69001 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [83-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 69002 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [84-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 69002 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [85-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 69003 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [86-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 69003 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [87-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 69004 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [88-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 69004 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [89-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 69005 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [90-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 69005 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [91-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 69006 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [92-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 69006 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [93-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 69007 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [94-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 69007 exited with success and will not be restarted Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [95-1] 2021-02-16 16:49:38: pid 68941: LOG: worker child process with pid: 69010 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [96-1] 2021-02-16 16:49:38: pid 68941: LOG: fork a new worker child process with pid: 69941 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [97-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 69851 exits with status 256 Feb 16 16:49:38 bjbmdwdbdev pgpool[68941]: [98-1] 2021-02-16 16:49:38: pid 68941: LOG: child process with pid: 69851 exited with success and will not be restarted Feb 16 16:49:47 bjbmdwdbdev pgpool: pg_ctl: no server running Feb 16 16:49:47 bjbmdwdbdev pgpool: + '[' 3 -eq 0 ']' Feb 16 16:49:47 bjbmdwdbdev pgpool: + echo follow_primary.sh: failed_nod_id=0 is not running. skipping follow primary command Feb 16 16:49:47 bjbmdwdbdev pgpool: follow_primary.sh: failed_nod_id=0 is not running. skipping follow primary command Feb 16 16:49:47 bjbmdwdbdev pgpool: + exit 0 Feb 16 16:49:47 bjbmdwdbdev pgpool[68941]: [99-1] 2021-02-16 16:49:47: pid 68941: LOG: child process with pid: 69888 exits with status 0 Feb 16 16:49:47 bjbmdwdbdev pgpool[68941]: [100-1] 2021-02-16 16:49:47: pid 68941: LOG: child process with pid: 69888 exited with success and will not be restarted ========================================= Feb 16 16:53:11 bjbmdwdbdev pgpool[69917]: [28-1] 2021-02-16 16:53:11: pid 69917: LOG: new connection received Feb 16 16:53:11 bjbmdwdbdev pgpool[69917]: [28-2] 2021-02-16 16:53:11: pid 69917: DETAIL: connecting host=bjbmwpostgre2 port=37636 Feb 16 16:53:14 bjbmdwdbdev pgpool[69920]: [28-1] 2021-02-16 16:53:14: pid 69920: LOG: new connection received Feb 16 16:53:14 bjbmdwdbdev pgpool[69920]: [28-2] 2021-02-16 16:53:14: pid 69920: DETAIL: connecting host=bjbmwpostgre2 port=37640 ========================================= from the logs above, 1. when n0 is stopped in step 2, looks like pgpool run failover.sh and after that trigger follow_primary.sh is that expected behavior? 2. when n0 is started in step 3, looks like pgpool recognize it too. |
|
|
> 3. start n0 (systemctl start postgresql-12) a. show pool_nodes: You should not had done that. To start node 0 as a standby, you should execute pcp_recovery_node. > 1. when n0 is stopped in step 2, looks like pgpool run failover.sh and after that trigger follow_primary.sh is that expected behavior? Yes. (but the follow_primary.sh skips a node which is not running). |
|
|
Hi, Thanks for your reply. I have tried using the pcp_recovery_node, but always hit invalid password for user postgres. I tried to generate it using: pg_md5 -f pgpool.conf -m -u postgres postgrespassword the result inserted into pool_passwd is NOT the same with select passwd from pg_shadow where usename = 'postgres'; i am missing something here? pls help. |
|
|
or should i just copy the result from select passwd from pg_shadow where usename = 'postgres'; into pool_passwd ? |
|
|
What error message do you get exactly when executing pcp_recovery_node? |
|
|
[root@bjbmdwdbdev pgpool-II]# psql -h 10.6.226.250 -p 9999 -U pgpool postgres -c "show pool_nodes" Password for user pgpool: node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay | replication_state | replication_sync_state | last_status_change ---------+---------------+------+--------+-----------+---------+------------+-------------------+-------------------+-------------------+------------------------+--------------------- 0 | bjbmwpostgre2 | 5432 | up | 0.500000 | primary | 0 | true | 0 | | | 2021-02-17 15:33:24 1 | bjbmdwdbdev | 5432 | down | 0.500000 | standby | 0 | false | 0 | | | 2021-02-17 15:26:12 (2 rows) this is from /var/lib/pgsql/12/data/log/postgresql-Wed.log in bjbmwpostgre2 2021-02-17 15:26:32.065 WIB [12616] ERROR: replication slot "bjbmdwdbdev" does not exist 2021-02-17 15:26:32.065 WIB [12616] STATEMENT: SELECT pg_drop_replication_slot('bjbmdwdbdev') 2021-02-17 15:27:07.513 WIB [12674] FATAL: password authentication failed for user "postgres" 2021-02-17 15:27:07.513 WIB [12674] DETAIL: Password does not match for user "postgres". Connection matched pg_hba.conf line 108: "host all postgres 0.0.0.0/0 md5" ============================================== this is from /var/log/pgpool/pgpool.log in bjbmdwdbdev Feb 17 15:27:06 bjbmdwdbdev pgpool[85344]: [30-1] 2021-02-17 15:27:06: pid 85344: LOG: forked new pcp worker, pid=85376 socket=8 Feb 17 15:27:06 bjbmdwdbdev pgpool[85376]: [30-1] 2021-02-17 15:27:06: pid 85376: LOG: starting recovering node 1 Feb 17 15:27:06 bjbmdwdbdev pgpool[85376]: [31-1] 2021-02-17 15:27:06: pid 85376: ERROR: node recovery failed, unable to connect to master node: 0 Feb 17 15:27:06 bjbmdwdbdev pgpool[85265]: [8-1] 2021-02-17 15:27:06: pid 85265: LOG: failover or failback event detected Feb 17 15:27:06 bjbmdwdbdev pgpool[85264]: [8-1] 2021-02-17 15:27:06: pid 85264: LOG: failover or failback event detected Feb 17 15:27:06 bjbmdwdbdev pgpool[85265]: [8-2] 2021-02-17 15:27:06: pid 85265: DETAIL: restarting myself Feb 17 15:27:06 bjbmdwdbdev pgpool[85264]: [8-2] 2021-02-17 15:27:06: pid 85264: DETAIL: restarting myself Feb 17 15:27:06 bjbmdwdbdev pgpool[85281]: [8-1] 2021-02-17 15:27:06: pid 85281: LOG: failover or failback event detected Feb 17 15:27:06 bjbmdwdbdev pgpool[85281]: [8-2] 2021-02-17 15:27:06: pid 85281: DETAIL: restarting myself Feb 17 15:27:06 bjbmdwdbdev pgpool[85267]: [8-1] 2021-02-17 15:27:06: pid 85267: LOG: failover or failback event detected Feb 17 15:27:06 bjbmdwdbdev pgpool[85283]: [8-1] 2021-02-17 15:27:06: pid 85283: LOG: failover or failback event detected Feb 17 15:27:06 bjbmdwdbdev pgpool[85270]: [8-1] 2021-02-17 15:27:06: pid 85270: LOG: failover or failback event detected Feb 17 15:27:06 bjbmdwdbdev pgpool[85267]: [8-2] 2021-02-17 15:27:06: pid 85267: DETAIL: restarting myself Feb 17 15:27:06 bjbmdwdbdev pgpool[85283]: [8-2] 2021-02-17 15:27:06: pid 85283: DETAIL: restarting myself Feb 17 15:27:06 bjbmdwdbdev pgpool[85270]: [8-2] 2021-02-17 15:27:06: pid 85270: DETAIL: restarting myself Feb 17 15:27:06 bjbmdwdbdev pgpool[85285]: [8-1] 2021-02-17 15:27:06: pid 85285: LOG: failover or failback event detected Feb 17 15:27:06 bjbmdwdbdev pgpool[85285]: [8-2] 2021-02-17 15:27:06: pid 85285: DETAIL: restarting myself Feb 17 15:27:06 bjbmdwdbdev pgpool[85269]: [8-1] 2021-02-17 15:27:06: pid 85269: LOG: failover or failback event detected Feb 17 15:27:06 bjbmdwdbdev pgpool[85269]: [8-2] 2021-02-17 15:27:06: pid 85269: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85272]: [8-1] 2021-02-17 15:27:06: pid 85272: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85272]: [8-2] 2021-02-17 15:27:06: pid 85272: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85273]: [8-1] 2021-02-17 15:27:06: pid 85273: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85273]: [8-2] 2021-02-17 15:27:06: pid 85273: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85295]: [8-1] 2021-02-17 15:27:06: pid 85295: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85295]: [8-2] 2021-02-17 15:27:06: pid 85295: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85291]: [8-1] 2021-02-17 15:27:06: pid 85291: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85291]: [8-2] 2021-02-17 15:27:06: pid 85291: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85271]: [8-1] 2021-02-17 15:27:06: pid 85271: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85271]: [8-2] 2021-02-17 15:27:06: pid 85271: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85274]: [8-1] 2021-02-17 15:27:06: pid 85274: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85274]: [8-2] 2021-02-17 15:27:06: pid 85274: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [31-1] 2021-02-17 15:27:06: pid 85256: LOG: child process with pid: 85264 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85277]: [8-1] 2021-02-17 15:27:06: pid 85277: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85293]: [8-1] 2021-02-17 15:27:06: pid 85293: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85293]: [8-2] 2021-02-17 15:27:06: pid 85293: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85277]: [8-2] 2021-02-17 15:27:06: pid 85277: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85268]: [8-1] 2021-02-17 15:27:06: pid 85268: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85268]: [8-2] 2021-02-17 15:27:06: pid 85268: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [32-1] 2021-02-17 15:27:06: pid 85256: LOG: fork a new child process with pid: 85377 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [33-1] 2021-02-17 15:27:06: pid 85256: LOG: child process with pid: 85265 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85296]: [8-1] 2021-02-17 15:27:06: pid 85296: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85296]: [8-2] 2021-02-17 15:27:06: pid 85296: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85344]: [31-1] 2021-02-17 15:27:06: pid 85344: LOG: PCP process with pid: 85376 exit with SUCCESS. Feb 17 15:27:07 bjbmdwdbdev pgpool[85344]: [32-1] 2021-02-17 15:27:06: pid 85344: LOG: PCP process with pid: 85376 exits with status 0 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [34-1] 2021-02-17 15:27:06: pid 85256: LOG: fork a new child process with pid: 85378 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [35-1] 2021-02-17 15:27:06: pid 85256: LOG: child process with pid: 85267 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85275]: [8-1] 2021-02-17 15:27:06: pid 85275: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85275]: [8-2] 2021-02-17 15:27:06: pid 85275: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85276]: [8-1] 2021-02-17 15:27:06: pid 85276: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85276]: [8-2] 2021-02-17 15:27:06: pid 85276: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85282]: [8-1] 2021-02-17 15:27:06: pid 85282: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85282]: [8-2] 2021-02-17 15:27:06: pid 85282: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [36-1] 2021-02-17 15:27:06: pid 85256: LOG: fork a new child process with pid: 85379 Feb 17 15:27:07 bjbmdwdbdev pgpool[85297]: [8-1] 2021-02-17 15:27:06: pid 85297: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85297]: [8-2] 2021-02-17 15:27:06: pid 85297: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [37-1] 2021-02-17 15:27:06: pid 85256: LOG: child process with pid: 85268 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85289]: [8-1] 2021-02-17 15:27:06: pid 85289: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85289]: [8-2] 2021-02-17 15:27:06: pid 85289: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [38-1] 2021-02-17 15:27:06: pid 85256: LOG: fork a new child process with pid: 85380 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [39-1] 2021-02-17 15:27:06: pid 85256: LOG: child process with pid: 85269 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85278]: [8-1] 2021-02-17 15:27:06: pid 85278: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85278]: [8-2] 2021-02-17 15:27:06: pid 85278: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85279]: [8-1] 2021-02-17 15:27:06: pid 85279: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85279]: [8-2] 2021-02-17 15:27:06: pid 85279: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85286]: [8-1] 2021-02-17 15:27:06: pid 85286: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85294]: [8-1] 2021-02-17 15:27:06: pid 85294: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85286]: [8-2] 2021-02-17 15:27:06: pid 85286: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85294]: [8-2] 2021-02-17 15:27:06: pid 85294: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [40-1] 2021-02-17 15:27:06: pid 85256: LOG: fork a new child process with pid: 85381 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [41-1] 2021-02-17 15:27:06: pid 85256: LOG: child process with pid: 85270 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [42-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85382 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [43-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85271 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85287]: [8-1] 2021-02-17 15:27:07: pid 85287: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85287]: [8-2] 2021-02-17 15:27:07: pid 85287: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85284]: [8-1] 2021-02-17 15:27:07: pid 85284: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85284]: [8-2] 2021-02-17 15:27:07: pid 85284: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85292]: [8-1] 2021-02-17 15:27:07: pid 85292: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85292]: [8-2] 2021-02-17 15:27:07: pid 85292: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [44-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85383 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [45-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85272 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85290]: [8-1] 2021-02-17 15:27:07: pid 85290: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85290]: [8-2] 2021-02-17 15:27:07: pid 85290: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [46-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85384 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [47-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85273 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85288]: [8-1] 2021-02-17 15:27:07: pid 85288: LOG: failover or failback event detected Feb 17 15:27:07 bjbmdwdbdev pgpool[85288]: [8-2] 2021-02-17 15:27:07: pid 85288: DETAIL: restarting myself Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [48-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85385 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [49-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85274 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [50-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85386 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [51-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85275 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [52-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85387 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [53-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85276 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [54-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85388 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [55-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85277 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [56-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85389 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [57-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85278 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [58-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85390 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [59-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85279 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [60-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85391 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [61-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85281 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [62-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85392 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [63-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85282 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [64-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85393 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [65-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85283 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [66-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85394 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [67-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85284 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [68-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85395 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [69-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85285 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [70-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85396 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [71-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85286 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [72-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85397 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [73-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85287 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [74-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85398 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [75-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85288 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [76-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85399 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [77-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85289 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [78-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85400 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [79-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85290 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [80-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85401 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [81-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85291 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [82-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85402 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [83-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85292 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [84-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85403 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [85-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85293 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [86-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85404 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [87-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85294 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [88-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85405 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [89-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85295 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [90-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85406 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [91-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85296 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [92-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85407 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [93-1] 2021-02-17 15:27:07: pid 85256: LOG: child process with pid: 85297 exits with status 256 Feb 17 15:27:07 bjbmdwdbdev pgpool[85256]: [94-1] 2021-02-17 15:27:07: pid 85256: LOG: fork a new child process with pid: 85408 Feb 17 15:27:49 bjbmdwdbdev pgpool[85261]: [9-1] 2021-02-17 15:27:49: pid 85261: LOG: watchdog: lifecheck started |
|
|
what i did before is: [root@BJBMWPOSTGRE2 pgpool-II]# su - postgres Last login: Wed Feb 17 14:11:02 WIB 2021 on pts/2 -bash-4.2$ cd /etc/pgpool-II/ -bash-4.2$ pg_md5 --md5auth --username=postgres postgrespassword the result in pool_passwd is already the same with select passwd from pg_shadow where usename = 'postgres'; |
|
|
Hi, Any idea where else / what else I should have done? I am kind of stuck here. Thanks. |
|
|
> Feb 17 15:27:06 bjbmdwdbdev pgpool[85376]: [31-1] 2021-02-17 15:27:06: pid 85376: ERROR: node recovery failed, unable to connect to master node: 0 This error indicates that recovery process failed to connect to node 0 using "recovery_user" and "recovery_password". You should check them. |
|
|
Hi, Thanks for your reply. That's the issue i am having here. I have used pg_md5 --md5auth --username=postgres postgrespassword which resulted the same md5 password with select passwd from pg_shadow where usename = 'postgres'; but still pgpool said Password does not match for user "postgres". I even tried copy paste the password from select passwd from pg_shadow where usename = 'postgres'; to pool_passwd, but pgpool throw the same error. anything i have missed? |
|
|
Can you share pgpool.conf? |
|
|
Also log_statement = 'all' log_connections = on will give useful info since 2021-02-17 15:27:07.513 WIB [12674] FATAL: password authentication failed for user "postgres" 2021-02-17 15:27:07.513 WIB [12674] DETAIL: Password does not match for user "postgres". Connection matched pg_hba.conf line 108: "host all postgres 0.0.0.0/0 md5" who (which process) got this error and which database is expected to be connected are important. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2021-02-16 19:11 | thesn@hotmail.com | New Issue | |
| 2021-02-16 19:35 | thesn@hotmail.com | Note Added: 0003740 | |
| 2021-02-16 21:05 | t-ishii | Note Added: 0003741 | |
| 2021-02-16 21:06 | t-ishii | Assigned To | => t-ishii |
| 2021-02-16 21:06 | t-ishii | Status | new => feedback |
| 2021-02-17 16:04 | thesn@hotmail.com | Note Added: 0003743 | |
| 2021-02-17 16:04 | thesn@hotmail.com | Status | feedback => assigned |
| 2021-02-17 16:06 | thesn@hotmail.com | Note Added: 0003744 | |
| 2021-02-17 17:05 | t-ishii | Note Added: 0003745 | |
| 2021-02-17 17:06 | t-ishii | Status | assigned => feedback |
| 2021-02-17 17:34 | thesn@hotmail.com | Note Added: 0003746 | |
| 2021-02-17 17:34 | thesn@hotmail.com | Status | feedback => assigned |
| 2021-02-17 17:41 | thesn@hotmail.com | Note Added: 0003747 | |
| 2021-02-18 11:55 | thesn@hotmail.com | Note Added: 0003748 | |
| 2021-02-18 14:19 | t-ishii | Note Added: 0003749 | |
| 2021-02-18 14:19 | t-ishii | Status | assigned => feedback |
| 2021-02-18 16:12 | thesn@hotmail.com | Note Added: 0003750 | |
| 2021-02-18 16:12 | thesn@hotmail.com | Status | feedback => assigned |
| 2021-02-18 16:39 | t-ishii | Note Added: 0003751 | |
| 2021-02-18 16:39 | t-ishii | Note Edited: 0003751 | |
| 2021-02-18 17:06 | t-ishii | Note Added: 0003752 | |
| 2021-02-18 17:06 | t-ishii | Status | assigned => feedback |
