Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31422: DEBUG: received heartbeat signal from "10.100.230.74(psql-node3):5000" node:Not_Set Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31422: DEBUG: received heartbeat signal from "10.100.230.73(psql-node2):5000" node:Not_Set Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31421: DEBUG: watchdog life checking by heartbeat Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31421: DETAIL: checking pgpool 1 (psql-node2:5000) Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31421: DEBUG: watchdog checking if pgpool is alive using heartbeat Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31421: DETAIL: the last heartbeat from "psql-node2:5000" received 0 seconds ago Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31421: DEBUG: checking pgpool status by heartbeat Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31421: DETAIL: OK; status OK Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31421: DEBUG: watchdog life checking by heartbeat Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31421: DETAIL: checking pgpool 2 (psql-node3:5000) Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31421: DEBUG: watchdog checking if pgpool is alive using heartbeat Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31421: DETAIL: the last heartbeat from "psql-node3:5000" received 0 seconds ago Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31421: DEBUG: checking pgpool status by heartbeat Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31421: DETAIL: OK; status OK Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: received watchdog packet type:f Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: reading packet type f of length 178 Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = MASTER Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: received packet, watchdog node:[psql-node2:5000 Linux psql-node2] command id:[12] type:[FAILOVER FUNCTION COMMAND] state:[MASTER] Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: LOG: watchdog received the failover command from remote pgpool-II node "psql-node2:5000 Linux psql-node2" Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from psql-node2:5000 Linux psql-node2 Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: LOG: we have got the consensus to perform the failover Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DETAIL: 2 node(s) voted in the favor Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: removing failover request from 2 nodes with ID:158 Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: LOG: received degenerate backend request for node_id: 2 from pid [31417] Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: sending signal:10 to the parent process with PID:31415 Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: sending packet, watchdog node:[psql-node2:5000 Linux psql-node2] command id:[12] type:[COMMAND REPLY IN DATA] state:[MASTER] Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: sending watchdog packet to socket:9, type:[-], command ID:12, data Length:54 Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31415: DEBUG: Pgpool-II parent process received SIGUSR1 Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31415: LOG: Pgpool-II parent process has received failover request Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31415: DEBUG: failover handler called Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31415: DEBUG: failover handler Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31415: DETAIL: kind: 1 flags: 3 node_count: 1 index:0 Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: LOG: new IPC connection received Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: LOG: received the failover indication from Pgpool-II on IPC interface Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: LOG: watchdog is informed of failover start by the main process Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: sending packet, watchdog node:[psql-node2:5000 Linux psql-node2] command id:[165] type:[UNKNOWN] state:[MASTER] Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: sending watchdog packet to socket:9, type:[F], command ID:165, data Length:0 Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: sending packet, watchdog node:[psql-node3:5000 Linux psql-node3] command id:[165] type:[UNKNOWN] state:[MASTER] Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: sending watchdog packet to socket:11, type:[F], command ID:165, data Length:0 Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: sending watchdog packet to socket:12, type:[7], command ID:0, data Length:54 Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31415: DEBUG: failover handler Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31415: DETAIL: starting to select new master node Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31415: LOG: starting degeneration. shutdown host psql-node3(5432) Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: received watchdog packet type:f Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: reading packet type f of length 178 Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = MASTER Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: received packet, watchdog node:[psql-node2:5000 Linux psql-node2] command id:[13] type:[FAILOVER FUNCTION COMMAND] state:[MASTER] Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: LOG: watchdog received the failover command from remote pgpool-II node "psql-node2:5000 Linux psql-node2" Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from psql-node2:5000 Linux psql-node2 Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: LOG: we have got the consensus to perform the failover Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DETAIL: 2 node(s) voted in the favor Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: removing failover request from 2 nodes with ID:162 Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: LOG: received degenerate backend request for node_id: 1 from pid [31417] Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: sending packet, watchdog node:[psql-node2:5000 Linux psql-node2] command id:[13] type:[COMMAND REPLY IN DATA] state:[MASTER] Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: sending watchdog packet to socket:9, type:[-], command ID:13, data Length:54 Apr 28 10:33:46 psql-node1 kernel: pgpool[31415]: segfault at 0 ip 00007f75e9ab9845 sp 00007fff15904128 error 4 in libc-2.28.so[7f75e995c000+1b9000] Apr 28 10:33:46 psql-node1 kernel: Code: 00 00 00 00 00 31 c0 c5 f8 77 c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 89 f9 48 89 fa c5 f9 ef c0 83 e1 3f 83 f9 20 77 2b fd 74 0f c5 fd d7 c1 85 c0 0f 85 eb 00 00 00 48 83 c7 20 83 e1 Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31415: DEBUG: failover/failback request details: STREAM: 1 reqkind: 1 detail: 1 node_id: 2 Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31415: LOG: Do not restart children because we are switching over node id 2 host: psql-node3 port: 5432 and we are in streaming replication mode Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: received watchdog packet type:f Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: reading packet type f of length 178 Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = MASTER Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: received packet, watchdog node:[psql-node2:5000 Linux psql-node2] command id:[14] type:[FAILOVER FUNCTION COMMAND] state:[MASTER] Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: LOG: watchdog received the failover command from remote pgpool-II node "psql-node2:5000 Linux psql-node2" Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from psql-node2:5000 Linux psql-node2 Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: LOG: we have got the consensus to perform the failover Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DETAIL: 2 node(s) voted in the favor Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: removing failover request from 2 nodes with ID:160 Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: LOG: received degenerate backend request for node_id: 0 from pid [31417] Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: sending packet, watchdog node:[psql-node2:5000 Linux psql-node2] command id:[14] type:[COMMAND REPLY IN DATA] state:[MASTER] Apr 28 10:33:46 psql-node1 pgpool[31415]: 2021-04-28 10:33:46: pid 31417: DEBUG: sending watchdog packet to socket:9, type:[-], command ID:14, data Length:54 Apr 28 10:33:46 psql-node1 systemd[1]: Started Process Core Dump (PID 32415/UID 0). -- Subject: Unit systemd-coredump@1-32415-0.service has finished start-up -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- Unit systemd-coredump@1-32415-0.service has finished starting up. -- -- The start-up result is done. Apr 28 10:33:47 psql-node1 pgpool[31415]: 2021-04-28 10:33:47: pid 31423: DEBUG: watchdog heartbeat: send heartbeat signal to psql-node2:9694 Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Main process exited, code=dumped, status=11/SEGV Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31417 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31421 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31422 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31423 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31424 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31425 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31426 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31427 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31428 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31429 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31430 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31431 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31432 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31433 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31434 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31435 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31436 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31437 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31438 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31439 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31440 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31441 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31442 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31443 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31444 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31445 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31446 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31447 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31448 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31449 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31450 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31451 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31452 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31453 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31454 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31455 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31456 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31457 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31458 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31459 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31460 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31461 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31462 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31463 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31464 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31465 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31466 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31467 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31468 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31469 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31470 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31471 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31472 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31473 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31474 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31475 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31476 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31477 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31478 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31479 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31480 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31481 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31482 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31483 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31484 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31485 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31486 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31487 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31488 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31489 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31490 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31491 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31492 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31493 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31494 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31495 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31496 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31497 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31498 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31499 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31500 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31501 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31502 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31503 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31504 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31505 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31506 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31507 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31508 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31509 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31510 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31511 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31512 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31513 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31514 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31515 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31516 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31517 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31518 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31519 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31520 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31521 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31522 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31523 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31524 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31525 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31526 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31527 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31528 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31963 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31964 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31965 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31966 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Killing process 31967 (pgpool) with signal SIGKILL. Apr 28 10:33:48 psql-node1 systemd[1]: pgpool-II-13.service: Failed with result 'core-dump'. -- Subject: Unit failed -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- The unit pgpool-II-13.service has entered the 'failed' state with result 'core-dump'. Apr 28 10:33:48 psql-node1 systemd-coredump[32417]: Process 31415 (pgpool) of user 1002 dumped core. Stack trace of thread 31415: #0 0x00007f75e9ab9845 __strlen_avx2 (libc.so.6) #1 0x0000000000490666 string_append_char (pgpool) #2 0x0000000000408041 trigger_failover_command.part.6 (pgpool) #3 0x000000000040bb7e failover (pgpool) #4 0x000000000040d6fd sigusr1_interupt_processor (pgpool) #5 0x000000000040e9c7 PgpoolMain (pgpool) #6 0x00000000004058cd main (pgpool) #7 0x00007f75e997f7b3 __libc_start_main (libc.so.6) #8 0x0000000000405e3e _start (pgpool) -- Subject: Process 31415 (pgpool) dumped core -- Defined-By: systemd -- Support: https://access.redhat.com/support -- Documentation: man:core(5) -- -- Process 31415 (pgpool) crashed and dumped core. -- -- This usually indicates a programming error in the crashing program and -- should be reported to its vendor as a bug. Apr 28 10:33:48 psql-node1 systemd[1]: systemd-coredump@1-32415-0.service: Succeeded. -- Subject: Unit succeeded -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- The unit systemd-coredump@1-32415-0.service has successfully entered the 'dead' state.