View Issue Details

IDProjectCategoryView StatusLast Update
0000504Pgpool-II[All Projects] Generalpublic2019-05-21 15:54
ReporterbhuwanAssigned ToMuhammad Usama 
PriorityurgentSeveritymajorReproducibilityalways
Status resolvedResolutionopen 
Product Version3.6.16 
Target VersionFixed in Version 
Summary0000504: Getting error during failover. Failover script is not running properly.
DescriptionHi,
We have configured two Pgpool and two Postgresql server and trying to setup High availability. On database side we replication is working fine. But when master database is goesdown its unable to run failover script properly. Due to this whole system get down.
Additional InformationApr 28 14:40:26 kca-vas-stg-pgp-1 pgpool[76354]: [40-1] 2019-04-28 14:40:26: pid 76354: LOG: processing failover command lock request from IPC socket
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76400: LOG: reading and processing packets
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76400: DETAIL: postmaster on DB node 0 was shutdown by administrative command
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76400: LOG: received degenerate backend request for node_id: 0 from pid [76400]
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool[76354]: [41-1] 2019-04-28 14:40:26: pid 76354: LOG: local pgpool-II node "Linux_kca-vas-stg-pgp-1_5432" is requesting to become a lock holder for failover ID: 63
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76354: LOG: new IPC connection received
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76354: LOG: watchdog received the failover command on IPC socket
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76354: LOG: watchdog received failover command [DEGENERATE_BACKEND_REQUEST]
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76354: LOG: ignoring the failover command [DEGENERATE_BACKEND_REQUEST] request, similar failover request is already in progress
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool[76354]: [42-1] 2019-04-28 14:40:26: pid 76354: LOG: local pgpool-II node "Linux_kca-vas-stg-pgp-1_5432" is the lock holder
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76378: LOG: reading and processing packets
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76378: DETAIL: postmaster on DB node 0 was shutdown by administrative command
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76378: LOG: received degenerate backend request for node_id: 0 from pid [76378]
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool[76354]: [43-1] 2019-04-28 14:40:26: pid 76354: LOG: new IPC connection received
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76354: LOG: new IPC connection received
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76354: LOG: watchdog received the failover command on IPC socket
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76354: LOG: watchdog received failover command [DEGENERATE_BACKEND_REQUEST]
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool[76354]: [44-1] 2019-04-28 14:40:26: pid 76354: LOG: watchdog received the failover command on IPC socket
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76400: LOG: degenerate backend request for 1 node(s) from pid [76400] is canceled by other pgpool
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool[76347]: [14-1] 2019-04-28 14:40:26: pid 76347: LOG: starting degeneration. shutdown host 10.183.126.166(5432)
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76354: LOG: ignoring the failover command [DEGENERATE_BACKEND_REQUEST] request, similar failover request is already in progress
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool[76354]: [45-1] 2019-04-28 14:40:26: pid 76354: LOG: watchdog received failover command [DEGENERATE_BACKEND_REQUEST]
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76378: LOG: degenerate backend request for 1 node(s) from pid [76378] is canceled by other pgpool
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool[76406]: [976-1] 2019-04-28 14:40:26: pid 76406: LOG: reading and processing packets
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76382: LOG: reading and processing packets
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76382: DETAIL: postmaster on DB node 0 was shutdown by administrative command
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76382: LOG: received degenerate backend request for node_id: 0 from pid [76382]
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool[76354]: [46-1] 2019-04-28 14:40:26: pid 76354: LOG: ignoring the failover command [DEGENERATE_BACKEND_REQUEST] request, similar failover request is already in progress
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76354: LOG: new IPC connection received
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76354: LOG: watchdog received the failover command on IPC socket
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76354: LOG: watchdog received failover command [DEGENERATE_BACKEND_REQUEST]
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76354: LOG: ignoring the failover command [DEGENERATE_BACKEND_REQUEST] request, similar failover request is already in progress
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool[76406]: [976-2] 2019-04-28 14:40:26: pid 76406: DETAIL: postmaster on DB node 0 was shutdown by administrative command
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76382: LOG: degenerate backend request for 1 node(s) from pid [76382] is canceled by other pgpool
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool[76406]: [977-1] 2019-04-28 14:40:26: pid 76406: LOG: received degenerate backend request for node_id: 0 from pid [76406]
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76379: LOG: reading and processing packets
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76379: DETAIL: postmaster on DB node 0 was shutdown by administrative command
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76379: LOG: received degenerate backend request for node_id: 0 from pid [76379]
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool[76354]: [47-1] 2019-04-28 14:40:26: pid 76354: LOG: new IPC connection received
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76354: LOG: new IPC connection received
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76354: LOG: watchdog received the failover command on IPC socket
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76354: LOG: watchdog received failover command [DEGENERATE_BACKEND_REQUEST]
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76354: LOG: ignoring the failover command [DEGENERATE_BACKEND_REQUEST] request, similar failover request is already in progress
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool[76380]: [550-1] 2019-04-28 14:40:26: pid 76380: LOG: degenerate backend request for 1 node(s) from pid [76380] is canceled by other pgpool
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76379: LOG: degenerate backend request for 1 node(s) from pid [76379] is canceled by other pgpool
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool[76354]: [48-1] 2019-04-28 14:40:26: pid 76354: LOG: remote pgpool-II node "Linux_kca-vas-stg-pgp-2_5432" is requesting to become a lock holder for failover ID: 63
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76347: LOG: failover: no valid backends node found
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76347: LOG: Restart all children
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool: 2019-04-28 14:40:26: pid 76347: LOG: execute command: /etc/pgpool-II-94/Scripts/failover.sh 0 10.183.126.166 /var/lib/pgsql/9.4/data -1 "" "" 0 0
Apr 28 14:40:26 kca-vas-stg-pgp-1 pgpool[76354]: [49-1] 2019-04-28 14:40:26: pid 76354: LOG: lock holder request denied to remote pgpool-II node "Linux_kca-vas-stg-pgp-2_5432"
Tagsfailover

Activities

bhuwan

2019-04-30 14:26

reporter  

messages (1,422,186 bytes)

bhuwan

2019-05-07 15:34

reporter   ~0002586

Hi,

My issue has resolved. Problem is due to pgpool_status file is not updating.

Issue History

Date Modified Username Field Change
2019-04-30 14:26 bhuwan New Issue
2019-04-30 14:26 bhuwan File Added: messages
2019-04-30 14:26 bhuwan Tag Attached: failover
2019-05-07 08:44 administrator Assigned To => Muhammad Usama
2019-05-07 08:44 administrator Status new => assigned
2019-05-07 08:44 administrator Description Updated View Revisions
2019-05-07 08:44 administrator Additional Information Updated View Revisions
2019-05-07 15:34 bhuwan Note Added: 0002586
2019-05-21 15:54 administrator Status assigned => resolved