View Revisions: Issue #288

Summary 0000288: FATAL: failed to read kind from backend
Revision 2017-02-23 16:59 by t-ishii
Description We are randomly getting following errors while using application.
###Pgpool Log##
2017-02-15 08:43:16: pid 12625: DEBUG: reading backend data packet kind
2017-02-15 08:43:16: pid 12625: DETAIL: master node id: 0
2017-02-15 08:43:16: pid 12625: DEBUG: pool_read: read 103 bytes from backend 0
2017-02-15 08:43:16: pid 12625: DEBUG: reading backend data packet kind
2017-02-15 08:43:16: pid 12625: DETAIL: backend:0 of 2 kind = 'E'
2017-02-15 08:43:16: pid 12625: DEBUG: pool_read: read 24 bytes from backend 1
2017-02-15 08:43:16: pid 12625: DEBUG: reading backend data packet kind
2017-02-15 08:43:16: pid 12625: DETAIL: backend:1 of 2 kind = '1'
2017-02-15 08:43:16: pid 12625: DEBUG: read_kind_from_backend max_count:1.000000 num_executed_nodes:2
2017-02-15 08:43:16: pid 12625: WARNING: packet kind of backend 1 ['1'] does not match with master/majority nodes packet kind ['E']
2017-02-15 08:43:16: pid 12625: FATAL: failed to read kind from backend
2017-02-15 08:43:16: pid 12625: DETAIL: kind mismatch among backends. Possible last query was: "set search_path = 'nmc_uat'" kind details are: 0[E: canceling statement due to statement timeout] 1[1]
2017-02-15 08:43:16: pid 12625: HINT: check data consistency among db nodes

##Application Log##

2017-02-15 08:42:14 [10.8.180.1] [Gideesh@nmc_uat] INFO CreateAction: Initialize action of type: com.insta.hms.billing.ClaimSubmissionAction
2017-02-15 08:43:16 [10.8.180.1] [Gideesh@nmc_uat] ERROR DBUtil: Unable to get connection/set search path: org.postgresql.util.PSQLException: FATAL: failed to read kind from backend
  Detail: kind mismatch among backends. Possible last query was: "set search_path = 'nmc_uat'" kind details are: 0[E: canceling statement due to statement timeout] 1[1]
  Hint: check data consistency among db nodes
2017-02-15 08:43:16 [10.8.180.1] [Gideesh@nmc_uat] ERROR DBUtil: Unable to get connection/set search path: org.postgresql.util.PSQLException: This connection has been closed.
2017-02-15 08:43:16 [10.8.180.1] [Gideesh@nmc_uat] ERROR HMSExceptionHandler: Global exception handler caught exception:
java.lang.NullPointerException
        at com.insta.hms.usermanager.RoleDAO.getPageStatVals(RoleDAO.java:233)
        at com.bob.hms.common.HomeAction.execute(HomeAction.java:46)
        at org.apache.struts.chain.commands.servlet.ExecuteAction.execute(ExecuteAction.java:58)
        at org.apache.struts.chain.commands.AbstractExecuteAction.execute(AbstractExecuteAction.java:67)
        at org.apache.struts.chain.commands.ActionCommandBase.execute(ActionCommandBase.java:51)
        at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:190)
        at org.apache.commons.chain.generic.LookupCommand.execute(LookupCommand.java:304)
        at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:190)
        at org.apache.struts.chain.ComposableRequestProcessor.process(ComposableRequestProcessor.java:283)

Revision 2017-02-15 14:54 by hcsatish
Description
We are randomly getting following errors while using application.
###Pgpool Log##
2017-02-15 08:43:16: pid 12625: DEBUG: reading backend data packet kind
2017-02-15 08:43:16: pid 12625: DETAIL: master node id: 0
2017-02-15 08:43:16: pid 12625: DEBUG: pool_read: read 103 bytes from backend 0
2017-02-15 08:43:16: pid 12625: DEBUG: reading backend data packet kind
2017-02-15 08:43:16: pid 12625: DETAIL: backend:0 of 2 kind = 'E'
2017-02-15 08:43:16: pid 12625: DEBUG: pool_read: read 24 bytes from backend 1
2017-02-15 08:43:16: pid 12625: DEBUG: reading backend data packet kind
2017-02-15 08:43:16: pid 12625: DETAIL: backend:1 of 2 kind = '1'
2017-02-15 08:43:16: pid 12625: DEBUG: read_kind_from_backend max_count:1.000000 num_executed_nodes:2
2017-02-15 08:43:16: pid 12625: WARNING: packet kind of backend 1 ['1'] does not match with master/majority nodes packet kind ['E']
2017-02-15 08:43:16: pid 12625: FATAL: failed to read kind from backend
2017-02-15 08:43:16: pid 12625: DETAIL: kind mismatch among backends. Possible last query was: "set search_path = 'nmc_uat'" kind details are: 0[E: canceling statement due to statement timeout] 1[1]
2017-02-15 08:43:16: pid 12625: HINT: check data consistency among db nodes

##Application Log##

2017-02-15 08:42:14 [10.8.180.1] [Gideesh@nmc_uat] INFO CreateAction: Initialize action of type: com.insta.hms.billing.ClaimSubmissionAction
2017-02-15 08:43:16 [10.8.180.1] [Gideesh@nmc_uat] ERROR DBUtil: Unable to get connection/set search path: org.postgresql.util.PSQLException: FATAL: failed to read kind from backend
  Detail: kind mismatch among backends. Possible last query was: "set search_path = 'nmc_uat'" kind details are: 0[E: canceling statement due to statement timeout] 1[1]
  Hint: check data consistency among db nodes
2017-02-15 08:43:16 [10.8.180.1] [Gideesh@nmc_uat] ERROR DBUtil: Unable to get connection/set search path: org.postgresql.util.PSQLException: This connection has been closed.
2017-02-15 08:43:16 [10.8.180.1] [Gideesh@nmc_uat] ERROR HMSExceptionHandler: Global exception handler caught exception:
java.lang.NullPointerException
        at com.insta.hms.usermanager.RoleDAO.getPageStatVals(RoleDAO.java:233)
        at com.bob.hms.common.HomeAction.execute(HomeAction.java:46)
        at org.apache.struts.chain.commands.servlet.ExecuteAction.execute(ExecuteAction.java:58)
        at org.apache.struts.chain.commands.AbstractExecuteAction.execute(AbstractExecuteAction.java:67)
        at org.apache.struts.chain.commands.ActionCommandBase.execute(ActionCommandBase.java:51)
        at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:190)
        at org.apache.commons.chain.generic.LookupCommand.execute(LookupCommand.java:304)
        at org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:190)
        at org.apache.struts.chain.ComposableRequestProcessor.process(ComposableRequestProcessor.java:283)