[Pgpool-general] Re: Bad degeneration when shutting down secondary backend

[Tatsuo Ishii]

> I am using pgpool 2.5.1 with linux kernel 2.6.8
> 
> I noticed this repeteable behaviour in a heavily loaded environment:
> When shutting down the linux box containing the backend configured as
> "secondary", while sending transactions to pgpool, degeneration mode shuts
> down primary backend instead of secondary. The log shows the following:
> 
> ---------------------------------------
> Aug 12 15:04:14 asimov pgpool: 2005-08-12 15:04:14 ERROR: pid 5888:
> pool_process_query: kind does not match between backends master(C)
> secondary(E)
> Aug 12 15:04:14 asimov pgpool: 2005-08-12 15:04:14 LOG:   pid 5888:
> notice_backend_error: master: 1 fail over request from pid 5888
> Aug 12 15:04:14 asimov pgpool: 2005-08-12 15:04:14 LOG:   pid 5876: starting
> degeneration. shutdown master host asimov(5432)
> Aug 12 15:04:15 asimov pgpool: 2005-08-12 15:04:15 LOG:   pid 5876:
> degeneration done. shutdown master host asimov(5432)
> ---------------------------------------
> 
> It seems that pgpool encounters a data mismatch between backends, when what
> really happens is that one of them is dying.
> I've found this in pgpool 2.6.1 changelog:
> 
> 	* do not raise failover when read() read encounters EOF. It seems it
> 	  is a oversight. Backend crush may cause that but it does not
> 	  immediately mean postmaster crush
> 
> Does this solve the problem I'm having? Should I move to pgpool 2.6.1?
> Thanks in advance.

Maybe.

But for particular error like this:

> pool_process_query: kind does not match between backends master(C)
> secondary(E)

I don't think upgrading to 2.6.2(the latest version is 2.6.2, not
2.6.21, FYI) solves the problem. The query results do not agree
between two servers. pgpool cannot judge which one is healthy. So it
just shuts down the master. 

What I would sugget here is:

1) set following in pgpoo.conf. This will avoid unwanted degeneration.

   replication_stop_on_mismatch = true

2) upgrade to 2.6.2. and use "health check" functionality. Heal check
   will occasionally checks if the servers are healthy and if
   something goes wrong, automatically starts the degeneration.
--
Tatsuo Ishii