[Pgpool-general] Re: Bad degeneration when shutting down secondary backend

[Tatsuo Ishii]

> Setting replication_stop_on_mismatch to false is insecure, isn't it?. What
> would happen if pgpool detects a data difference between master and slave?.

Unfortunately pgpool's data difference detection is not perfect, and
sometimes it raise false alarm. That's why I recommend setting
replication_stop_on_mismatch to false.

> I think pgpool should understand the EOF in the secondary backend as a fail
> in that backend and not as a data mismatch... I understood this is solved un
> 2.6.1, insn't it?

But your problem is not EOF from the secondary server, is it?

> pool_process_query: kind does not match between 
> backends master(C) secondary(E)
--
Tatsuo Ishii

> > -----Original Message-----
> > From: Tatsuo Ishii [mailto:t-ishii at sra.co.jp] 
> > Sent: Martes, 16 de Agosto de 2005 10:35 a.m.
> > To: Pablo G. Milano
> > Cc: pgpool-general at pgfoundry.org
> > Subject: Re: Bad degeneration when shutting down secondary backend
> > 
> > Oops.
> > 
> > replication_stop_on_mismatch = false
> > 
> > was what I wanted to say.
> > --
> > Tatsuo Ishii
> > 
> > > Hi Tatsuo, thanks for your response.
> > > 
> > > I am already using health check and I have also set
> > > "replication_stop_on_mismatch = true". The only difference 
> > is the pgpool
> > > version. The problem can be reproduced as follows:
> > > 
> > > 1) Setup a pgpool host with two backends in linux boxes 
> > with the settings
> > > explained above.
> > > 2) Start "stressing" the pgpool host with a constant 
> > transaction rate (about
> > > 4 tps).
> > > 3) Halt or reboot the secondary backend.
> > > 
> > > PS: I know that 2.6.2 is the latest version, but the 
> > changelog about the EOF
> > > error was in the 2.6.1 version
> > > 
> > > 
> > > > -----Original Message-----
> > > > From: Tatsuo Ishii [mailto:t-ishii at sra.co.jp] 
> > > > Sent: Domingo, 14 de Agosto de 2005 09:10 a.m.
> > > > To: Pablo G. Milano
> > > > Cc: pgpool-general at pgfoundry.org
> > > > Subject: Re: Bad degeneration when shutting down secondary backend
> > > > 
> > > > > I am using pgpool 2.5.1 with linux kernel 2.6.8
> > > > > 
> > > > > I noticed this repeteable behaviour in a heavily loaded 
> > environment:
> > > > > When shutting down the linux box containing the backend 
> > > > configured as
> > > > > "secondary", while sending transactions to pgpool, 
> > > > degeneration mode shuts
> > > > > down primary backend instead of secondary. The log shows 
> > > > the following:
> > > > > 
> > > > > ---------------------------------------
> > > > > Aug 12 15:04:14 asimov pgpool: 2005-08-12 15:04:14 
> > ERROR: pid 5888:
> > > > > pool_process_query: kind does not match between 
> > backends master(C)
> > > > > secondary(E)
> > > > > Aug 12 15:04:14 asimov pgpool: 2005-08-12 15:04:14 LOG: 
> >   pid 5888:
> > > > > notice_backend_error: master: 1 fail over request from pid 5888
> > > > > Aug 12 15:04:14 asimov pgpool: 2005-08-12 15:04:14 LOG:   
> > > > pid 5876: starting
> > > > > degeneration. shutdown master host asimov(5432)
> > > > > Aug 12 15:04:15 asimov pgpool: 2005-08-12 15:04:15 LOG: 
> >   pid 5876:
> > > > > degeneration done. shutdown master host asimov(5432)
> > > > > ---------------------------------------
> > > > > 
> > > > > It seems that pgpool encounters a data mismatch between 
> > > > backends, when what
> > > > > really happens is that one of them is dying.
> > > > > I've found this in pgpool 2.6.1 changelog:
> > > > > 
> > > > > 	* do not raise failover when read() read encounters 
> > > > EOF. It seems it
> > > > > 	  is a oversight. Backend crush may cause that 
> > but it does not
> > > > > 	  immediately mean postmaster crush
> > > > > 
> > > > > Does this solve the problem I'm having? Should I move to 
> > > > pgpool 2.6.1?
> > > > > Thanks in advance.
> > > > 
> > > > Maybe.
> > > > 
> > > > But for particular error like this:
> > > > 
> > > > > pool_process_query: kind does not match between 
> > backends master(C)
> > > > > secondary(E)
> > > > 
> > > > I don't think upgrading to 2.6.2(the latest version is 2.6.2, not
> > > > 2.6.21, FYI) solves the problem. The query results do not agree
> > > > between two servers. pgpool cannot judge which one is 
> > healthy. So it
> > > > just shuts down the master. 
> > > > 
> > > > What I would sugget here is:
> > > > 
> > > > 1) set following in pgpoo.conf. This will avoid unwanted 
> > degeneration.
> > > > 
> > > >    replication_stop_on_mismatch = true
> > > > 
> > > > 2) upgrade to 2.6.2. and use "health check" 
> > functionality. Heal check
> > > >    will occasionally checks if the servers are healthy and if
> > > >    something goes wrong, automatically starts the degeneration.
> > > > --
> > > > Tatsuo Ishii
> > > > 
> > > 
> > 
>