[Pgpool-hackers] Promoting any node as master with SR slaves follow

Mon Jan 24 20:17:21 UTC 2011

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Le 24/01/2011 17:38, Gilles Darold a écrit :
> Hi,
> 
> Le 24/01/2011 15:38, Tatsuo Ishii a écrit :
>>> Le 20/01/2011 20:15, Gilles Darold a écrit :
>>>> Hi,
>>>>
>>>> [...]
>>>>
>>>> All this make me do some work on PgPool source code as I think it may
>>>> handle that easily. The patch do the following:
>>>>
>>>> Add a PCP command to promote any node in the pgpool line as the new
>>>> master : pcp_promote_node
>>>> Add a follow master configuration option to call an external command
>>>> when the failover is over.
>>>>
>>>> So that if I run :
>>>>
>>>>     pcp_promote_node 10 192.168.1.11 9898 postgres postgres 3
>>>>
>>>> Node 3 is promoted as master and all other nodes are degenerated.
>>> great. +1
>> Thinking a little bit more, there seems to be an use case where it's
>> not a good idea. Consider a heavyly read system rarely updated. In a
>> case data is not so often updated, maybe once a day. Nodes that were
>> not promoted do not have old data until daily updation occurs. So
>> there's no need to degenerate those nodes until that. If you do not
>> like this, you could always use "delay_threthold". I admit that
>> there's a case immediate degeration is appropreate. So we may need a
>> switch to control the behavior.
> The way I've coded that is very simple : automatic slave degeneration
> and recovery over the new master is done only if you have filled the
> "follow_master_command" configuration directive. In the other case
> nothing is done than the failover and you have all your time to manually
> handle slaves detach and recovery stage once the update happens :-)
> 
> Using a threshold value can be used to delay degeneration during a
> certain laps of time, but I'm not sure that this is something valuable.
> When a cluster is broken what we may want is to be back asap in a stable
> situation and not waiting an hypothetical update on the new master.
> 
> The worse situation with this delay_threshold is when a failover happen
> and wait some minutes/hours the update query. In this laps of time the
> fresh new master crash so that a new failover is done on an other node, etc.
> 
>>>> When the failover is ended PgPool fork a child to execute the command
>>>> given in the follow_master_command successively for each node
>>>> degenerated. The follow master command is simply a call to the
>>>> pcp_recovery_node.
>>> IIRC, we talked a bit about that together a week or two ago.
>>> it seems to me it would be cleaner and simpler to use the parameter
>>> "recovery_end_command" from the postgresql recovery.conf file to send the
>>> command follow_master_command to the pgpool service as soon as the new master is
>>> effectively up and ready.
>> I'll catch up the discussion.
>>
>> BTW how can we determine the promoting standby has finished the
>> reconvery and become ok to accept write queries?
> 
> I'm not agree jehan-guillaume, I think that the context can be fully and
> better handled with PgPool. PostgreSQL on a standby node doesn't  have
> any idea of who are the others standby nodes and then how it can
> advertised them to follow ? 

This is not my point.

0/ we want to promote a slave
1/ we choose and promote the slave PG1
2/ as soon as PG1 exits its standby mode, it executes its "recovery_end_command"
from the recovery.conf file. the "recovery_end_command" contains a script S1
that execute some pcp commands
3/ S1 tells pgpool to degenerate other slaves pgpool knows or not depending on
the use case Tatsuo is describing previously.
4/ S1 tells pgpool to resynchronize any known slaves with PG1
5/ pgpool execute the "follow_master_command" for all slaves

IMO, introducing any kind of delay independant from the PostgreSQL instance or
some try-to-guess-the-situation in the pgpool part would definitely leave a
potential dangerous failure case/situation.

> PgPool always maintained the list and the
> states of the standby nodes.
> 
> Regards
> 

- -- 
Jehan-Guillaume de Rorthais
DBA
http://www.dalibo.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk093lAACgkQXu9L1HbaT6K3FACg4Mx5vb3WfiACrI1RE28/6KBx
UvcAn347sz8dZDCVfssHUmbIWpH3gIsf
=Q/iU
-----END PGP SIGNATURE-----